Kfogel on Launchpad Blog

Getting Patches Upstream

Karl Fogel — Wed, 03 Mar 2010 23:00:51 +0000

We heard from Ubuntu maintainers and upstream developers that it was too easy to lose track of patches in bugs filed on Launchpad — or, put another way, that Launchpad wasn’t doing enough to bring patches to developers’ attention.

According to the Launchpad Upstream Improvements Spec that came out of a recent UDS:

Upstreams frequently complain that it is difficult for them to find Ubuntu patches. patches.ubuntu.com is a diff between Ubuntu and Debian, so it’s not really useful to some upstreams. We want an upstream patch report that shows patches in packages and their age, so we can act on them. …

To help solve this problem, we’ve added two new features to Launchpad.

1. The upstream report now shows bugs carrying patches.

The upstream report now shows the number of bugs carrying patches — see the rightmost column in this screenshot:

(click image to enlarge)

If you click on the number in that column, it brings you to a listing of the bugs with patches.

2. A new “patch report” is available wherever bugs are available.

The second feature is a new patch report available for projects, project groups, source packages, people, teams, distributions, and distroseries. It lists bugs that have patches attached — for those of you who’ve seen Bryce Harrington’s mockup, this is the same thing implemented directly in Launchpad.

The goal of the patch report is to make it easier for packagers and upstream developers to find patches that could be upstreamed, and, especially, to enable them to evaluate those patches quickly.

You can reach the patch report from a new link on any bugs page. On the project bugs page below, for example, see the link “7 bugs with patches” in the portlet on the right (and you can click on any of these screenshots to get a larger version):

Clicking on the “7 bugs with patches” link appends “+patches” to the URL (hint, hint) which takes you to the corresponding patch report. There, patches are listed by bugtask, sorted by the age of the most recent patch in each bug, on the theory that new patches are probably more interesting than old ones:

But you can sort by other things as well — say, by bugtask importance:

Sorting applies across all applicable patches, not just the ones that happen to be on the current page of a potentially multi-page list. Thus changing the sort order might change the set of bugs visible on the current page:

When you hover over a patch, a popup shows details about that patch (who uploaded it, the patch’s filename, and its attachment message):

The patch report for entities like distributions and distroseries arranges patches by source package, since the same bug may have a different bugtask for each package, and each bugtask may have its own status, importance, etc:

You can also view the patch report on a particular source package:

Note that distroseries, persons, teams, and project groups all support this feature too. I didn’t include screenshots for them, to save space, but their patch reports can be reached in the expected way.

Naturally, clicking on a patch takes you straight to the diff:

We’re hoping that these features, combined with the recently-landed patch notification improvements and bug heat, will make it easier to find the patches worth immediate attention.

These patch reports are new in Launchpad 10.02. About a month after they go live, we’re going to look at the stats on how people have been using them, and we’re going to survey some upstreams and Ubuntu maintainers to see what they feel can be improved. Please also file bug reports as you use these new features, of course (if you feel like going the extra mile, tag your bug report with the patch-tracking tag — it’ll save us some time).

Finally, please feel free to help improve the features yourself! The bug links below will lead you to the branches on which these changes were made, so you can see what the code looks like. The Launchpad development wiki is the place to start for hacking on Launchpad.

References:

Launchpad Upstream Improvements Spec
The Bugs/PatchTracking page in the Launchpad Development Wiki
Bug #283097 is about adding a ‘has patch’ column to the +packagebugs page.
Bug #283941 is about showing patches in the upstream report (see linked branch for code change).
Bugs tagged with story-patch-report are about the patch report (again, see linked branches for code changes).

Things we didn’t get to this cycle, but that we’re thinking about:

Finding patches in other distributions (but see Daniel Holbach’s amazing Harvest tool).
Bug #515674 is about showing the patches a distribution is carrying in its packages (i.e., in its debian/patches directories). Note in particular comment #4 there, about UI complexity.
Showing patches in upstream trackers. Such patches might be downstreamable, or they might be the same as patches Launchpad already has; either way, it’d be good to know. See bug #403443.
Package ↔ Branch equivalence. Really, a branch attached to a bug is equivalent to a patch attached to a bug (assuming the base source can be easily located, which it often can be). Ideally, we could represent either one as the other, and treat them as the same in filters and reports.
Launchpad isn’t yet making use of the Debian DEP-3 Patch Tagging Guidelines metadata, which is also the standard in Ubuntu now.
In general, see bugs tagged with patch-tracking and patch-tracking-external.

CORRECTION: Launchpad read-only 23.00 UTC Wed 3rd March - 00.30 UTC Thu 4th March, 2010

Karl Fogel — Mon, 01 Mar 2010 22:32:29 +0000

(This is a correction to a previous post.) Launchpad’s web interface will be read-only for roughly 90 minutes, from 23.00 UTC Wednesday 3rd March to 00.30 UTC Thursday 4th March, 2010. Other aspects of Launchpad — including code hosting, PPAs, the API and email interface — will be unavailable during that time.

Starts: 23.00 UTC, Wednesday 3rd March 2010
Expected back: 00.30 UTC, Thursday 4th March 2010

During this time, we’ll be releasing the code for Launchpad 10.02.

The next planned down-time for Launchpad will be for our 10.03 release on the 31st of March. We’ll publish the exact time, nearer to the release date, on our status page.

You can also see the Launchpad release calendar to check for future release down-time.

Launchpad read-only (corrected; see new post)

Karl Fogel — Mon, 01 Mar 2010 13:22:15 +0000

This post has been corrected; please see the new post.

Connecting Ubuntu, Upstreams and Users.

Karl Fogel — Tue, 15 Dec 2009 07:38:51 +0000

Since we started focusing on bridging the gap between upstream projects, users, and distribution packagers, we’ve been talking to a lot of people, to help us better understand the size and nature of those gaps.

(Those conversations are continuing, too — if you’d like to be a source, especially if you’re an upstream developer whose software is packaged in Ubuntu or Debian and you’ve seen bug reports, translations, or other data come to your project via Launchpad, then please see Matthew Revell’s invitation.)

Meanwhile, we’ve started to synthesize what we’re hearing so far. Recently I went through the raw data to distill it into key points. I filtered out some stuff that had nothing to do with bridging the gap, or that was only mentioned by one or two people. (Conversely, I kept some stuff that I know we’ve heard mentioned by multiple people, even though I may only be able to offer one or two citations here.)

Filtering notwithstanding, what’s interesting is how consistent a lot of the feedback is. The same points kept showing up over and over:

Speed: Launchpad needs to be faster.

Cited by:
- Sandy Armstrong
- Jussi Schultink
- Ted Gould
- Marjo Mercado
- Jorge Castro
There is universal agreement on this. We’re working on it everywhere, as there isn’t one magic fix that will solve it for all of Launchpad.
PPA-of-the-day is highly desirable! The easier it is for users to test, the more testing they will do. Therefore, instant packaging of latest upstream code would be awesomelicious.

Cited by:
- Scott Ritchie
- Sandy Armstrong (more or less)
- Carl Richell
- Ted Gould (see “continuous integration” in his page)
- Jorge Castro
Yup. We’re working on this right now.
Forward bugs more easily, more quickly, and link upstream back to original reporter. (Bryce Harrington, for example, scripts Launchpad very heavily to get what he wants; he points out that the original reporter of a bug should not have to create an account in the upstream tracker to be contacted by upstream — simply having filed in Launchpad should be enough.)

Cited by:
- Sandy Armstrong (as an upstream, wants timely forwarding)
- Bryce Harrington (as a reporter, wants ease-of-forwarding)
- Jorge Castro
We’re trying to nail bug heat and bug Q&A first, but we’re keen to do this too.
Branch/patch equivalence: Transform one into the other in one click, and push changes easily to upstream in the form upstream prefers.

Cited by:
- Jorge Castro
- Cody Russell (sort of)
- Sidnei da Silva
This is a high priority; we’re working on it!
Be able to instantly see attributes of comments or commenters in a bug. E.g., tag the comment as “workaround”, have the commenter be displayed differently because of their role, their credibility, or some other quality. Seeing that kind of information in the UI would save a lot of time.

Cited by:
- Sandy Armstrong (sort of)
- Jussi Schultink
- Ted Gould
This would be great. It’s not on Canonical’s roadmap right now, but we’d certainly be interested in helping any community developer who wanted to have a go at it.
Get more, better information from users in bug reports (e.g., many don’t say the version number of the software). Jussi Schultink mentioned some kind of LP-integrated screen capture would be great; I’m not sure how that would work, but it’s a neat idea.

Cited by:
- Sandy Armstrong
- Jussi Schultink
- Bryce Harrington
This is more a set of related ideas than a fleshed-out requirement, but bug Q&A is probably a step in the right direction.
Launchpad’s search is ill-placed in the UI and/or ineffective

Cited by:
- Jussi Schultink
- Ted Gould
- Ryan Paul
Insofar as placement goes, that should be a relatively easy fix. As for functionality, we should identify exactly what is going wrong (the raw data has more details). On a semi-related note: probably a lot of the searching people do in Launchpad is with the expectation of finding a match somewhere in the vast bugs database, in which case the new inline dupe-finding may help improve that experience, or at least reduce the necessity of having it.
Make it easier to find patches that distro should integrate. This could be a tool that would help everyone: not just Ubuntu, but other distros and upstreams too. Part of the trick is to look at other distros’ trackers for patches, not just at upstream.

Cited by:
- Jorge Castro
- (Others say this too, just happens not to be in the notes I trawled this time… but I know it’s out there.)
This is ultimately the “patch report” functionality that was discussed at UDS. While the full story is quite extensive and probably will not be in Launchpad 4.0, some of the sub-stories will be, in particular the parts that make it much easier to quickly see patches attached to Launchpad bugs, and to generate reports about patches.
Better bug dup searching, better UI for handling bug dup scenarios

Cited by:
- Ted Gould
- Marjo Mercado (esp bugs in other trackers)
Access to older PPAs would be great. (Then users could figure out when a bug was introduced, for example.)

Cited by:
- Scott Ritchie
- Ted Gould
Cross-distro bug forwarding. E.g., LP copies a bug from Fedora’s tracker and then forwards it to the proper upstream.

Cited by:
- Sandy Armstrong
- Jorge Castro (in private conversation)
More ways to view bugs. E.g., omit “fix committed” bugs from list.

Cited by:
- Sandy Armstrong
- Jorge Castro (IIRC, not sure it’s actually in his page)

Interesting that three of those last five related to searching & reporting.

I know much of the above just confirms the direction we’re already going, but I wanted to share it anyway. The emphasis on speed over features was a bit of a surprise to me; good to know.

If you’re really feeling squirrelly, go look over the raw data itself. I left a lot. out above; users have lots of ideas about how to improve Launchpad (as well as much praise for it), and it’s kind of fascinating reading: https://dev.launchpad.net/Strategy/RawData.

The interviewees were:

Bryce Harrington	Canonical - X.org stack
Carl Richell	System 76. Linux computers, yay!
Cody Russell	Canonical - Desktop Experience
Jorge Castro	Canonical - Ubuntu Community
Jussi Schultink	IRC Council, Kubuntu / Ubuntu Studio
Marjo Mercado	Canonical - Ubuntu QA
Ryan Paul	Gwibber upstream
Sandy Armstrong	Tomboy upstream
Scott Ritchie	Packager, mostly for universe, esp WINE
Sidnei da Silva	Canonical - Landscape
Ted Gould	Canonical - indicators / sys status

The sources include a fair number of Canonical employees. That’s just because we had easy access to them right away, while we were still tracking down upstream developers to interview. We will continue to talk to both, but our emphasis is now on upstream developers and on the people who bridge the gap between upstreams and distribution.

Code Hosting offline from 10:00-10:30 UTC, Thursday 1 Oct

Karl Fogel — Tue, 29 Sep 2009 12:38:25 +0000

Launchpad’s Code Hosting service (code.launchpad.net and all Bazaar access) will be offline from 10:00am-10:30am UTC, Thursday, 1 October 2009. Sorry for the short notice; we need to do some unexpected hardware maintenance.

     Going offline: 10am UTC, 1 Oct 2009
     Expected back: 10:30am UTC, 1 Oct 2009

That’s 11am London (BST), 12 noon Western Europe, 2pm Moscow, 6pm Beijing, 8pm Sydney, 3am California, 6am New York.

Removing Launchpad's "Mentoring" Feature.

Karl Fogel — Tue, 15 Sep 2009 21:01:06 +0000

This is a last call for user experience data with Launchpad’s “Mentoring” feature for bugs and blueprints.

We’ve seen only limited usage of this feature, and haven’t gotten much positive feedback on it. Right now, it’s dropped from our 3.0 redesign; but before 3.0 is finalized, we’re interested in hearing people’s experiences with mentoring.

If you’ve used the Mentoring feature, either as a mentor or a mentee, please leave your feedback as a comment here. We’re particularly interested in answers to these questions:

Would you say that Mentoring caused any difference in the rate of resolution between mentored and non-mentored bugs? (It could be a positive or a negative difference; both are interesting to us.)
For people who offered mentoring: in your experience, did you have responses on your mentoring offers? Were you successful in integrating the mentee’s changes?
Do you have any thoughts on how to improve the feature?

Please note: this is about Launchpad’s specific implementation of Mentoring, not about mentoring features in the abstract. We’re interested in your experiences with what’s actually been available in Launchpad. If a few people found the Mentoring feature useful, but most who tried it did not, then we will probably leave it out of Launchpad 3.0. On the other hand, if we hear something very unexpected from the feedback, that could affect our decision of course.

Launchpad is now open source.

Karl Fogel — Tue, 21 Jul 2009 06:01:32 +0000

This is a post I’ve been looking forward to for a long time:

Launchpad is now open source!

We released it today under the GNU Affero General Public license, version 3. Note that although we had previously announced that we’d be holding back two components (codehosting and soyuz), we changed our minds: they are included — all the code is open.

Big congratulations (and thanks) to the Canonical Launchpad team, who worked overtime to make this happen sooner rather than later, and to Mark Shuttleworth, whose decision it was to open source Launchpad in the first place.

Rather than repeat the various release announcements, I’ll just point to them:

The Canonical Launchpad developers will be on IRC in channel #launchpad-dev on irc.freenode.net. That’s the place to go for real time development discussion and questions. For usage issues, #launchpad is still the place, as before.

The mailing list is launchpad-dev {AT} lists.launchpad.net, which you can subscribe to by joining the ~launchpad-dev team. Again, that’s the development mailing list; user questions should still go to launchpad-users {AT} lists.launchpad.net.

Please bear with us as we learn how to be an open source team. Many of the Launchpad developers have open source experience already, of course, but as a team we’ve been working on Launchpad in-house for some years. This is a big change. We’ve been looking forward to it, though, and are ready and eager.

That’s all. Happy hacking :-).

-Karl Fogel

Who's using the Launchpad API?

Karl Fogel — Tue, 31 Mar 2009 17:41:21 +0000

People are starting to integrate Launchpad into applications, talking to it directly through its API — either by using the launchpadlib Python library or by doing RESTful HTTP requests.

Below are some of the uses we’ve found — if you know of some not on this list, please tell us (in the comments here is fine). This is partly to help us see what areas of the API are getting the most mileage, and partly because this collection might be a good resource for people who want example code for using the Launchpad API themselves.

(Note: we’ve now collected this information into a Launchpad API Usage directory, and will maintain the list there from now on.)

bzr-eclipse, a Bazaar plugin for Eclipse, that talks to the Launchpad/Bazaar integration using launchpadlib. The developer says: “The API is straightforward to learn, also if you can use launchpadlib it’s far easier, just start the python interactive interpreter, import launchpadlib and start prototyping your app :)”
Tarmac, “The Launchpad Landing Strip”: an automatic branch lander for the Bazaar branches hosted on Launchpad. Tarmac uses the Launchpad API to manage a development focus branch’s proposed merges. It will automatically merge approved branch merge proposals and push them back up to Launchpad.
apport, a crash detection system that automatically generates reports with debugging information from crashed programs and provides UI frontends for handling these reports. Apport uses launchpadlib in a script (apport-collect) that adds apport data to existing bug reports.
ubuntu-dev-tools, a collection of useful tools that Ubuntu developers use to make their packaging work a lot easier (bug filing, packaging preparation, package analysis, etc). Ubuntu Dev Tools uses launchpadlib in the manage-credentials, requestsync, massfile, lp-set-dup, grab-attachments, and hugdaylist scripts. They’re all sharing the ubuntutools/lp/libsupport.py library, which might be a good place to start.
ubuntu-qa-tools, a collection of tools used by the Ubuntu QA Team (sort of parallel to ubuntu-dev-tools, but for QA). See the ml-fixes-report.py, ml-team-fixes-report.py, b-tool, package-bugs-gravity.py, team-reported-bug-tasks.py, and team-assigned-bug-tasks.py scripts.
ilaunchpad-shell, an interactive shell to Launchpad.net.

See also the API category on the Launchpad Blog, which has posts with code examples, news, etc. And this thread on the Launchpad Users mailing list may accumulate more examples, even after this blog post goes up.

How we're open sourcing Launchpad.

Karl Fogel — Fri, 16 Jan 2009 06:28:17 +0000

Recently we announced that we’re open sourcing Launchpad, with a self-imposed deadline of 21 July 2009. (22 June 2009 ed. note: we may push the date a bit, for testing purposes; it could be in August instead.)

Behind that announcement is a lot of activity. Obviously we’re not going to just throw the code over the wall and wish everyone good luck. The point of doing this is to bring the community that uses Launchpad.net into the development of Launchpad.net. There’s a philosophical reason for this: for a site whose primary purpose is to enable free software projects to collaborate, it’s natural that the users should be involved in improving the site itself.

But there’s a more practical reason, too.

One of my favorite metrics for judging a project’s success is to watch its bug tracker: the more bugs filed, and the faster the rate of new filings, the more successful the project is.

Those who don’t work in software sometimes express puzzlement at that metric: “What? How can more bugs be better?” But software developers know exactly what I mean: the rate of bug filing is proportional to the number of users, not to the number of actual bugs. (More precisely, it’s proportional to some combination of the number of users and their level of technical sophistication, since certain kinds of users are much more likely to file bugs than others.)

Well, I’ve been watching the Launchpad.net bug tracker for several weeks now, and judging by the bug-filing metric, Launchpad.net is very successful. Great. But what that means in practice is that we can’t resolve the majority of incoming bug reports and feature requests. The Launchpad development team at Canonical simply does not have enough surface area to handle it all. If we doubled — indeed, if we quadrupled — the size of the team, it still wouldn’t be enough. With a site like this, whose user base is large, and growing rapidly, there are really only two options:

triage mercilessly, and leave most things undone
open it up and let the users help

Clearly, answer 2 is better.

It implies some preparatory work on our part, however. We’re seeding the ground with developer documentation: this has already begun, though we’re still transferring over much of our internal documentation (remember, we have to go through everything and make sure we don’t accidentally open up any confidential business-related things — all this stuff was written by people who thought it was internal to the company at the time).

We also need to go through our dependencies and make sure we don’t have any license compatibility issues. The license on Launchpad will be the AGPLv3, but we still have to vet dependencies and work out any problems. By definition, this must happen before we release, so it’s work that Canonical must do internally.

Most importantly, we need to move our development discussion forums out into the open. Fortunately, most of Canonical’s Launchpad developers already have plenty of experence working in open source communities: they’re ready for this move, and we’re planning to do it before the code itself is opened (see the schedule), to minimize the number of simultaneous changes. We’ll also be publishing policies on how we’d like the community to work, and on how changes will be accepted back into the mainline and deployed on Launchpad.net. (The short answer is that Canonical decides, of course: we host the site, therefore we are responsible for the software that runs on it. The longer answer is that we want to make it easy for good changes to make it onto the site, and have some concrete plans for how to do that.)

In the near term, the next things you’ll see are more independent modules being opened up: code extracted from Launchpad and made into independent libraries for anyone to use (for example, Storm, LAZR.config, and LAZR.delegates).

And that’s all for now. Watch this space for more.