1

Getting started with launchpadlib: Launchpad’s Python library

Published by Matthew Revell March 11, 2010 in API

Launchpad’s strategist, Jonathan Lange, has started a series of blog posts on getting started with Launchpad’s Python library, launchpadlib:

launchpadlib is the Python client-side library that talks to Launchpad’s own REST API. It turns out that customize scripted control of a bug-tracker-code-hosting-translation-distribution-building-cross-project-collaboration thing is actually quite handy.

Catch-up with the first two posts in the series:

And you can subscribe to Jonathan’s blog or follow blogs from both Jonathan and other members of the Launchpad community on Planet Launchpad.

1

Merge proposals: improved comment handling

Published by Matthew Revell March 4, 2010 in Code

Tim has made a couple of improvements to the way that comments are displayed on merge proposals.

The first is pretty simple but makes it far easier to see which merge proposals need attention. Up until now, if you gave your merge proposal a description it would show up as a comment. This made it harder to see which merge proposals had zero activity.

Now, your description no longer shows up as a comment. And you can edit your description in-line using the same Javascript editor as you’ll find on bug pages.

The other change affects merge proposals that supersede another merge proposal. The merge proposal page explicitly states that it supersedes a previous proposal.

Notice showing the merge proposal supersedes another

It also lists all the comments from that previous review, giving them a yellow background and a link to the superseded review so you can easily tell them apart from comments on the current review.

A comment from the superseded merge proposal

2

You can’t be too helpful: Better handling of large blobs by +filebug

Published by Graham Binns in Bug Tracking

Imagine the scene, dear reader. You’re running the latest Ubuntu pre-release, dilligently testing that everything works the way it ought. An application crashes horribly; Apport pops up to tell you that it spotted the crash – would you like to report a bug?

“Of course I would,” you cry, “I am a desktop testing hero!” And so Apport does its thing and takes you to Launchpad to file the bug report. And then Launchpad times out.

At this point, if you’re like me, you might shout a bit. You refresh and refresh with all your might but still Launchpad will only give you an error page. Finally, defeated, broken, you close your browser and shuffle off into the corner of your room, where you bury yourself under the mountain of discarded CD-Rs that contain daily ISOs of Ubuntus past and sob into your coffee.

Okay, so maybe I’ve exaggerated it a bit, but that doesn’t change the basic fact that timeouts on the +filebug page when you’re filing a bug via Apport are intensely, soul-destroyingly annoying. They’re annoying to us in the Launchpad Bugs team too, because we see them in our error reports. They’re so annoying, dear reader, that we’re moved to hyperbole when writing about them for the official Launchpad blog.

The problem with processing data supplied by Apport has always been one of scale. Originally the extra data that Apport would upload to Launchpad wouldn’t be massive, maybe hitting 10MB if the application was particularly busy. Because of this, Launchpad simply processed those data synchronously whilst loading the bug-filing form for you, so that the data you’d uploaded would be included with the bug.

Of course, that approach doesn’t scale very well, and recently we’ve been seeing the data blobs that Apport uploads hit ~100MB in size. That’s far too big for Launchpad to handle in a timely manner whilst doing all the additional work of rendering the bug filing form for you, so in those circumstances it invariably times out and you get that ever-annoying error page.

This was an interesting problem to solve, and we came up with a number of different possible solutions. One viable solution for this, which we eventually decided not to implement, involved having a separate request for processing the extra bug data and then loading the data into the filebug form with AJAX. We discarded that idea because there was always the chance that that request would time out, leaving us in an only slightly better position than we were already.

There was some discussion on the Launchpad developers mailing list about whether we could just defer loading the extra data into the bug until after it had been filed, but we quickly realised that not only could the extra data carry an initial subject for the bug, but it could also indicate that the bug should be filed as private, something which currently can’t be done via the Launchpad web UI.

The solution we chose to implement, which we’ve now rolled out to the Launchpad edge and production servers, is to have a queue of blobs waiting to be processed. We already have quite a robust job-processing system built into the Launchpad codebase, which we use for creating the diffs in merge proposals and calculating bug heat, amongst other things. Adding support for processing uploaded blobs was quite simple, since the existing blob parsing code was well documented. The blob processing jobs are picked up and run by a cron script that runs every minute or so, and the data retrieved from the blobs are stored with the original processing job as a JSON-formatted string. When you hit the +filebug page it checks to see if the relevant blob has been processed. If not, it waits until processing is complete. Once processing is complete the serialized data are loaded into the +filebug page for use later on.

The advantage to you as a user, then, is that you should never again see a timeout on the +filebug page due to the size of the blob that Apport has uploaded. With the upcoming release of Ubuntu Lucid, we’re hopeful that this will make a real difference to those people testing the pre-release versions of the distro.

0

Getting Patches Upstream

Published by Karl Fogel March 3, 2010 in Bug Tracking

bugs+patches=luv

We heard from Ubuntu maintainers and upstream developers that it was too easy to lose track of patches in bugs filed on Launchpad — or, put another way, that Launchpad wasn’t doing enough to bring patches to developers’ attention.

According to the Launchpad Upstream Improvements Spec that came out of a recent UDS:

Upstreams frequently complain that it is difficult for them to find Ubuntu patches. patches.ubuntu.com is a diff between Ubuntu and Debian, so it’s not really useful to some upstreams. We want an upstream patch report that shows patches in packages and their age, so we can act on them. …

To help solve this problem, we’ve added two new features to Launchpad.

1. The upstream report now shows bugs carrying patches.

The upstream report now shows the number of bugs carrying patches — see the rightmost column in this screenshot:

Upstream report, showing number of patches in rightmost column.
(click image to enlarge)

If you click on the number in that column, it brings you to a listing of the bugs with patches.

2. A new “patch report” is available wherever bugs are available.

The second feature is a new patch report available for projects, project groups, source packages, people, teams, distributions, and distroseries. It lists bugs that have patches attached — for those of you who’ve seen Bryce Harrington’s mockup, this is the same thing implemented directly in Launchpad.

The goal of the patch report is to make it easier for packagers and upstream developers to find patches that could be upstreamed, and, especially, to enable them to evaluate those patches quickly.

You can reach the patch report from a new link on any bugs page. On the project bugs page below, for example, see the link “7 bugs with patches” in the portlet on the right (and you can click on any of these screenshots to get a larger version):

The product bugs page bugfilters portlet shows the number of bugs with patches.

Clicking on the “7 bugs with patches” link appends “+patches” to the URL (hint, hint) which takes you to the corresponding patch report. There, patches are listed by bugtask, sorted by the age of the most recent patch in each bug, on the theory that new patches are probably more interesting than old ones:

The patch report for a project.

But you can sort by other things as well — say, by bugtask importance:

Selecting a sort by importance from the dropdown box of sort orders.

Sorting applies across all applicable patches, not just the ones that happen to be on the current page of a potentially multi-page list. Thus changing the sort order might change the set of bugs visible on the current page:

Patch tasks can be sorted by importance.

When you hover over a patch, a popup shows details about that patch (who uploaded it, the patch’s filename, and its attachment message):

Hovering over a patching displays a popup of information about that patch.

The patch report for entities like distributions and distroseries arranges patches by source package, since the same bug may have a different bugtask for each package, and each bugtask may have its own status, importance, etc:

Patch report for Ubuntu distro.

You can also view the patch report on a particular source package:

Patch report for a source package, about to click on the single patch.

Note that distroseries, persons, teams, and project groups all support this feature too. I didn’t include screenshots for them, to save space, but their patch reports can be reached in the expected way.

Naturally, clicking on a patch takes you straight to the diff:

A patch file.

We’re hoping that these features, combined with the recently-landed patch notification improvements and bug heat, will make it easier to find the patches worth immediate attention.

These patch reports are new in Launchpad 10.02. About a month after they go live, we’re going to look at the stats on how people have been using them, and we’re going to survey some upstreams and Ubuntu maintainers to see what they feel can be improved. Please also file bug reports as you use these new features, of course (if you feel like going the extra mile, tag your bug report with the patch-tracking tag — it’ll save us some time).

Finally, please feel free to help improve the features yourself! The bug links below will lead you to the branches on which these changes were made, so you can see what the code looks like. The Launchpad development wiki is the place to start for hacking on Launchpad.

References:

Things we didn’t get to this cycle, but that we’re thinking about:

0

CORRECTION: Launchpad read-only 23.00 UTC Wed 3rd March – 00.30 UTC Thu 4th March, 2010

Published by kfogel March 1, 2010 in General

(This is a correction to a previous post.) Launchpad’s web interface will be read-only for roughly 90 minutes, from 23.00 UTC Wednesday 3rd March to 00.30 UTC Thursday 4th March, 2010. Other aspects of Launchpad — including code hosting, PPAs, the API and email interface — will be unavailable during that time.

Starts: 23.00 UTC, Wednesday 3rd March 2010
Expected back: 00.30 UTC, Thursday 4th March 2010

During this time, we’ll be releasing the code for Launchpad 10.02.

The next planned down-time for Launchpad will be for our 10.03 release on the 31st of March. We’ll publish the exact time, nearer to the release date, on our status page.

You can also see the Launchpad release calendar to check for future release down-time.

0

Launchpad read-only (corrected; see new post)

Published by kfogel in Notifications

This post has been corrected; please see the new post.

2

Meet Jelmer Vernooij

Published by Matthew Revell February 19, 2010 in Meet the devs

Jelmer Vernooij recently joined the Launchpad team at Canonical, so I caught up with him to give him the usual Meet the developers interview.

Matthew: What do you do on the Launchpad team?

Jelmer: I work in the Soyuz team, where I hack on the build system, package management and archive publishing in Launchpad.

Matthew: Can we see something in Launchpad that you’ve worked on?

Jelmer: Come to think of it, there aren’t yet a lot of visible things that I have worked on in Soyuz so far. However, some of the work that I did on Bazaar is indirectly visible in Launchpad. For example, the Git, Mercurial and (some) Subversion code imports are done through the foreign branch plugins, of which I am the main developer.

Matthew: Where do you work?

Jelmer: I work from home, and at at the moment that’s in Utrecht, a city with about 300k inhabitants in the center of the Netherlands.

Matthew: What can you see from your office window?

Jelmer: I look out over a canal and a small park in front of my apartment. It’s looking nice at the moment, covered in snow.

Matthew What did you do before working at Canonical?

Jelmer: Before joining Canonical I was working on my bachelors degree in Computer Science at the University of Utrecht. I also did part-time work for hacking on Windows software, small device firmware as well various free software projects.

Matthew: How did you get into free software?

Jelmer: In high school I used to help maintain several FreeBSD and Linux servers with Samba. We had a complex script that kept the Samba database in sync with the master database of students (in MySQL) and since this script kept falling over, I decided to add support to Samba to allow it to access our database directly.

Unfortunately the abstractions for Samba’s user database API needed some work and one thing led to another. Before I knew it, I was deep down in Samba development — almost ten years ago now.

Matthew: What’s more important? Principle or pragmatism?

Jelmer: They both have their place. Principles are important, but pragmatism might be necessary to get closer to a point where you can conform to your principles.

I’m certainly a free software pragmatist; I’ll use non-free software as long as there are no viable free software alternatives. The only non-free packages left on my system are skype (there just isn’t anything free that works as well) and the flash plugin for Mozilla (gnash isn’t there yet).

Matthew: Do you/have you contribute(d) to any free software projects?

Jelmer: When I can and where time permits I fix my pet bugs myself, so I’ve contributed a lot of small patches to a variety of projects.

Aside from that there are a few projects I work on on a regular basis. Samba was the first project I worked on and (apart from Launchpad) it still remains the project I spend most of my time on. I seem to have a thing for trying to figure out proprietary protocols — I’m also a developer on the OpenChange (Microsoft Exchange/Outlook) and BitlBee (MSN/AIM/ICQ) projects.

A couple of years ago I became interested in distributed version control systems and started playing with Bazaar. Since Samba was maintained in Subversion I started working on the bzr-svn plugin and later the bzr-git plugin so I could work on Samba with Bazaar, but I’ve also contributed to other areas of Bazaar.

For most of the projects I’m active in I’m also involved in the packaging for Debian and Ubuntu.

Matthew: Tell us something really cool about Launchpad that not enough people know about.

Jelmer: The upstream report gives a nice overview of the state of Ubuntu and how well upstream projects are tracked in Launchpad.

Matthew: Is there one developer who has been an inspiration to you? If so, why?

Jelmer: There are quite a few developers that have inspired me over the years.

I much appreciate the encouragement, mentoring and patience from Andrew Bartlett, Andrew Tridgell and Jerry Carter when I first started contributing to Samba. Without them I probably wouldn’t be involved in free software as much as I am today.

Pair-programming with Robert Collins and John Arbash Meinel during the first few Bazaar sprints taught me a lot about extreme programming and got me hooked on test driven development.

While I occasionally get annoyed by what seems like nitpicking in code reviews for Bazaar, I also realize that it’s great to get so much high-quality feedback. It has definitely made me a better programmer.

Matthew: Okay, Kiko’s special question! You’re at your computer, you reach for your wallet: what are you most likely to be doing?

Jelmer: I’m probably reaching for my credit card to pay for something online, most likely CDs or books.

Matthew: Thanks Jelmer!

0

Sikuli — scripting your use of GUIs

Published by Matthew Revell February 17, 2010 in Projects

Sikuli logo
The Sikuli project recently switched to using Launchpad. I asked Tsung-Hsiang Chang to tell me more about the project.

Matthew: Your website says, “Sikuli is a visual technology to search and automate graphical user interfaces (GUI) using images (screenshots).” That sounds like it’s a general purpose screen-scraping solution. Is that right?

Tsung-Hsiang: Not exactly. Sikuli is not about extracting data from screen.

The current release of Sikuli is called Sikuli Script, which focuses on only automation using screenshots of GUI widgets.

We have another project called Sikuli Search, which queries a search engine using screenshots instead of keywords.

Although Sikuli Script is supposed to be able to “search” buttons or text on the screen, it isn’t good at scraping or analyzing information from screenshots yet.

Matthew: In your YouTube demo video, you show how you can write a Sikuli script that will open an app by name. Then, you can take a screen shot of one of the icons in that app and have Sikuli click it as part of the script. You even show how you just have to take a screen shot of the option you want from a drop-down list and Sikuli will select that option. That’s really cool but you say that Sikuli will even tolerate small changes in the icons you ask it to click. How does that work?

Tsung-Hsiang: Matching between a target image and the screen image is done by computing the normalized cross-correlation between the two images.

This is a standard technique in computer vision for finding patterns when variations are known to be small. This technique works incredibly well for matching desktop GUI patterns.

Matthew: Where do you think Sikuli will be most useful?

Tsung-Hsiang: Whenever the internal API of an application is not exposed. Lots of people have created their scripts to play the applications that were used to be very difficult to be automated, such as facebook (flash) games and testing android systems.

Matthew: Particularly on Mac OS X and Linux-based systems, where does Sikuli become a better option than standard shell scripting?

Tsung-Hsiang: If a command line interface is available, Sikuli may be not a good choice for a shell scripting guru. But sometimes you just can’t find command line tools or don’t want to learn complicated commands and parameters. In fact, the core of Sikuli is just a Jython library, so it can be mixed with other Python scripts or command line tools easily. Therefore, Sikuli can be an additional handy tool for command line gurus.

Besides, the primary goal of Sikuli is to help ordinary users who know nothing about command line tools and shell scripting to automate their tasks. We hope everyone can enjoy using computer efficiently.

Matthew: What’s next for Sikuli?

Tsung-Hsiang:Better, faster, more accurate.

We have a long list of planned improvements for Sikuli. Among the top of list are:

  1. Social programming: the ability to share scripts and search scripts by visual patterns. For example, when a user takes a screen shot of a recycle bin, the user can search a database for all the other scripts written by other users that involve the image of a recycle bin.
  2. Event-driven programming: the ability to register event handlers to handle visual events. For example, a user can define a function to pop up a warning message to handle the visual event that involves the appearance of the “low battery image”.
  3. Face detection: the ability to find faces on the screen.
  4. Recorder: the ability to record a sequence of clicking and typing operations and generate a visual script automatically.
  5. Tutorial converter: the ability to convert an existing step-by-step instruction with screenshots into executable scripts.

Matthew: Why did you choose Launchpad?

Tsung-Hsiang:We were using Trac before moving to Launchpad. Trac is more developer-oriented, just like Github.

But what we really want is a user-oriented project hosting site. We want a place to report and discuss bugs, ask and answer questions, and also download and track the development of Sikuli.

We compared Github and Launchpad, and at last chose Launchpad over Github because Launchpad has almost everything we need, except a wiki for writing documents. But we already had a wiki in our Trac system, so this was not a big problem.

Matthew: Do you have any requests for the Launchpad community?

Tsung-Hsiang: It would be great if we can write documents on Launchpad. A wiki or something like EtherPad would be fantastic.

Matthew: Thanks very much and good luck with Sikuli!

0

Launchpad blog round-up

Published by Matthew Revell February 15, 2010 in General

Just a quick note to link to some interesting Launchpad-related posts/articles that have cropped up over the past week or so:

If you haven’t seen it yet, take a look at Planet Launchpad.

0

Report on Launchpad down-time of 4th Feb 2010

Published by Matthew Revell February 5, 2010 in General

If you visited Launchpad between 13.30 and 15.30 UTC yesterday (4th Feb), you’ll have seen that Launchpad was largely unavailable.

Since then, I’ve spoken to quite a few people who use Launchpad regularly and I want to say thanks to everyone for your patience while we fixed the problem. As we all use Launchpad for our own development, we know just how painful unplanned down-time is and we’re sorry for the disruption to your work.

I’d like to explain what happened, how we fixed the problem and what we’re doing to avoid a similar situation in future.

As you’d probably expect, we run more than one database server for Launchpad. There are two master databases and then slaves, which are copies of the masters. The master databases replicate constantly to the slaves.

When Launchpad makes a read-only request, such as fetching the title and description of a bug report, we can reduce the load on the master databases by fetching that data from one of the slaves. However, to ensure the data you see is up to date, each time Launchpad is about to fetch data from the slave database, it checks how long it has been since the last replication from the relevant master database. If, for whatever reason, the replication wasn’t recent enough, Launchpad will instead grab the data from the master database.

Yesterday, it was this check that was taking far longer than expected and so causing the problems that you may have seen. We were able to implement a temporary fix, to bring Launchpad back online, by directing all database queries straight to the correct master.

In the longer-term, we’re going to overhaul the way that Launchpad checks the freshness of the data in the slave databases. Rather than checking each time a query is made, Launchpad will check once every so often and cache the result, meaning that this problem shouldn’t arise again.

Thanks again for your patience.

Previous Entries
Next Entries