Gmb on Launchpad Blog

Fixing a simple bug in Launchpad - A screencast

Graham Binns — Wed, 20 Jun 2012 16:06:27 +0000

When I ran the Launchpad Development Clinics at the last UDS, I was asked if I would give a presentation on how to fix a simple bug in Launchpad. This sounded like a great idea - after all there’s a lot of parts to the Launchpad development process that apply to every single branch you ever write, so it seemed well worth the effort. Rather than a presentation, though, I figured a screencast would be of more use, so here is just such a screencast. If you’ve got any questions or comments, don’t hesitate to get in touch.

A story of a clinic

Graham Binns — Fri, 01 Jun 2012 10:27:38 +0000

You’ll remember that a while back, dear reader, we announced that we’d be running a couple of Launchpad Development Clinics at UDS (we called them Launchpad Clinics at the time, and I lost count of how many people commented that that sounded as though Launchpad was ill. It isn’t; it’s in the same rude health that it always was). We’ll come up with a better name next year!

Anyway, we made the announcement, put up the wiki page, saw a few names and bugs added to it, and didn’t really expect to be hugely busy. Indeed, when Laura, Matthew, Huw, Raphaël and I convened in one of the UDS meeting rooms for the first clinic, it was mostly empty except for the stragglers from the previous session.

Then a person appeared. Actually, it was two people squeezed into the skin of one person: Tim Penhey, who looks like he’s been chiseled out of granite and has a tendency to loom well-meaningly, like an avuncular greek pillar. We figured that since he’s an ex-Launchpadder, he didn’t count, and so went back to self-deprecatory joking whilst worrying that we’d massively miscalculated how many people would want to come to the clinics.

And then another person arrived. And another, and another. Soon, we’d gone from having a couple of attendees who really just needed to run ideas past a Launchpad core developer to having ten people who all needed questions answered, or a development instance spun up. After that, things get a bit fuzzy, because I was always answering someone’s question or being root on an EC2 instance for someone else. When Laura told us that our time was up I was, I have to confess, somewhat surprised.

Thursday’s session was a quieter affair, in part at least because we had to reschedule it at the last minute (UDS schedules are like quicksand, and if it weren’t for the amazing UDS admin team everyone would be thoroughly lost for much of the time), but there were still people there with bugs to be fixed and questions to be answered. I had preliminary discussions with Chris Johnston about adding API support to Blueprints, and worked with Ursula Junque on working out how to add activity logging to the same.

The upshot of the clinics is, I think, massively positive. There is a genuine development community out there for Launchpad, and people really are keen to make changes to the dear old Beast whilst the Launchpad core developers are working elsewhere or fixing things that are horribly complex (and usually not user-facing). For someone like me, who had been somewhat skeptical about the kind of response the clinics would receive (even though they were partly my idea), this is immensely gratifying news.

There are, of course, many things that we need to improve on, and many lessons that we can learn. People want to know how to fix a simple bug without having to come to a session at UDS, so I’m going to record a screencast of just such a procedure, right from finding the bug to working out where the fix lives, all the way through the coding and testing process, right up to the point of getting the branch reviewed and landed. Hopefully this will give everyone a great jumping-off point.

When we set out on this particular journey, one of the criteria I wrote down for considering the clinics a success was “we’ll want to do it again at the next UDS.” Well, I do. We did well; we can and will do better next time. Who’s with me?

That Juju that you do (Part II: A magical balm to sooth your ills)

Graham Binns — Tue, 03 Apr 2012 10:16:42 +0000

In my previous post I talked about the pain of having to set up a testing environment for our parallelised test work, and how there were an awful lot of hoops to jump through in order to get something usable up and running. Now, dear reader, let me tell you a tale of strangeness and charms.

Enter Juju

If you’re not familiar with Juju, I’d urge you to pay a visit to the Juju website to learn more, but in brief, I’ll explain: Juju is an orchestration service for Ubuntu. Using Juju allows you to deploy services rapidly, scaling up or down as you need. Each service is contained within a Charm, which is at it simplest a set of scripts that ensure that a given Juju unit does what it’s supposed to do at the appointed time (for example: install and config_changed are two of the most common hook scripts for a charm to have). We realised that in order to make our life simpler when testing our parallelisation work we could develop a pair of Buildbot Charms (one for the master, one for the slave) which when deployed through Juju, and given the right set of configuration options, would give us a working Buildbot setup on which to test Launchpad.

More about the charms…

The charms need to be able to automatically configure themselves to talk to each other (this is usually managed in Buildbot by static configuration files). Luckily, Juju provides for exactly that situation with the notion of “relations”; one charm can declare that it provides a particular interface as part of a relation and another can say that it requires that interface in order to be able to be a part of that relation. For our Buildbot charms, we have the following in the master’s metadata.yaml:

provides:
  buildbot:
    interface: master

And in the slave:

provides:
  buildbot:
    interface: slave
requires:
  buildbot:
    interface: master

Each charm has a couple of hooks that deal with relation, named in the form buildbot-relation-* where * is joined, changed or broken. These are run by Juju at the appropriate point in the process of connecting one instance to another. With all this in place, then, we can set up a working Buildbot environment by doing something like this:

$ juju bootstrap # create the Juju environment
$ juju deploy buildbot-master —config=/path/to/master/config.yaml # deploy the master charm
$ juju deploy buildbot-slave —config=/path/to/slave/config.yaml # deploy the slave charm
$ juju add-relation buildbot-slave buildbot-master

The last line - juju add-relation buildbot-slave buildbot-master- tells Juju to connect the buildbot slave node to the master node. The two then do a bit of a dance to configure each other properly (in fact, it’s mostly a case of the slave saying: “Hey, I’m here, what do you want me to do?” and the master passing back configuration instructions). Once this is all done, you have a working Buildbot master and slave, ready to accept work to build.

What have we discovered about Juju?

First and foremost, we’ve learned just how powerful Juju actually is. We’ve taken a fairly complex-to-configure build environment, for which we normally use dedicated machinery whose configuration is not to be touched without sysadmin blessing on pain of pain, and turned it into something that we can deploy with four or five commands and a couple of configuration files. Sure, Juju has its quirks and oddnesses, but when we’ve run across them the Juju development team has been amazingly helpful with workarounds or, more usually, bug fixes. The current version of Juju is implemented in Python, too, so we find it pretty easy to contribute fixes of our own if we need to.

Where can I find out more?

As I said above, if you want to know more about Juju, you can check out the Juju website. If you want to take a look at our Buildbot charms and how we’ve built our hooks (they’re written in Python because that happens to be our language of choice, but in fact they can be written in anything so long as they’re executable), you can grab our code from Launchpad:

For the master: bzr branch lp:~yellow/charms/oneiric/buildbot-master/trunk buildbot-master
For the slave: bzr branch lp:~yellow/charms/oneiric/buildbot-slave/trunk buildbot-slave

If you’ve got questions about Juju in general, the folks in #juju on Freenode are always tremendously helpful. If you’ve got any questions about our charms, ask them in the comments here and I’ll do my best to answer them.

((Image by http://www.samcatchesides.com/ under a Creative Commons license)

That Juju that you do (Part I: Bring the pain)

Graham Binns — Fri, 30 Mar 2012 14:44:28 +0000

Benji’s blog post earlier this week gave you all some insight into what the Launchpad Yellow Squad has been doing recently in its attempt to parallelise the Launchpad test suite. One of the side effects of this is that we’ve been making quite a lot of use of Juju, and we thought it’d be nice to actually spell out what we’ve been doing.

The problem

We’re working to parallelise Launchpad’s test suite so that it doesn’t take approximately one epoch to get a branch from being approved for merging until it lands. A lofty goal, sure, and one that presents some interesting problems from the perspective of building an environment to test our work in. You see, Launchpad’s build infrastructure is a pretty complicated beast. It’s come a long way since the time when submitting a branch for merging meant sending an email to our PQM bot, which would then run the test suite and kick the branch out if it failed, but now it’s something of a behemoth.

Time for some S&M

We use Buildbot as our continuous integration system. There are two parts to Buildbot: the master and the slave. Broadly put, the slave is the part of Buildbot that is responsible for doing the actual work of compilation and running tests and the master is responsible for telling the slave when to do things. Each master can be responsible for several slaves. When it became obvious that we were going to need to essentially replicate our existing setup in order to test our parallelisation work, we considered asking Canonicals system administrators, in our sweetest tones, to give us a box upon which to do our testing work, but we spotted two reasons that this would be problematic:

We didn’t actually know at the outset what the best architecture was for our project.
Asking for a machine without knowing what you actually need is likely to earn you a look so old it could have come from an ammonite, at least if you have sensible sysadmins.

So instead, the obvious solution: use Amazon EC2. After all, that would allow us to play with different architectures without there being any huge cost in terms of physical resources. Moreover, we’d be able to have root access on the instances on which we were testing, which makes debugging such a complicated process so much easier.

However…

There was still a problem. How to actually set up the test instances, given that there are five of us spread between three timezones, that it takes a significant amount of time to set up a machine for Launchpad development, and finally that we don’t really want to leave EC2 instances running overnight if we don’t have to (because it’s expensive).

The sequence of steps we’d have to take to up an instance tends to look something like this:

Launch a new EC2 instance (this happens pretty quickly, thanks, Amazon)
Make sure that everyone’s public SSH keys are usable on that instance
Run our Launchpad setup script(s) (this takes about an hour, usually).
Install buildbot.
Configure buildbot correctly as master or slave.
Run buildbot (or buildslave, if this is a slave) and make sure it’s hooked up correctly to the other type of buildbot.
Get some code into buildbot and make it run the test suite.

As you can see, this is pretty long-winded and rather fragile; it’s very easy for us to miss out a step or misconfigure something, get confused and then be left with a broken instance and a bit of a headache. Now, you’d be quite right to argue that we could just write a checklist - or better yet, a shell script - to do a lot of the setup work for us. A good idea, true. But there’s a better way…

To be continued (or some other phrase that doesn’t sound so hammy that it almost goes “oink”)…

(Image by http://www.samcatchesides.com/ under a Creative Commons license)

You can't be too helpful: Better handling of large blobs by +filebug

Graham Binns — Thu, 04 Mar 2010 07:35:26 +0000

Imagine the scene, dear reader. You’re running the latest Ubuntu pre-release, dilligently testing that everything works the way it ought. An application crashes horribly; Apport pops up to tell you that it spotted the crash - would you like to report a bug?

“Of course I would,” you cry, “I am a desktop testing hero!” And so Apport does its thing and takes you to Launchpad to file the bug report. And then Launchpad times out.

At this point, if you’re like me, you might shout a bit. You refresh and refresh with all your might but still Launchpad will only give you an error page. Finally, defeated, broken, you close your browser and shuffle off into the corner of your room, where you bury yourself under the mountain of discarded CD-Rs that contain daily ISOs of Ubuntus past and sob into your coffee.

Okay, so maybe I’ve exaggerated it a bit, but that doesn’t change the basic fact that timeouts on the +filebug page when you’re filing a bug via Apport are intensely, soul-destroyingly annoying. They’re annoying to us in the Launchpad Bugs team too, because we see them in our error reports. They’re so annoying, dear reader, that we’re moved to hyperbole when writing about them for the official Launchpad blog.

The problem with processing data supplied by Apport has always been one of scale. Originally the extra data that Apport would upload to Launchpad wouldn’t be massive, maybe hitting 10MB if the application was particularly busy. Because of this, Launchpad simply processed those data synchronously whilst loading the bug-filing form for you, so that the data you’d uploaded would be included with the bug.

Of course, that approach doesn’t scale very well, and recently we’ve been seeing the data blobs that Apport uploads hit ~100MB in size. That’s far too big for Launchpad to handle in a timely manner whilst doing all the additional work of rendering the bug filing form for you, so in those circumstances it invariably times out and you get that ever-annoying error page.

This was an interesting problem to solve, and we came up with a number of different possible solutions. One viable solution for this, which we eventually decided not to implement, involved having a separate request for processing the extra bug data and then loading the data into the filebug form with AJAX. We discarded that idea because there was always the chance that that request would time out, leaving us in an only slightly better position than we were already.

There was some discussion on the Launchpad developers mailing list about whether we could just defer loading the extra data into the bug until after it had been filed, but we quickly realised that not only could the extra data carry an initial subject for the bug, but it could also indicate that the bug should be filed as private, something which currently can’t be done via the Launchpad web UI.

The solution we chose to implement, which we’ve now rolled out to the Launchpad edge and production servers, is to have a queue of blobs waiting to be processed. We already have quite a robust job-processing system built into the Launchpad codebase, which we use for creating the diffs in merge proposals and calculating bug heat, amongst other things. Adding support for processing uploaded blobs was quite simple, since the existing blob parsing code was well documented. The blob processing jobs are picked up and run by a cron script that runs every minute or so, and the data retrieved from the blobs are stored with the original processing job as a JSON-formatted string. When you hit the +filebug page it checks to see if the relevant blob has been processed. If not, it waits until processing is complete. Once processing is complete the serialized data are loaded into the +filebug page for use later on.

The advantage to you as a user, then, is that you should never again see a timeout on the +filebug page due to the size of the blob that Apport has uploaded. With the upcoming release of Ubuntu Lucid, we’re hopeful that this will make a real difference to those people testing the pre-release versions of the distro.

Inline dupe-finding: an exercise in pain reduction

Graham Binns — Thu, 10 Dec 2009 09:35:06 +0000

For the last million years¹ or so I’ve been working on a cool new feature for Launchpad Bugs: an inline, AJAXified, asynchronous dupe finder.

For quite some time now people have encountered timeouts or long response times when trying to file bugs, particularly when they enter a long bug summary or the project that they’re filing the bugs on has a lot of bugs through which Launchpad has to search in order to be able to find possible duplicates. The upshot was that whenever a timeout occurred people were unable to file a bug and would have to back up and start again. Needless to say, this was frustrating for all involved.

The new inline dupefinder, which you’ll now find on the “Report a bug” page of any project in Launchpad (when viewed on edge.launchpad.net) is designed to stop this from being a problem, or at least to reduce the problem to a more manageable level and stop it from getting in peoples’ way. It does this in two ways:

The inline list of duplicates is much quicker to render than a full Launchpad page.
If the search for duplicates times out for some reason you’ll still be able to file a bug.

Here’s the catch: we need your help. Launchpad’s development cycle this month is very short due to the approaching year-end holiday period, so we need to get as much testing done on this as possible. Check out the dupe finder, see if it works for you and, most importantly, report a bug if it doesn’t.

One last thing: at the time of writing, the inline dupe finder only works for projects (like Launchpad Bugs), not for packages or project groups. We’ll hopefully be enabling it for project groups today and with a bit of luck for packages, too. We started off with projects only because it’s the simplest implementation of the concept and it gives us a good base to test from.

Thanks in advance for your help. Let’s make Launchpad awesome together!

¹ This might be an exaggeration.

Making bug migration that bit easier

Graham Binns — Wed, 17 Jun 2009 10:41:01 +0000

Greetings, readers!

For a while now I’ve been the Launchpad developer in charge of helping users of other bug trackers migrate their projects to Launchpad. We do this reasonably regularly for users of Trac, SourceForge and Bugzilla, and we’re always open to helping people migrate from other solutions should they wish to.

Several months back, the Elisa project migrated from Trac to Launchpad. In doing so, they wrote a script to take their Trac database and convert it to the Launchpad bug interchange format, which is an XML schema that we use for bug imports (you give us an XML document containing your bugs that conforms to the interchange format schema and we can then import it into Launchpad for you).

More recently, we realised that we were getting quite a few requests from Trac users for help with migration. We went back to the Elisa developers and asked if they’d be willing to let us have their migration tool under the GPL so that we could share it with everyone. They kindly said yes, passed the code to us under GPLv3, and yesterday I made it available on Launchpad as the Trac to Launchpad Migrator project.

It’s a bit rough around the edges at the moment and needs some documentation, but more than anything else it needs people to test it and use it so that we can make migrating from Trac to Launchpad as smooth as possible.

Update: Matthew’s written a blog post about our bug import format.

Links to external bug trackers right where you need them

Graham Binns — Thu, 26 Feb 2009 00:25:12 +0000

Linking Launchpad bugs to upstream bugs can be hard work. You don’t necessarily know whether the bug is reported on the upstream tracker, and if you don’t know where the upstream project tracks its bugs you have to go and find out before you can file the bug and link the Launchpad bug to it.

We understand that this process can be a bit of a pain, so we’re introducing a feature that should make linking to upstream bugs a bit easier: Launchpad will now give you direct links to the bug search and filing forms in a project’s external tracker, so long as Launchpad knows the location of the project’s tracker and, if the tracker tracks more than one project, which project on the external tracker the Launchpad project refers to (so, for example, Launchpad needs to know that the Launchpad project ’evolution’ links to the ‘Evolution’ product in the Gnome Bugzilla).

And because we know that a project maintainer shouldn’t have to go back to Launchpad and add the link to the remote project when Launchpad should be able to work it out for itself we’ve fixed that problem, too: Launchpad will check all the projects that are linked to a remote bug tracker and will try to deduce which remote project they’re linked to by looking at the bug watches that are registered against that bug tracker. Of course, this isn’t an exact science, and we’ll have to correct some of these auto-generated links manually, but hopefully it won’t take more than a few days for us to straighten out the kinks.

In order to get to remote bug tracker links, all you have to do is click Also affects project on the bug report and select the project that you want to link to. Launchpad will then offer you the links.

We’re going to be working to improve this further in the coming Launchpad development cycle, so keep your eyes on this blog for further information.

Announcing the Launchpad bug plugin API spec

Graham Binns — Thu, 04 Dec 2008 16:12:08 +0000

A while ago we announced the beta versions of the Launchpad plugins for Trac and Bugzilla. These allow Launchpad to sync not just bug statuses but also comments and, eventually, whole bug reports with the remote bug trackers that have them installed, bi-directionally. The practical upshot of this is that upstream projects that don’t use Launchpad for bug tracking can install one of these plugins and never again have to worry about not seeing bugs that people file using Launchpad but don’t forward upstream.

We in the Launchpad team realise, of course, that Bugzilla and Trac aren’t the only two bug trackers in the world. A project that uses Mantis, for example, may want its bug tracker to interact with Launchpad in the same way that Bugzilla and Trac now can. Unfortunately we can’t develop a plugin for every bug tracker out there as we don’t have the resources or the in-depth knowledge of those trackers’ internals necessary to be able to do so.

To make it possible for anyone to develop a plugin for any bug tracker, we’ve now opened up our Plugin API spec. The spec details all the APIs that need to be implemented for Launchpad to be able to sync bi-directionally with a remote bug tracker. It’s pretty detailed, and we’ve rolled into it our experiences in developing the Bugzilla and Trac plugins so as to make sure that new plugins don’t repeat the issues that we came across when we started interacting with the first bug trackers to install those plugins.

Once you’ve written an API plugin for your bug tracker of choice you can contact us by filing a question against Launchpad Bugs to let us know about it. We’ll then work to write the code on the Launchpad side that will interact with the API you’ve written. Once that’s done we’ll continue to work with you to iron out any bugs that we may come across.

I’m looking forward to seeing your plugins!

Bugzilla and Trac plugins now in beta

Graham Binns — Wed, 20 Aug 2008 23:59:00 +0000

A few weeks ago, Matt announced the new Launchpad plugins for Bugzilla and Trac.

The plugins allow bidirectional communication between Launchpad and the remote bug trackers that have them installed. Obviously, we need to test the plugins - and that’s where we need your help.

If you know of any Trac or Bugzilla instances whose administrators might be interested in installing the requisite Launchpad plugin - or indeed if you run such a bug tracker yourself - you can find details on how to install the plugins and what you need to do to get Launchpad to work with your bugtracker on the Launchpad Help wiki:

For Bugzilla, see http://help.launchpad.net/Bugs/BugzillaPlugin
For Trac, see http://help.launchpad.net/Bugs/TracPlugin

So, what will installing the plugins do? Well, initially, installing one of the plugins will mean that:

Launchpad will be able to import comments from upstream bugs which are being watched in Launchpad. If you register a bug watch against an upstream bug you’ll be able to read the conversation about that bug on Launchpad, including the comments that are made on the upstream bug tracker.
Launchpad will be able to push comments to upstream bugs which are linked to a Launchpad bug, so if you add a comment on Launchpad in reply to a comment which was imported from an upstream bug tracker, the comment will automatically be added to the conversation on the upstream bug tracker, too. This means that users of both Launchpad and the upstream tracker can view the whole conversation about the bug without leaving their preferred environments.

Once we’re happy that the plugins are working correctly we can use them to add some even cooler functionality to Launchpad:

Launchpad will be able to forward bugs to upstream bug trackers that have the plugin installed. Users of those bug trackers won’t have to check Launchpad for bugs registered against their project; they’ll be forwarded straight to the upstream tracker.
If a bug watch is created against an upstream bug, Launchpad will tell the upstream bug tracker which Launchpad bug is watching it. This means that not only is there a link in Launchpad to the upstream bug, but there’s also a link in the upstream bug tracker to the Launchpad bug.

The Bugzilla plugin is licensed under the Mozilla Public License and the Trac plugin is licensed under GPLv2.

If you’ve got any questions, don’t hesitate to contact us by email at feedback@launchpad.net.