Allenap on Launchpad Blog

Launchpad's Bug Watch system and other animals

Gavin Panella — Fri, 19 Mar 2010 14:15:31 +0000

Launchpad has a feature where it periodically checks the status of remote bugs (as in, bugs recorded in another bug tracker, like bug 12720 in Django).

When someone links a bug on Launchpad with a remote bug it’s called a bug watch. All the bug watches for a bug appear in the Launchpad bug page in an area called “Remote bug watches”. Check out bug 513719 to see a bug watch for bug 12720 in Django.

If the remote bug tracker has been set as the bug tracker for a project in Launchpad, bug tasks for that project can be linked to a specific remote bug too. When the status of the remote bug changes, Launchpad changes the status of the bug task to match, and sends out email to subscribers, the same as if the status had been changed in Launchpad. See the Django bug task in bug 513719 for an example.

Going further, comments can be synchronized too, in both directions. Recent versions of Bugzilla have this capability built in, but older versions can be supported with a plugin. There’s also a plugin for Trac.

This is all very nifty stuff, but it suffers because it doesn’t work very well! Yet.

Part of this is down to complexity

We support 7 different remote bug tracker types: Debian Bugs, Trac, Bugzilla, Mantis, RT (Request Tracker), Roundup and SourceForge.

We try to support a range of versions for each of these trackers, and a range of different access methods.

For example, with Bugzilla, we support old (v2) installations, more recent (v3) ones, recent ones with the Launchpad plugin, and very recent ones with the API built-in. And we support Issuezilla, a variant of Bugzilla.

We try to work around many idiosyncrasies and customizations in the remote systems.

With Mantis, for example, sometimes we need to log in anonymously before we can download bug statuses. Some Mantis installations allow us to download status information in CSV form, in a batch, but not all. If not, we screen-scrape the individual bug pages. Even if we get CSV, it’s often slightly corrupt, so we try and correct for that where we can.

Then there are simply hundreds of things that are beyond our control that Launchpad must cope with and move on, like HTTP errors, errors without correct HTTP codes, slow responses, unrecognized responses, and so forth. We check about 7k remote bugs every day (we aim to do more than 30k a day, but we’re not there yet) so there are a huge number of errors to sift through. It’s a big task to figure out if each problem is transient, a problem in Launchpad, or a remote problem, never mind actually fixing it!

We must also be gentle with the remote systems, and not issue too many requests in a short time, or otherwise make unreasonably large demands on them. We must be nice.

Part of it is down to the development of the code base

The bug watch code in Launchpad is a bit creaky. Originally it was designed to run in a single thread in a single process. This means that one badly performing bug tracker can starve the whole system of updates.

In the past we’ve tried to alleviate this by choosing the bug watches to check more carefully, or by batching requests, but this has not been enough for a some time.

Also, while we do record a lot of information about errors, it is still a Herculean task to constantly monitor the errors coming out of the system.

We also can’t test or do QA against other people’s systems. For testing we must use test doubles to simulate the behaviour of remote systems, but this can get confusing. For QA, sometimes we must test against remote systems. This is acceptable for testing the status fetching code, but not for comment synchronization.

However, we have not had the time available to give the bug watch system a big overhaul, only to make small improvements and bug fixes. For a long time it’s needed some big changes to make it work with the volumes of work it’s expected to cope with.

Fixing it

We’re trying to fix these issues now.

We’ve made checkwatches - the program that drives the bug watch machinery - run multi-threaded. This works, but it hammers our database, so we need to figure out how to alleviate that next.

We’ve started the work to move the code base over to using Twisted. This is a better model for managing a lot of concurrent network activity. As more and more bug watches are registered with Launchpad, we’re going to need it.

We’re going to keep more history of success and failure in the database, so that checkwatches can throttle back checks for remote bugs that persistently error. This is really important because it will reduce the work that checkwatches needs to do each day. It will also, we hope, reduce the deluge of errors to a more manageable stream, so that we’re better able to spot genuine bugs that need our attention. Lastly, we hope to display this information on the web pages so that users can help to diagnose problems themselves.

We’re getting some more infrastructure in place, so that we can develop and QA against real installations of Bugzilla, Trac, et al.

We’re also fixing up as many bugs in checkwatches as we can.

Who are we?

Abel Deuring, Graham Binns and Gavin Panella. We’re meeting up next week in Norwich, England for a sprint, a culmination of a couple of months of dedicated effort on the bug watch system.

Deryck Hodge is also coordinating with the Canonical IS team to deliver the QA infrastructure.

For all of us, the aim is to get checkwatches to work reliably, even if it’s not the most efficient or elegant system inside. We also want future development to be more sustainable than it has been in the past.

We want users to be able to rely on this feature.

Commenting on questions

Gavin Panella — Fri, 30 Oct 2009 15:58:06 +0000

If you’re using edge, you can now just comment on a question in Launchpad. For all questions on Answers, the “Just Add a Comment” button is now always visible.

Previously, you might have only seen “Add Answer” and “Add Information Request” (or others; the exact buttons vary), both of which add a comment and cause the question status to change. But often, for example, all you want to do is clarify an earlier comment, add some detail, or give a progress update. For that, “Just Add a Comment”.

It’s been put at the rightmost position of all the buttons because we think it should be the least used option. Normally it’s appropriate to use one of the other buttons to move the workflow forward.

The button will land in production with the 3.1.10 release next week.

Extra options when filing bugs

Gavin Panella — Fri, 26 Jun 2009 09:07:59 +0000

You may have noticed that we introduced some useful new options when filing bugs. These are especially useful to anyone who files lots of bugs.

Something for everyone.

Everyone can now set tags when filing a bug. Previously, only people going via the advanced bug filing page would have the option.

One caveat: the tag field is not yet wired up with the magical tag auto-completer that you can use on the bug page itself, but that’s coming.

Something for bug supervisors.

If you’re filing a bug against a project for which you’re a bug supervisor, some additional fields appear in the new Extra options area (which is collapsed by default, but can be expanded by clicking on it). There are options to let you to set the initial status, the importance and the milestone of the bug, and also assign it directly to someone to work on.

If you have any problems with these new features, please file bugs against Launchpad Bugs, or talk to the help contact in #launchpad on freenode.

Searching bugs with tags now with wings!

Gavin Panella — Wed, 24 Jun 2009 09:30:06 +0000

Even if you’ve been living on Earth, you could be forgiven for not knowing about the richer tag syntax in the advanced bug search (e.g. in Bugs in Ubuntu: Advanced search), frankly because it’s very new and it’s not mentioned anywhere in the UI (yet). There are two additions to the syntax that a handful of hardcore Launchpadders have been yearning after for some time.

First up, you can search for bugs with any tag, doesn’t matter which, and for bugs with no tags. To search for the presence of one or more tags, use “*” (asterisk) in the query, and to search for the complete absence of tags, use “-*” (minus asterisk).

Secondly, you can search for the absence of a specific tag. Simply prefix the tag with a minus, e.g. “-toaster”.

You can combine these new forms as well. For example, to search for bugs with no tags at all or with no crumpet tag, you could search for “-* -crumpet”, making sure the Any radio button is selected. Everybody needs crumpets!

Have fun!

If you run into any problems, please report the bug in Malone.