launchpadblog

Some bugs attract many, many comments.

For a while now, Launchpad has displayed only the first 80 comments on any bug report, with the option of viewing the full comment history. That’s been good for speeding up page loads but not so great at offering an accurate view of the current state of discussion about the bug.

Bryce has fixed that. Now, a bug report page still shows only 80 comments, by default. However, to give a better overview of the state of discussion, it now shows the first 40 and the last 40 comments.

So, half way down the comments you’d see something like:

Here's what you see in the middle of the bug comment history

Then, at the bottom of the summarised comment history there’s this:

Comment history summary

With any luck, this should result in new bug comments that take into account the most recent discussion.

One of my favourite things about Launchpad’s bug tracker is the dupe finder: when you report a new bug, it’ll search to see if there’s already a similar bug report. It’s the same for questions in Launchpad Answers, too.

Getting to see possible dupes before you file a bug or question is a great time saver for you and the people on the other end. However, the dupe finder has been timing out a lot lately.

Rob Collins, Launchpad’s new Technical Architect, has introduced some changes that should make the dupe finder more reliable.

Other than fewer timeouts, here’s what you might notice:

• the dupe finder now returns fewer matches — three or four rather than ten or more
• the results should be more relevant.

We want to know how this works in practice. Let us know how you get on with the new dupe finder. Either leave a comment here, mail feedback@launchpad.net or join us on the launchpad-users mailing list.

How Rob did it

The previous dupe finder had a number of problems, not least that the search engine it’s built on is less efficient than we need. We’re planning to replace the search engine but not straight away, so Rob looked for a temporary solution that would work for the next five or six months.

I’ll hand over to Rob to explain what he actually did:

The old search did a pre-pass over every possible hit, which is 400,000 items for Ubuntu bugs and very slow to do. It then did a search matching any document that had a rare search term in it.

So, by rare we mean that the term showed up in less than half of the possible hits.

For example, if you searched for “firefox crashes on <website> in flash” on /ubuntu/+filebug it would search for any bug with any of “firefox” (< 50% of bugs are on firefox), "crash" (<50% of bugs say "crash"), "<can switch this off easily if we have to, so we do want feedback about how people find this.

Improvements, you say?

Very recently we saw the beta release of a new feature on Launchpad: building packages from recipes. Recipes bring Bazaar branches for the upstream source code and a packaging branch together to generate a Debian source package. We informally referred to this internally as “the Wellington project” because we first embarked on this long road of development back in November 2009 with a coding sprint in Wellington, New Zealand.

So, why do we need to use the build farm for this?

Very early on we realised that this would be a challenging project because packaging recipes allow untrusted arbitrary code to run as part of building the package, so we decided that the only place we could do this operation was in the same place that we do one of the other untrusted operations – the PPA build machines.

The PPA build machines work for untrusted code because they run as a virtual machine. After the build job has finished, the machine is simply ripped out and restarted so if any code did something nasty, we throw all the nastiness away!

Why can’t the build farm do this already?

The Launchpad build farm has been around for a while now, but it’s of course only used for building source packages into binary debs. We realised that we’d need a whole new data model on the Launchpad side, changes to the master/slave protocol and changes in the slave code to make it actually run this new job type – a lot of work! It was also about this time I became aware of a further untrusted job type, generation of translation templates for Launchpad Translations, so we began work on making the build farm able to process any kind of job.

What had to change?

By far the biggest change for this was to fix how we store the build/job information in the database. The canonical way to refer to a “job” on the build farm was via “build” record, and this is what all of the history pages in the UI were using. We’ve now re-architected this to use a generic “build farm job” and a bunch of new database tables that re-factor all the information needed for various jobs on the build farm, in fact we found out that the recipe build jobs and the source package build jobs had quite a lot in common which resulted in some merciless code re-factoring indeed.

The other major change was the slave builder code itself. This runs as a Twisted executive that we talk to using XMLRPC from the build manager, which kicks off external processes to run the job and monitors them. The only job it knew about was to run sbuild to build source packages. We’ve changed this to also run bzr builder and a custom script to generate the translations templates.

So, will I see anything different?

A recipe build in action

You won’t see much visually that’s changed in the Build Farm. If you’re observant, you’ll see the occasional job appear in the list that announces it’s a recipe build, but they’re generally pretty quick to complete!

We’ll post a more detailed explanation of how to use recipe builds in the future, but for now it’s remaining in beta until all the problems are ironed out. If you have any feedback, please file a bug or send us an email.

You can now use SFTP to upload source packages to your Personal Package Archive!

If you’re already familiar with uploading to a PPA, all you need to do is ensure your dput.cf includes the following:


method = sftp
login = <your Launchpad account name>


If you’re new to PPAs, but already know how to create packages for Ubuntu, take a look at our guide.

Via Planet Launchpad: Launchpad’s Strategist, Jonathan Lange, writes on his blog:

We’re starting to get a picture of what we want to do in the second half of 2010. In particular, lots of people within the team and within Canonical are starting to get fed up with our privacy and permissions model, which is quite patchy. It’s currently tangled up with the way we send emails to people, and we’d love to untangle them.

The Foundations team are already at work making Launchpad faster, and that’s something we want work on even more in the coming months. Derivative distributions – that is, a Linux distribution that extends or customizes another one, generally Ubuntu – have always been a key part of the vision for Launchpad, and they are finally going to get the effort they deserve.

Finally, we want to do whatever we can to make the Ubuntu Software Center rock harder than it does already. Launchpad occupies a special place in the Software Center world, since it can help make it easier to get applications for your desktop and make it easier to develop those applications.

If you’re keen to know what we have planned for Launchpad, you can subscribe to the roadmap page on our dev wiki.

Gavin just landed a very welcome branch to show the number of users affected by a bug on the bug page (bug 494040). You can also sort a list by the number of users affected, and doing this shows that two other popular bugs are related: provide a list of bugs affecting a user (283539) and show the number of affected users in a bug list (271332).

Needless to say, the bugs that affect the most users aren’t always the ones that should be fixed first. But it’s useful data, and more useful now that it’s easily available.

You’ll be able to use this after the release of Launchpad 3.1.12 on Wednesday the 16th December or on Edge right away.

For the last million years¹ or so I’ve been working on a cool new feature for Launchpad Bugs: an inline, AJAXified, asynchronous dupe finder.

For quite some time now people have encountered timeouts or long response times when trying to file bugs, particularly when they enter a long bug summary or the project that they’re filing the bugs on has a lot of bugs through which Launchpad has to search in order to be able to find possible duplicates. The upshot was that whenever a timeout occurred people were unable to file a bug and would have to back up and start again. Needless to say, this was frustrating for all involved.

The new inline dupefinder, which you’ll now find on the “Report a bug” page of any project in Launchpad (when viewed on edge.launchpad.net) is designed to stop this from being a problem, or at least to reduce the problem to a more manageable level and stop it from getting in peoples’ way. It does this in two ways:

1. The inline list of duplicates is much quicker to render than a full Launchpad page.
2. If the search for duplicates times out for some reason you’ll still be able to file a bug.

Here’s the catch: we need your help. Launchpad’s development cycle this month is very short due to the approaching year-end holiday period, so we need to get as much testing done on this as possible. Check out the dupe finder, see if it works for you and, most importantly, report a bug if it doesn’t.

One last thing: at the time of writing, the inline dupe finder only works for projects (like Launchpad Bugs), not for packages or project groups. We’ll hopefully be enabling it for project groups today and with a bit of luck for packages, too. We started off with projects only because it’s the simplest implementation of the concept and it gives us a good base to test from.

Thanks in advance for your help. Let’s make Launchpad awesome together!

¹ This might be an exaggeration.

If you’re using edge, you can now just comment on a question in Launchpad. For all questions on Answers, the “Just Add a Comment” button is now always visible.

Previously, you might have only seen “Add Answer” and “Add Information Request” (or others; the exact buttons vary), both of which add a comment and cause the question status to change. But often, for example, all you want to do is clarify an earlier comment, add some detail, or give a progress update. For that, “Just Add a Comment”.

It’s been put at the rightmost position of all the buttons because we think it should be the least used option. Normally it’s appropriate to use one of the other buttons to move the workflow forward.

The button will land in production with the 3.1.10 release next week.

Over the past few months we’ve been slowly introducing a grand new feature in Launchpad Translations, and with 2.2.7, we are finally ready to announce it to the wide world: message sharing.

Introducing message sharing

Message sharing between different releases of a product or distribution in Launchpad means that translations done in one release (eg. trunk) would immediately apply to translations in another release (eg. stable). This should benefit everyone using Launchpad for translations, one way or another.

• Translators will not have to worry about back-porting translation fixes to older releases anymore, and they can simply translate the latest release: translations will automatically propagate to older releases. Also, this works both ways, so if you are translating a current stable release, newer development release will get those updates too!
• Project maintainers hosting their translations in Launchpad, when they upload a template to a new release series and it gets imported, will instantly get all existing translations from previous releases shared with the new series and translators won’t have to re-do their work. They won’t have to worry about uploading correct versions of translated PO files, and can just care about POT files instead.
• For Ubuntu, there’s another benefit: opening a new release for translations will take minutes instead of days. Message sharing also improves the scalability of the system, and we should soon start seeing more performance improvements as the result of migrating to this new way of managing translations.

How to benefit from message sharing?

Over the next few weeks, we’ll be migrating all existing projects to make the best use of message sharing. If there are any potential problems with existing projects, we’ll be emailing their maintainers and resolving it as we go along. Messages will be shared only between templates in one particular project, and only if they all have the same name. For Ubuntu, they will be shared only between templates in one particular source package, and again, only if they are of the same name.

For example, Ubuntu Jaunty and Ubuntu Karmic already share their messages. If you update a translation for one single string in Karmic GNOME Panel, and that same string exists in Jaunty GNOME Panel, Jaunty translation will be instantly and automatically updated to exactly the same translation.

If you want one particular release to have a different translation from all the other releases, that’s still possible. So, if you do not want a translation in Jaunty to be modified when a translation is changed in Karmic, you have that option too. Since we don’t expect this option to be used often, it’s hidden in the “zoomed in” view on each single message.

What next?

We’ll have to carry out a data migration for each of the projects using Launchpad Translations. Luckily, this migration will be completely transparent, so there’s nothing for anyone else to do. On the outside, nothing will change, until you start doing translations for more than one release series, when they’ll just be better.

New projects will get benefits of message sharing right away, and existing ones should be a bit patient because doing the migration takes some time, but allows to keep the full history of translations intact.

If you are interested to learn more, come talk to us on #launchpad on FreeNode, or join the launchpad-users mailing list.

Launchpad has a little known feature that is getting better. The product release finder is a process that runs everyday to locate new releases and import them to Launchpad. The process uses the series’ Release file pattern to locate files and import them to the appropriate release. It can even create releases for series. The process is robust and worth consideration if you want to upload large release files for your project.

The project owner and series release manager can set the Release URL pattern by the series edit page. The pattern is an ftp, http, or https URL with a glob (*) in the part of the file name that varies with each release for a series. For example:

http://widgets.dom/downloads/widget-2.*

describes all files that start with ‘widget-2.’. This might be the source for two different releases, widget-2.1.tar.gz and widget-2.2.3.tar.gz. The pattern will also match multiple files that belong to a single release, such as widget-2.1.tar.gz, widget-2.1.zip, widget-2.1.changelog.

Many projects choose to group files in series in a directory of their own, in which case the Release URL pattern would look something like:

http://widgets.dom/downloads/2.8/*

You can tell the product release finder to search multiple directories by using a glob for a directory. For example, if your project separates release files in directories for each OS then you can try

http://widgets.dom/downloads/*/widget-2.*

to scan downloads/ubuntu/widget-2.* and downloads/mac/widget-2.*.

Be careful to include the common part of the series in the URL, otherwise files from different series will be imported to the wrong series. Do not do something like:

http://widgets.dom/all-releases/*

because any file that looks like it has version information in it will be imported to one series.

In all cases, the product release finder will extract the version from the file name, and match it to a milestone name. It will create the milestone and release it if necessary. If a version cannot be extracted, the file is ignored.

The version numbers extracted from file names conform to Launchpad URL name rules. So if your release files have underscores or pluses in their version names, dashes will be substituted. Flavour information is also ignored in the file name. For example these file names yield these versions:


emacs-21.10.tar.gz => 21.10
vpnc-0.2-rm+zomb-pre1.tar.gz => 0.2-rm-zomb-pre1
warzone2100-2.0.5_rc1.tar.bz2 => 2.0.5-rc1
furiusisomount-0.8.1.0_de_DE.tar.gz => 0.8.1.0
glow-0.2.1_i386.deb => 0.2.1
Bazaar-1.16.1.win32-py2.5.exe => 1.16.1


What’s Next

The product release finder should notify owners and release managers when there are problems with imports. A lot of problems were fixed recently, but there are two issues we are still seeing in the logs that indicate the Release url pattern must be updated for some projects. The product release finder cannot access the server or directory in some of the URLs. There are also a few URLs that have no glob. They appear to be the URL to a single file, where a glob is needed so that the series can have many releases. If the product release finder does not find your files after a few days, review the Release url pattern and check the remote site to verify they are fine.

We will update the UI to make the Release url pattern more prominent, and easier to set for each series.