launchpadblog

Launchpad has quite a few background tasks that update things outside of your browser request, and the only way you know that they finished is to reload a page to see if it changes.

Updating the preview diff on a merge proposal is one example.

At our Launchpad sprint in Dublin a couple of weeks ago, we added a so-called “long poll” where the browser opens a long-running connection using XHR (a background Javascript connection) to a new piece of infrastructure called the “long poll server”

This server is something that the Landscape team originally wrote and we’ve now open-sourced it so we can use it in Launchpad. The server connects to a RabbitMQ instance and relays information back to the browser when the background task completes so that the user doesn’t need to keep reloading the page – the page will automatically update thanks to the magic of Javascript.

The proof-of-concept is done but we now need to make it work in our production environment, which means making it more reliable, configuring the deployment and add some stats collection so we can see how it behaves. We’ll start off by only letting a small subset of people use it and grow it over time until we consider it stable enough for all users.

Photo by Jay Gooby. Licence: CC BY 2.0

Over the past few months, my team have been working to make it easier to create and manage derivative distributions in Launchpad.

Although most of this will be of interest only to distro admins and contributors, we recently released a beta of a new feature that lets you visualize the differences between Ubuntu’s Oneiric series and Debian Sid.

There’s a new portlet on the distroseries page that will list the numbers of packages in both distributions, numbers only in Sid and numbers only in Oneiric. Clicking on the links takes you to pages that show the differences in more detail and allows you to request debdiffs and add comments on the packages.

Please let me know what you think of this change. We’re tracking bugs using the ‘derivation’ bug tag. Note that this is not meant to be a replacement for MoM – yet. We have a long way to go for that.

Coming soon, actual syncing from Debian to Ubuntu. Watch this space!

Photo by Anton Lindqvist. Licence: CC BY 2.0.

Launchpad has an anonymous FTP server that people use to upload source packages to Ubuntu and their PPAs. When Launchpad processes the upload it does a huge number of checks, one of which is verifying the GPG signature on the upload’s .changes file.

One of the problems with this is that if there’s a problem with the signature, or there is no signature at all, Launchpad simply throws the upload away as it cannot be sure who uploaded the package. If it tried to send an email it would also quickly become a spam vector! (See bug 374019)

In today’s release, there is a brand new FTP server that will do preliminary GPG signature checks right in the FTP session itself. If you upload a package that is not signed properly, you’ll see a message that looks like this:

Uploading to launchpad (via ftp to ppa.launchpad.net):
Uploading hello_2.5-1ubuntu1.dsc: done.
Uploading hello_2.5-1ubuntu1.diff.gz: done.
Uploading hello_2.5-1ubuntu1_source.changes: 1k/2k550 (‘Changes file must be signed with a valid GPG signature: Verification failed 3 times: [“(7, 8, u\’Bad signature\’)”, “(7, 8, u\’Bad signature\’)”, “(7, 8, u\’Bad signature\’)”] ‘,): Permission denied.
Note: This error might indicate a problem with your passive_ftp setting.
Please consult dput.cf(5) for details on this configuration option.

which means you get immediate feedback instead of your upload disappearing entirely!

As a bonus, this new FTP server also fixes another long-standing bug where uploads would hang with 1k left to go.

This checking is not available on the SFTP service yet, but we hope to implement that in the near future.

An long-requested feature in Launchpad is to let people see who’s using a PPA. Finally, we’ve implemented this!

Initially, the stats are only available on Launchpad’s webservice API. but we aim to show something useful in the web UI at some point.

If you are already familiar with the webservice API, then you can use the following binary_package_publishing_history object methods to retrieve the information:

• getDailyDownloadTotals
• getDownloadCount
• getDownloadCounts

Fabien Tassin is already using the stats to see how many people are using his daily build PPAs, and wrote an interesting blog post about it.

Continuing with the recent improvements to the build farm – Jelmer has made another massive one.

The last major scalability problem that we had was one where the whole farm was blocked when a single builder was ready to upload a build. In the case of large packages, like the kernel, the manager process could block for over a minute while it waited for the upload processor to unpack the package and verify its contents.

Jelmer’s work has decoupled the upload processing from the build farm manager process. What happens now is that the files collected from the builder are thrown into a staging queue area and then the manager process immediately continues with polling the rest of the builders, unblocked. A cron job will then process the builder upload queue at 1 minute intervals.

You can see the dramatic effect this has had on the overall queue for the PPA builders here:

This is quite an incredible improvement as you can see! But we’re not stopping there, we’re currently doing a massive refactoring of the builder dispatching code so it’s all fully asynchronous. When this is all done we’re going to be in superb shape to support an increase in load that’s anticipated from the increasing number of people using the package recipes.

Improvements, you say?

Very recently we saw the beta release of a new feature on Launchpad: building packages from recipes. Recipes bring Bazaar branches for the upstream source code and a packaging branch together to generate a Debian source package. We informally referred to this internally as “the Wellington project” because we first embarked on this long road of development back in November 2009 with a coding sprint in Wellington, New Zealand.

So, why do we need to use the build farm for this?

Very early on we realised that this would be a challenging project because packaging recipes allow untrusted arbitrary code to run as part of building the package, so we decided that the only place we could do this operation was in the same place that we do one of the other untrusted operations – the PPA build machines.

The PPA build machines work for untrusted code because they run as a virtual machine. After the build job has finished, the machine is simply ripped out and restarted so if any code did something nasty, we throw all the nastiness away!

Why can’t the build farm do this already?

The Launchpad build farm has been around for a while now, but it’s of course only used for building source packages into binary debs. We realised that we’d need a whole new data model on the Launchpad side, changes to the master/slave protocol and changes in the slave code to make it actually run this new job type – a lot of work! It was also about this time I became aware of a further untrusted job type, generation of translation templates for Launchpad Translations, so we began work on making the build farm able to process any kind of job.

What had to change?

By far the biggest change for this was to fix how we store the build/job information in the database. The canonical way to refer to a “job” on the build farm was via “build” record, and this is what all of the history pages in the UI were using. We’ve now re-architected this to use a generic “build farm job” and a bunch of new database tables that re-factor all the information needed for various jobs on the build farm, in fact we found out that the recipe build jobs and the source package build jobs had quite a lot in common which resulted in some merciless code re-factoring indeed.

The other major change was the slave builder code itself. This runs as a Twisted executive that we talk to using XMLRPC from the build manager, which kicks off external processes to run the job and monitors them. The only job it knew about was to run sbuild to build source packages. We’ve changed this to also run bzr builder and a custom script to generate the translations templates.

So, will I see anything different?

A recipe build in action

You won’t see much visually that’s changed in the Build Farm. If you’re observant, you’ll see the occasional job appear in the list that announces it’s a recipe build, but they’re generally pretty quick to complete!

We’ll post a more detailed explanation of how to use recipe builds in the future, but for now it’s remaining in beta until all the problems are ironed out. If you have any feedback, please file a bug or send us an email.

Our intrepid communications expert, the venerable Matthew Revell, has been quite busy interviewing the Launchpad developers of late. I thought I’d turn the tables on him and ask a few questions!

So, here goes:

Julian: Mr Revell, I’ve heard your name pronounced in different ways. Can you please slay this dragon once and for all, is it RE-vell or re-VELL?

Matt: I pronounce it with the emphasis on the second syllable, to rhyme with
“bell”. But I’m not precious about it 🙂

Julian: Ok excellent, I’m glad we got that sorted out. So, tell us about your role in Launchpad. What are your day-to-day tasks, and do you have any special duties on top of that?

Matt: I work in the new Launchpad Strategy team, with Karl Fogel and Jonathan Lange. My main responsibility is communication with the people who use Launchpad: documentation, blogging, in-line help, the Launchpad tour, announcements, that sort of thing. So, you should speak to me (mrevell on Freenode) if you have ideas or suggestions for that sort of thing.

With Karl and Jonathan I’m also talking to lots of people who use Launchpad to find out how we can improve what we offer. So, at November’s Ubuntu Developer Summit in Dallas we’ll be organising lots of meetings to talk to people who work on Ubuntu to find out how Launchpad can better connect them with upstream projects.

Something else I’m interested in is usability and I’m now one of Launchpad’s trainee UI reviewers, which can be fun and daunting.

As for day-to-day tasks, most days are pretty different for me but I do have the fun job of weeding out the spam sent to our feedback@launchpad.net address each day. With such a huge volume of spam, even those few messages that slip through the filter still add up to a hundred or more a day.

Julian: Where do you work?

Matt: Right now, I’ve converted a corner of our dining room into an office and work there mostly. I live in the city of Wolverhampton, which is about 40 miles east of the England/Wales border and around 130 miles north-west of London. From time to time I pop up to an arts centre in the city, as they have free wifi in their cafe, for a change of scene.

We’re moving to a village a few miles from here soon, though.

Julian: What can you see from your office window? Having been to your office, I can appreciate you might want to make up an answer here.

Matt: Heh, I can see other houses in my street and that’s about it 🙂

Julian: What did you do before working at Canonical?

Matt: I did a similar job at an ISP/web host. I spent most of my time writing articles about how to set up a dedicated server’s DNS and that sort of thing.

Julian: Apparently you used to do stuff with LUG radio! Tell us more about that.

Matt: Yeah, I was a presenter on the LugRadio podcast from the first show until May 2007, when I left to do a politics show on a local station in Wolverhampton. Although the LugRadio podcast is now over, the LugRadio Live event continues and is taking place this Saturday in Wolverhampton.

Julian: How did you get into Free software?

Matt: I got my first computer when I was six. It was a Sinclair ZX81 and was a great introduction to computing. A couple of Sinclair Spectrums, a Commodore Amiga (many hours spent in writing bad code in Amos) and several Amstrad PCWs later, my dad got a PC. I had always enjoyed trying to get my machines to do something different so we ended up putting IBM’s OS/2 on it.

From there I ran a Fidonet BBS and then got into the internet, where free software was pretty hard to avoid 🙂 It wasn’t until 2001, though, that I first installed SuSe Linux on my PC, and from there went through various versions of Red Hat, with a Gentoo pit-stop, until I settled on Debian. When the Ubuntu Warty beta became available, I tried that out and have run Ubuntu since then.

Julian: What’s more important? Principle or pragmatism?

Matt: I think the pragmatic answer is that it’s important to find a route between the two 🙂 I believe in Free software both as a matter of principle and because of the practical benefits it offers me.

I think the only non-Free software I use right now is Skype but that’s because it’s the only way to speak to certain people free of charge and I think that benefit outweighs the relatively minor risks of a VOIP app being proprietary. I also use SIP quite a bit but even then it’s through a hardware adapter so, while it’s an open standard, it’s actually using proprietary firmware.

More generally, I think it’s important to have principles that guide you but none of us is ever right all the time and so it’s important to re-think from time to time.

Julian: Do you/have you contribute(d) to any free software projects?

Matt: Yes, Launchpad 🙂 I used to do a lot for Ubuntu’s Fridge news site and the Ubuntu marketing team. I’ve also worked on some documentation for Bazaar.

My wife and I have two small children so any time I’m not doing paid work is rarely mine 🙂 I keep hoping I’ll find some time to get stuck into the Gnome Documentation Project and to take my Python skills beyond less-than-rudimentary; I’m not holding my breath, though.

Julian: There is a vicious rumour circulating that you like to talk like a Leprechaun when drunk. Can you confirm or deny this rumour?

Matt: I don’t have to be drunk.

Julian: Okay, Kiko’s special question! You’re at your computer, you reach for your wallet: what are you most likely to be doing?

Matt: Usually buying something I don’t have time to use. I bought a Bluetooth remote control a few weeks ago with intention of turning an old laptop into a Boxee machine. The remote is still in its packaging.

Thanks very much Matt! And now, I return you to your regularly scheduled programming.

The Soyuz team was very, very busy over the last year fixing lots of bugs and adding plenty of new features. These are the highlights that I’d like to mention!

New features

• Multiple PPAs per person — split your packages into different repositories without the hassle of creating new LP users.
• Karma for uploads! Distro and PPA uploaders (and the package creator if different) will be recognised for their work and get karma.
• Massively improved webservice APIs to control various operations, such as copying packages, manipulating builds, inspecting PPAs etc. Allows script-based control of many soyuz-related operations.
• Hugely faster build farm scanner, builds are dispatched a LOT quicker now. That means there’s less waiting around for your packages to get built.
• Private PPA subscriptions / tokens. People can now control who is able to download their PPA software.
• Package sets and their upload ACLs implemented for Ubuntu. Karmic and onwards will be using package sets for upload permissioning.
• Security in Soyuz. Complete support for the Canonical security team to use a private PPA and directly unembargo packages from it.

3.0 UI

We also did a lot of complete page redesigns for 3.0:

• PPA page split into two pages; one user-focused, one developer-focused.
• New build page, with a cleaner look.
• New global /builders page; sortable table data and a time-based estimate of the queue sizes.
• New distribution source package release page; much better presentation of data.
• New distribution source package page; makes more use of upstream package description/logo etc.
• Distro series source package release page gets a new layout.
• Builder +admin and builder +edit collapsed into a single page.
• New builder page.

But the work doesn’t stop here. We’re already thinking about 4.0!

The release of 2.2.3 brings with it a change for PPAs — we are now supporting multiple PPAs per person!

You will have probably already noticed that we made a change in 2.2.1 that altered the URLs you see for a PPA’s index page and its sources.list entry; these now have the extra “ppa” in there. This is actually the default name for the first PPA that anyone creates.
Example old-style URL:

https://launchpad.net/~mythbuntu/+archive

Which is now:

https://launchpad.net/~mythbuntu/+archive/ppa

You can create additional PPAs from your own profile page or your team’s profile page. You are able to name them whatever you like with one exception — we are not allowing you to name them “ubuntu”. This is so that we can maintain backwards compatibility with the old upload paths where you don’t need to specify the default PPA’s name.

To upload to the additional PPAs your .dput.cf upload path should look like:

incoming = ~myname/ubuntu/otherppaname

These additional PPAs will behave as any other PPA, the only difference being that they clearly belong to a person or a team, and you won’t need to create dummy teams/people to have more PPAs.

We hope you enjoy this new feature!

Last week was the first of some Launchpad Performance Weeks that we’re embarking upon. The Soyuz team worked hard on improving the performance of the PPA page, which had a tendency to be very slow or even time out on PPAs that have a lot of binaries published.

The more observant of you will have noticed that the PPA page on edge is now using asynchronous requests to fetch both the “Repository disk usage” section, and the package information section that is expanded when you click on the triangle at the left of each source package row. This means that the initial page load time is a lot quicker! There’s still some more speed we can get out of the page by reducing the query load that it places on the database server, and we’ll deliver that change soon.

We also fixed a smaller snafu — as part of the webservice work that we’re doing, we managed to accidentally enable a URL that let you view the main Ubuntu repository as if it were a PPA page. These links nearly always timed out because of the size of the Ubuntu archive, so that URL will now redirect to the /ubuntu page on Launchpad instead.

Also in the pipeline is a major overhaul of the PPA page. Watch out for that soon!