launchpadblog

Improvements, you say?

Very recently we saw the beta release of a new feature on Launchpad: building packages from recipes. Recipes bring Bazaar branches for the upstream source code and a packaging branch together to generate a Debian source package. We informally referred to this internally as “the Wellington project” because we first embarked on this long road of development back in November 2009 with a coding sprint in Wellington, New Zealand.

So, why do we need to use the build farm for this?

Very early on we realised that this would be a challenging project because packaging recipes allow untrusted arbitrary code to run as part of building the package, so we decided that the only place we could do this operation was in the same place that we do one of the other untrusted operations – the PPA build machines.

The PPA build machines work for untrusted code because they run as a virtual machine. After the build job has finished, the machine is simply ripped out and restarted so if any code did something nasty, we throw all the nastiness away!

Why can’t the build farm do this already?

The Launchpad build farm has been around for a while now, but it’s of course only used for building source packages into binary debs. We realised that we’d need a whole new data model on the Launchpad side, changes to the master/slave protocol and changes in the slave code to make it actually run this new job type – a lot of work! It was also about this time I became aware of a further untrusted job type, generation of translation templates for Launchpad Translations, so we began work on making the build farm able to process any kind of job.

What had to change?

By far the biggest change for this was to fix how we store the build/job information in the database. The canonical way to refer to a “job” on the build farm was via “build” record, and this is what all of the history pages in the UI were using. We’ve now re-architected this to use a generic “build farm job” and a bunch of new database tables that re-factor all the information needed for various jobs on the build farm, in fact we found out that the recipe build jobs and the source package build jobs had quite a lot in common which resulted in some merciless code re-factoring indeed.

The other major change was the slave builder code itself. This runs as a Twisted executive that we talk to using XMLRPC from the build manager, which kicks off external processes to run the job and monitors them. The only job it knew about was to run sbuild to build source packages. We’ve changed this to also run bzr builder and a custom script to generate the translations templates.

So, will I see anything different?

You won’t see much visually that’s changed in the Build Farm. If you’re observant, you’ll see the occasional job appear in the list that announces it’s a recipe build, but they’re generally pretty quick to complete!

We’ll post a more detailed explanation of how to use recipe builds in the future, but for now it’s remaining in beta until all the problems are ironed out. If you have any feedback, please file a bug or send us an email.

Tags: front-page

This entry was posted by Julian Edwards on Monday, August 2nd, 2010 at 3:23 pm and is filed under Cool new stuff. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.