launchpadblog

Continuing with the recent improvements to the build farm – Jelmer has made another massive one.

The last major scalability problem that we had was one where the whole farm was blocked when a single builder was ready to upload a build. In the case of large packages, like the kernel, the manager process could block for over a minute while it waited for the upload processor to unpack the package and verify its contents.

Jelmer’s work has decoupled the upload processing from the build farm manager process. What happens now is that the files collected from the builder are thrown into a staging queue area and then the manager process immediately continues with polling the rest of the builders, unblocked. A cron job will then process the builder upload queue at 1 minute intervals.

You can see the dramatic effect this has had on the overall queue for the PPA builders here:

This is quite an incredible improvement as you can see! But we’re not stopping there, we’re currently doing a massive refactoring of the builder dispatching code so it’s all fully asynchronous. When this is all done we’re going to be in superb shape to support an increase in load that’s anticipated from the increasing number of people using the package recipes.

You can now use SFTP to upload source packages to your Personal Package Archive!

If you’re already familiar with uploading to a PPA, all you need to do is ensure your dput.cf includes the following:


method = sftp
login = <your Launchpad account name>


If you’re new to PPAs, but already know how to create packages for Ubuntu, take a look at our guide.

Since we first launched Personal Package Archives, we’ve set a starting size limit of 1 GiB.

However, we’ve also said “yes” pretty much every time someone’s ask us for more space. So, seeing as how most requests have been for an increase to 2 GiB, we’ve gone ahead and upgraded every PPA to a minimum of 2 GiB. If you already have a larger allowance, it’ll stay in place.

If you upload source packages to Launchpad — either to be built and distributed in a PPA or for Ubuntu’s primary archive — you’ve probably spotted the term “build score” and wondered what it represents.

A build score is the priority that Launchpad has assigned to a particular build job. I’ve just added a new page to the Launchpad help wiki that explains how Launchpad decides that priority but here’s the quick version.

When you upload a source package, Launchpad looks at:

• the target component
• the target pocket
• source urgency — defined by the packager
• time since upload to the queue
• whether the package is destined for a public or private PPA.

Launchpad then gives the build a score in each of those categories, adds it all up and the total is the build score you see on the build details page. The higher number, the higher the priority.

So, for example, a build destined for the security pocket gets a higher priority than a backport. However, to prevent lower priority packages from never making it to a builder, packages get an extra few points as they get older.

Get the full details in the guide over on the help wiki.

Recently I spent a day with Julian Edwards, one of the developers working on Launchpad’s Personal Package Archives feature.

We wanted to come up with a plan for improving the help materials we have for PPAs, including:

• documentation on the Launchpad help wiki
• the PPA page on the Launchpad tour
• user-interface text and in-line help.

Right now, our PPA help guide is one long and somewhat unwieldy wiki page. Over the next couple of months, I’m going to break it down into smaller guides each based around one particular task, such as creating the source package and uploading the package, including how to deal with errors.

Up until now, we haven’t included any help on actually creating a Debian-style package ready for uploading to a PPA. One of the reasons for that was that there’s a wealth of packaging material and support already available in the Ubuntu community and more widely. However, one of the new guides I’m going to create will look at basic packaging, tailored specifically to the needs of PPAs.

If you use PPAs, I’d love your input on this work. Either mail me directly, leave a comment here or visit the bug report and blueprints I’ve created:

• packaging guide blueprint
• package upload guide blueprint
• package publication lifecycle reference blueprint
• downloading and installing from PPAs guide blueprint
• bug report tracking changes needed for the PPA tour page.

The release of 2.2.3 brings with it a change for PPAs — we are now supporting multiple PPAs per person!

You will have probably already noticed that we made a change in 2.2.1 that altered the URLs you see for a PPA’s index page and its sources.list entry; these now have the extra “ppa” in there. This is actually the default name for the first PPA that anyone creates.
Example old-style URL:

https://launchpad.net/~mythbuntu/+archive

Which is now:

https://launchpad.net/~mythbuntu/+archive/ppa

You can create additional PPAs from your own profile page or your team’s profile page. You are able to name them whatever you like with one exception — we are not allowing you to name them “ubuntu”. This is so that we can maintain backwards compatibility with the old upload paths where you don’t need to specify the default PPA’s name.

To upload to the additional PPAs your .dput.cf upload path should look like:

incoming = ~myname/ubuntu/otherppaname

These additional PPAs will behave as any other PPA, the only difference being that they clearly belong to a person or a team, and you won’t need to create dummy teams/people to have more PPAs.

We hope you enjoy this new feature!

If you’ve ever contributed to a Debian-based distribution or used the neat Launchpad PPA services then you’ve probably used dput before. We use dput to upload packages to a Debian repository such as the Debian Archive, Ubuntu Archive, or personal archives such as a Personal Package Archive on Launchpad. Getting the hard work you’ve done to the archive for the whole world to enjoy is an important part of a packager’s work flow and should be easy as possible. This is probably why dput is so popular – it does its job and stays out of the way.

However, recently while adding support to dput for uploading packages via sftp, I realized that one’s dput configuration file can get rather lengthy and messy. Being the Lazy Engineer (TM) that I am, I decided to fix this problem. What I noticed was that you needed to configure an upload target (aka host) for each PPA you might want to upload to but all the configuration stanzas for PPAs were exactly the same except for a part of the path. In fact, it seemed like a number of the stanzas for other archives in my dput configuration file were the same except for the same spot. Wouldn’t it be nice if I could just have a single configuration stanzas for these cases and only need to supply an argument for dput to automagically do the right thing? I thought so which is why I’ve added a new feature to dput that allows you to have an upload target “template” where you can pass a single argument to the target specified that will be used to fill in the missing pieces.

So, instead of having a .dput.cf file in your home directory that looks like the following:


[my-ppa]
fqdn = upload.launchpad.net
method = ftp
incoming = ~cody-somerville/ppa/ubuntu
login = anonymous

[xubuntu-ppa]
fqdn = upload.launchpad.net
method = ftp
incoming = ~xubuntu-dev/ppa/ubuntu
login = anonymous

….

I can simply have the following:


[ppa]
fqdn = upload.launchpad.net
method = ftp
incoming = ~%(ppa)s/ubuntu
login = anonymous


To upload to my PPA now, I’d type: dput ppa:cody-somerville/ppa <changesfile>


To upload to the Xubuntu Developers’ ppa, I’d type: dput ppa:xubuntu-dev/ppa <changesfile>


Once Launchpad supports multiple PPAs per user or team, simply replace “ppa” with
the name of the PPA. If you’re lazy, you can modify your .dput.cf to have the
ppa part automatically entered for you until then.

Anyhow, I hope you enjoy. Cheers!

Last week was the first of some Launchpad Performance Weeks that we’re embarking upon. The Soyuz team worked hard on improving the performance of the PPA page, which had a tendency to be very slow or even time out on PPAs that have a lot of binaries published.

The more observant of you will have noticed that the PPA page on edge is now using asynchronous requests to fetch both the “Repository disk usage” section, and the package information section that is expanded when you click on the triangle at the left of each source package row. This means that the initial page load time is a lot quicker! There’s still some more speed we can get out of the page by reducing the query load that it places on the database server, and we’ll deliver that change soon.

We also fixed a smaller snafu — as part of the webservice work that we’re doing, we managed to accidentally enable a URL that let you view the main Ubuntu repository as if it were a PPA page. These links nearly always timed out because of the size of the Ubuntu archive, so that URL will now redirect to the /ubuntu page on Launchpad instead.

Also in the pipeline is a major overhaul of the PPA page. Watch out for that soon!

Last month I mentioned that we were generating a unique key for each Personal Package Archive.

Well, that’s complete meaning each PPA now has its own key that’s used to sign its packages. And if you create a new PPA, Launchpad will generate a new key for it within a couple of hours.

This means that you now need to add the PPA’s key to apt before you install any of its packages. It’s really easy: all you need to do is copy the PPA’s public key and import it using System->Administration->Software Sources and then the Authentication tab.

Here’s a screencast that takes you through the steps:

(Higher quality Ogg Theora version)

Of course, you can also do it in the terminal. There’s more on the PPA help page.

Note: the PPA keys help you see that the package hasn’t been altered since Launchpad built it on behalf of the PPA owner. It does not mean that Launchpad, Ubuntu or Canonical endorse the packages. You should ensure you trust the PPA owner before you install their software.

If you’ve ever come to file a bug and found that it’s already been reported, you may have wanted to let the project know that you too have been affected.

Trouble is: many projects find “me too” comments unhelpful because they don’t add much to the discussion.

Launchpad’s new “This bug affects me too” feature lets you record just that but without the guilt! Give it a go in our staging environment.

We’ll use the “me too” data in future to help projects identify “hot” bugs.