launchpadblog

Launchpad has supported building for riscv64 for a while, since it was a requirement to get Ubuntu’s riscv64 port going. We don’t actually have riscv64 hardware in our datacentre, since we’d need server-class hardware with the hypervisor extension and that’s still in its infancy; instead, we do full-system emulation of riscv64 on beefy amd64 hardware using qemu. This has worked well enough for a while, although it isn’t exactly fast.

The biggest problem with our setup wasn’t so much performance, though; it was that we were just using a bunch of manually-provisioned virtual machines, and they weren’t being reset to a clean state between builds. As a result, it would have been possible for a malicious build to compromise future builds on the same builder: it would only need a chroot or container escape. This violated our standard security model for builders, in which each build runs in an isolated ephemeral VM, and each VM is destroyed and restarted from a clean image at the end of every build. As a result, we had to limit the set of people who were allowed to have riscv64 builds on Launchpad, and we had to restrict things like snap recipes to only use very tightly-pinned parts from elsewhere on the internet (pinning is often a good idea anyway, but at an infrastructural level it isn’t something we need to require on other architectures).

We’ve wanted to bring this onto the same footing as our other architectures for some time. In Canonical’s most recent product development cycle, we worked with the OpenStack team to get riscv64 emulation support into nova, and installed a backport of this on our newest internal cloud region. This almost took care of the problem. However, Launchpad builder images start out as standard Ubuntu cloud images, which on riscv64 are only available from Ubuntu 22.04 LTS onwards; in testing 22.04-based VMs on other relatively slow architectures we already knew that we were seeing some mysterious hangs in snap recipe builds. Figuring this out blocked us for some time, and involved some pretty intensive debugging of the “strace absolutely everything in sight and see if anything sensible falls out” variety. We eventually narrowed this down to a LXD bug and were at least able to provide a workaround, at which point bringing up new builders was easy.

As a result, you can now enable riscv64 builds for yourself in your PPAs or snap recipes. Visit the PPA and follow the “Change details” link, or visit the snap recipe and follow the “Edit snap package” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out, including riscv64. This now means that all Ubuntu architectures are fully virtualized and unrestricted in Launchpad, making it easier for developers to experiment.

Since they were introduced in 2007, Launchpad’s Personal Package Archives (PPAs) have always been hosted on ppa.launchpad.net. This has generally worked well, but one significant snag became clear later on: it was difficult to add HTTPS support for PPAs due to the way that cookies work on the web.

Launchpad uses a cookie for your login session, which is of course security-critical, and because we use multiple domain names for the main web application (bugs.launchpad.net, code.launchpad.net, and so on), the session cookie domain has to be set to allow subdomains of launchpad.net. We set the “Secure” flag on session cookies to ensure that browsers only ever send them over HTTPS, as well as the “HttpOnly” flag to prevent direct access to it from JavaScript; but there are still ways in which arbitrary JS on an HTTPS subdomain of launchpad.net might be able to exfiltrate or abuse users’ session cookies. As a result, we can never allow any HTTPS subdomain of launchpad.net to publish completely user-generated HTML that we don’t process first.

We don’t currently know of a way to get ppa.launchpad.net to serve arbitrary HTML as Content-Type: text/html, but this is quite a brittle protection as there are certainly ways (used for things like installer uploads) to upload arbitrary files to ppa.launchpad.net under a user-controlled directory structure, and we don’t want the webapp’s security to depend on nobody figuring out how to convince a browser to interpret any of that as arbitrary HTML. The librarian is already on a separate launchpadlibrarian.net domain name for a similar reason.

To resolve this dilemma, we’ve added a new ppa.launchpadcontent.net domain name which supports both HTTP and HTTPS (and similarly private-ppa.launchpadcontent.net for private PPAs, which as before is HTTPS-only). add-apt-repository in Ubuntu 22.04 will use the new domain name by default.

The old names will carry on working indefinitely – we know they’re embedded in lots of configuration and scripts, and we have no inclination to break all of those – but we recommend moving to the new names where possible. ppa.launchpad.net will remain HTTP-only.

Some systems may need to be updated to support the new domain name, particularly things like HTTP(S) proxy configuration files and no_proxy environment variables.

Personal package archives on Launchpad only build for the amd64 and i386 architectures by default, which meets most people’s needs.  Anyone with an e-mail address can have a PPA, so they have to be securely virtualised, but that’s been feasible on x86 for a long time.  Dealing with the other architectures that Ubuntu supports (currently arm64, armhf, powerpc, and ppc64el) in a robust and scalable way has been harder.  Until recently, all of those architectures were handled either by running one builder per machine on bare metal, or in some cases by running builders on a small number of manually-maintained persistent virtual machines per physical machine.  Neither of those approaches scales to the level required to support PPAs, and we need to make sure that any malicious code run by a given build is strictly confined to that build.  (We support virtualised armhf PPAs, but only by using qemu-user-static in an amd64 virtual machine, which is very fragile and there are many builds that it simply can’t handle at all.)

We’ve been working with our sysadmins for several months to extend ScalingStack to non-x86 architectures, and at the start of Ubuntu’s 16.04 development cycle we were finally able to switch all ppc64el builds over to this system.  Rather than four builders, we now have 30, each of which is reset to a clean virtual machine instance between each build.  Since that’s more than enough to support Ubuntu’s needs, we’ve now “unrestricted” the architecture so that it can be used for PPAs as well, and PPA owners can enable it at will.  To do this, visit the main web page for your PPA (which will look something like “https://launchpad.net/~<person-name>/+archive/ubuntu/<ppa-name>”) and follow the “Change details” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out.  This also means that you can disable amd64 or i386 builds for your PPA if you want to.

We’re working to extend this to all the existing Ubuntu architectures at the moment.  arm64 is up and running but we’re still making sure it’s sufficiently robust; armhf will run on arm64 guests, and just needs a kernel patch to set its uname correctly; and powerpc builds will run in different guests on the same POWER8 compute nodes as ppc64el once we have suitable cloud images available.  We’ll post further announcements when further architectures are unrestricted.

We’ve made a number of improvements to the Launchpad build farm in the last month, with the aim of improving its performance and robustness.  This sort of work is usually invisible to users except when something goes wrong, so we thought it would be worth taking some time to give you a summary.  Some of this work was on the Launchpad software itself, while some was on the launchpad.net hardware.

(To understand some of the rest of this post, it’s useful to be aware of the distinction between virtualised and devirtualised builders in Launchpad.  Virtualised builders are used for most PPAs: they build untrusted code in a Xen guest which is initialised from scratch at the start of each build, and are only available for i386, amd64, and a small number of ARM builds by way of user-mode QEMU.  Devirtualised builders run on ordinary hardware with less strict containment, and are used for Ubuntu distribution builds and a few specialised PPAs.)

ARM builders have been a headache for some time.  For our devirtualised builders, we were using a farm of PandaBoards, having previously used BeagleBoards and Babbage boards.  These largely did the job, but they’re really a development board rather than server-class hardware, and it showed in places: disk performance wasn’t up to our needs and we saw build failures due to data corruption much more frequently than we were comfortable with.  We recently installed a cluster of Calxeda Highbank nodes, which have been performing much more reliably.

It has long been possible to cancel builds on virtualised builders: this is easy because we can just reset the guest.  However, it was never possible to cancel builds on devirtualised builders: killing the top-level build process isn’t sufficient for builds that are stuck in various creative ways, and you need to make sure to go round and repeatedly kill all processes in the build chroot until they’ve all gone away.  We’ve now hooked this up properly, and it is possible for build daemon maintainers to cancel builds on devirtualised builders without operator assistance, which should eliminate situations where we need urgent builds to jump the queue but can’t because all builders are occupied by long-running builds.  (People with upload privileges can currently cancel builds too, which is intended mainly to allow cancelling your own builds; please don’t abuse this or we may need to tighten up the permissions.)  As a bonus, cancelling a build no longer loses the build log.

Finally, we have been putting quite a bit of work into build farm reliability.  A few problems have led to excessively long queues on virtual builders:

• Builders hung for some time when they should have timed out, due to a recent change in su; this is now fixed in the affected Ubuntu series.
• Xen guests often fail to restore for one reason or another, and when this happened builders would fail in ways that required an operator to fix.  We had been dealing with this by having our operators do semi-automatic builder fixing runs a few times a day, but in recent months the frequency of failures has been difficult to keep up with in this way, especially at the weekend.  Some of this is probably related to our current use of a rather old version of Xen, but the builder management code in Launchpad could also handle this much better by trying to reset the guest again in the same way that we do at the start of each build.  As of this morning’s code deployment, we now do this, and the build farm seems to be holding up much more robustly.

This should make things better for everyone, but we aren’t planning to stop here.  We’re intending to convert the virtual builders to an OpenStack deployment, which should allow us to scale them much more flexibly.  We plan to take advantage of more reliable build cancellation to automatically cancel in-progress builds that have been superseded by new source uploads, so that we don’t spend resources on builds that will be rejected on upload.  And we plan to move Ubuntu live file system building into Launchpad so that we can consolidate those two build farms and make better use of our available hardware.

The Ubuntu Foundations team has sponsored work on various improvements to Launchpad’s archive handling lately, mainly to expose various new facilities on the API where we were previously using privileged scripts.  This has involved cleaning up a substantial amount of old code along the way, and it has become possible to fix some other old bugs as spin-offs.

One of these old bugs is “Archive:+copy-packages nearly unusable due to timeouts”.  The +copy-packages page allows anyone who can upload to a PPA to instead copy packages from another PPA.  This saves effort, and in the “Copy existing binaries” mode it can save a substantial amount of build time as well.  For example, the LibreOffice packaging team uses this to deliver packages to different sets of users after they have passed various levels of testing.

Unfortunately, the very cases where this is most useful, namely large and complex packages, are also the cases where it is most likely to break.  Copying large numbers of binary packages involves large numbers of database queries and can quite easily overrun the timeout for a single request to the Launchpad web application.  Doing this for several series at once, a common case which seems reasonable, is proportionally less likely to work.  Various attempts have been made to optimise the database interactions here, but ultimately doing lots of complex synchronous work in time for a single web request is doomed to failure.

The solution to all this is to copy packages asynchronously.  For some time Launchpad has had the ability to schedule “package copy jobs” which run very shortly after the request (typically within a minute) but not immediately.  For example, the Ubuntu team uses these when copying new versions of packages from Debian unstable in cases where there are no Ubuntu-specific modifications, and when releasing proposed updates to stable releases for general use after verification.  A similar facility has been present in the code for the +copy-packages page for some time, but not exposed due to various bugs.  We believe that these bugs have been fixed now, and so we would like to start copying packages asynchronously when requested via the web UI.

We have exposed this to beta testers first.  The effect is that, if you are a beta tester when you ask for packages to be copied, you will be told something like “Requested sync of 2 packages.  Please allow some time for these to be processed.”  The processing should normally happen within a minute or two, and you will be able to see it in progress on the +packages page for the target archive.  If it succeeds, the in-progress notification will be removed and you will be able to see the changes in the target archive.  Otherwise, you will see a failure notification along these lines:

If beta-testing goes well, then we will enable this for all users, and remove the old synchronous copying code shortly afterwards; so please do report any problems you see.

If you are relying on package copies in the web UI happening immediately rather than within a few minutes, firstly, please contact us (e.g. #launchpad-dev on freenode IRC, or launchpad-users@lists.launchpad.net) as we would like to understand your requirements in more detail; secondly, you may be able to use the Archive.syncSource API method instead, which also has timeout constraints but is at least guaranteed to remain synchronous.  However, we hope that most people will not have such a requirement.

Setting up a commercial project in Launchpad has gotten easier. You can now quickly register a proprietary project and enable private bugs. You can create private teams and private personal package archives, AKA private PPA or P3A without the assistance of a Launchpad admin.

When you select the Other/Proprietary license while registering a project, or changing the project’s details,

it is given a complimentary 30-day commercial subscription.

The delay between the moment when a commercial project was registered and when the commercial subscription was purchased and then applied to the project caused a lot of confusion. During this delay, proprietary data could be disclosed. We chose to award the project with a short term commercial subscription which enabled the project to be properly configured while the 12-month commercial subscription was being purchased and applied to the project.

Any project with a commercial subscription can enable

Default private bugs

Once enabled by configuring the project’s bug tracker, all new reported bugs are private. You can choose to make the report public.

Default private branches

You can request a Launchpad admin to configure private branches for your teams. (You will be able to do this yourself in the near future when projects gain proprietary branches.)

As the maintainer of a project with a commercial subscription, you can register

Private teams

When you register a team, you can choose to set the team visibility to private. The team’s members and data is hidden from non-members.

Private mailing lists

When you create a mailing list for a private team, the archive is also private. Only team members may see the messages in the archive.

Private PPAs

When you create a PPA for your public team, you may choose to make it private; private teams can only have private PPAs. You can subscribe users to your archive so that they may install packages without revealing all your team’s members and data to the subscriber.

A secondary benefit of this change is that you can now try Launchpad’s commercial features before purchasing a 12-month commercial subscription. The features will be disabled at the end of 30-days. Your test data will remain private to ensure your data is not disclosed.

Any open source project may also have a commercial subscription to enable commercial features. You can purchase a commercial subscription at the Canonical store. Commercial subscriptions cost US$250/year/project + applicable V.A.T.

(Photo by Fred Dawson on flickr, creative commons license)

Short story: http://pad.lv/12345 takes you to bug 12345, and pad.lv describes more abbreviations.

Sometimes you’d like to point people to an interesting bug in a project that uses Launchpad, like bug 685380 (that ‘1’ and ‘l’ may need to be more distinct in the new Ubuntu Font).

Typing out https://launchpad.net/bugs/685380 is a bit tedious, and it uses up a fair bit of space in a microblog entry. You can use any of innumerable URL-shortening services, but then the URL’s opaque; which is a shame since it really just wants to represent a 6-digit number.

Therefore: pad.lv (pad love), transparent short URLs for bugs, and other things including projects, people, bug-filing forms, packages, and more.

Maybe someone would like to make bookmarklets that generate these links, or add them into the Launchpad UI?

Thanks to Latvia for letting us use a fraction of their domain name space!

You may start getting “Failed to fetch” error messages when updating your software sources (e.g. through “apt-get update” or “Reload package information” in Synaptic), which may be due to a bug we’ve just cleaned up in Launchpad’s PPAs.

The error looks like this:

  W: Failed to fetch http://ppa.launchpad.net/chromium-daily/ppa/ubuntu/dists/maverick/Release
  Unable to find expected entry  restricted/binary-i386/Packages in Meta-index file (malformed Release file?)

  E: Some index files failed to download, they have been ignored, or old ones used instead.

(more…)