launchpadblog

Launchpad has supported building for riscv64 for a while, since it was a requirement to get Ubuntu’s riscv64 port going. We don’t actually have riscv64 hardware in our datacentre, since we’d need server-class hardware with the hypervisor extension and that’s still in its infancy; instead, we do full-system emulation of riscv64 on beefy amd64 hardware using qemu. This has worked well enough for a while, although it isn’t exactly fast.

The biggest problem with our setup wasn’t so much performance, though; it was that we were just using a bunch of manually-provisioned virtual machines, and they weren’t being reset to a clean state between builds. As a result, it would have been possible for a malicious build to compromise future builds on the same builder: it would only need a chroot or container escape. This violated our standard security model for builders, in which each build runs in an isolated ephemeral VM, and each VM is destroyed and restarted from a clean image at the end of every build. As a result, we had to limit the set of people who were allowed to have riscv64 builds on Launchpad, and we had to restrict things like snap recipes to only use very tightly-pinned parts from elsewhere on the internet (pinning is often a good idea anyway, but at an infrastructural level it isn’t something we need to require on other architectures).

We’ve wanted to bring this onto the same footing as our other architectures for some time. In Canonical’s most recent product development cycle, we worked with the OpenStack team to get riscv64 emulation support into nova, and installed a backport of this on our newest internal cloud region. This almost took care of the problem. However, Launchpad builder images start out as standard Ubuntu cloud images, which on riscv64 are only available from Ubuntu 22.04 LTS onwards; in testing 22.04-based VMs on other relatively slow architectures we already knew that we were seeing some mysterious hangs in snap recipe builds. Figuring this out blocked us for some time, and involved some pretty intensive debugging of the “strace absolutely everything in sight and see if anything sensible falls out” variety. We eventually narrowed this down to a LXD bug and were at least able to provide a workaround, at which point bringing up new builders was easy.

As a result, you can now enable riscv64 builds for yourself in your PPAs or snap recipes. Visit the PPA and follow the “Change details” link, or visit the snap recipe and follow the “Edit snap package” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out, including riscv64. This now means that all Ubuntu architectures are fully virtualized and unrestricted in Launchpad, making it easier for developers to experiment.

Since they were introduced in 2007, Launchpad’s Personal Package Archives (PPAs) have always been hosted on ppa.launchpad.net. This has generally worked well, but one significant snag became clear later on: it was difficult to add HTTPS support for PPAs due to the way that cookies work on the web.

Launchpad uses a cookie for your login session, which is of course security-critical, and because we use multiple domain names for the main web application (bugs.launchpad.net, code.launchpad.net, and so on), the session cookie domain has to be set to allow subdomains of launchpad.net. We set the “Secure” flag on session cookies to ensure that browsers only ever send them over HTTPS, as well as the “HttpOnly” flag to prevent direct access to it from JavaScript; but there are still ways in which arbitrary JS on an HTTPS subdomain of launchpad.net might be able to exfiltrate or abuse users’ session cookies. As a result, we can never allow any HTTPS subdomain of launchpad.net to publish completely user-generated HTML that we don’t process first.

We don’t currently know of a way to get ppa.launchpad.net to serve arbitrary HTML as Content-Type: text/html, but this is quite a brittle protection as there are certainly ways (used for things like installer uploads) to upload arbitrary files to ppa.launchpad.net under a user-controlled directory structure, and we don’t want the webapp’s security to depend on nobody figuring out how to convince a browser to interpret any of that as arbitrary HTML. The librarian is already on a separate launchpadlibrarian.net domain name for a similar reason.

To resolve this dilemma, we’ve added a new ppa.launchpadcontent.net domain name which supports both HTTP and HTTPS (and similarly private-ppa.launchpadcontent.net for private PPAs, which as before is HTTPS-only). add-apt-repository in Ubuntu 22.04 will use the new domain name by default.

The old names will carry on working indefinitely – we know they’re embedded in lots of configuration and scripts, and we have no inclination to break all of those – but we recommend moving to the new names where possible. ppa.launchpad.net will remain HTTP-only.

Some systems may need to be updated to support the new domain name, particularly things like HTTP(S) proxy configuration files and no_proxy environment variables.

Personal package archives on Launchpad only build for the amd64 and i386 architectures by default, which meets most people’s needs.  Anyone with an e-mail address can have a PPA, so they have to be securely virtualised, but that’s been feasible on x86 for a long time.  Dealing with the other architectures that Ubuntu supports (currently arm64, armhf, powerpc, and ppc64el) in a robust and scalable way has been harder.  Until recently, all of those architectures were handled either by running one builder per machine on bare metal, or in some cases by running builders on a small number of manually-maintained persistent virtual machines per physical machine.  Neither of those approaches scales to the level required to support PPAs, and we need to make sure that any malicious code run by a given build is strictly confined to that build.  (We support virtualised armhf PPAs, but only by using qemu-user-static in an amd64 virtual machine, which is very fragile and there are many builds that it simply can’t handle at all.)

We’ve been working with our sysadmins for several months to extend ScalingStack to non-x86 architectures, and at the start of Ubuntu’s 16.04 development cycle we were finally able to switch all ppc64el builds over to this system.  Rather than four builders, we now have 30, each of which is reset to a clean virtual machine instance between each build.  Since that’s more than enough to support Ubuntu’s needs, we’ve now “unrestricted” the architecture so that it can be used for PPAs as well, and PPA owners can enable it at will.  To do this, visit the main web page for your PPA (which will look something like “https://launchpad.net/~<person-name>/+archive/ubuntu/<ppa-name>”) and follow the “Change details” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out.  This also means that you can disable amd64 or i386 builds for your PPA if you want to.

We’re working to extend this to all the existing Ubuntu architectures at the moment.  arm64 is up and running but we’re still making sure it’s sufficiently robust; armhf will run on arm64 guests, and just needs a kernel patch to set its uname correctly; and powerpc builds will run in different guests on the same POWER8 compute nodes as ppc64el once we have suitable cloud images available.  We’ll post further announcements when further architectures are unrestricted.

It’s been a while since we posted much regularly on this team blog, not least because for a while Launchpad was running more or less in maintenance mode.  That’s no longer the case and we’re back to the point where we can do feature development work again, as exemplified by our recent addition of Git code hosting support.

Lots of other things have been happening in the Launchpad world lately, though, and the half-way point in the year seems like a good time to start talking about them.  I’m going to try to do this a bit more regularly, aiming for about once a month when we also update our internal stakeholders.  This post covers roughly the last three months.

(more…)

Setting up a commercial project in Launchpad has gotten easier. You can now quickly register a proprietary project and enable private bugs. You can create private teams and private personal package archives, AKA private PPA or P3A without the assistance of a Launchpad admin.

When you select the Other/Proprietary license while registering a project, or changing the project’s details,

it is given a complimentary 30-day commercial subscription.

The delay between the moment when a commercial project was registered and when the commercial subscription was purchased and then applied to the project caused a lot of confusion. During this delay, proprietary data could be disclosed. We chose to award the project with a short term commercial subscription which enabled the project to be properly configured while the 12-month commercial subscription was being purchased and applied to the project.

Any project with a commercial subscription can enable

Default private bugs

Once enabled by configuring the project’s bug tracker, all new reported bugs are private. You can choose to make the report public.

Default private branches

You can request a Launchpad admin to configure private branches for your teams. (You will be able to do this yourself in the near future when projects gain proprietary branches.)

As the maintainer of a project with a commercial subscription, you can register

Private teams

When you register a team, you can choose to set the team visibility to private. The team’s members and data is hidden from non-members.

Private mailing lists

When you create a mailing list for a private team, the archive is also private. Only team members may see the messages in the archive.

Private PPAs

When you create a PPA for your public team, you may choose to make it private; private teams can only have private PPAs. You can subscribe users to your archive so that they may install packages without revealing all your team’s members and data to the subscriber.

A secondary benefit of this change is that you can now try Launchpad’s commercial features before purchasing a 12-month commercial subscription. The features will be disabled at the end of 30-days. Your test data will remain private to ensure your data is not disclosed.

Any open source project may also have a commercial subscription to enable commercial features. You can purchase a commercial subscription at the Canonical store. Commercial subscriptions cost US$250/year/project + applicable V.A.T.

(Photo by Fred Dawson on flickr, creative commons license)

On Monday I posted a video showing how to add a PPA to Ubuntu using a terminal.

And here’s a video showing how to do it using Ubuntu’s Software Centre.

I noticed over the weekend that the search term bringing most visitors to this blog was “how to add a PPA to Ubuntu” and variants.

So, here’s a screencast showing you how.

For more, see our help guide.

Update: here’s a video showing how to do it with the Ubuntu Software Centre.

Here’s a quick pub quiz:

Question: How do you make packages for Ubuntu?

You can choose from the following answers:

1. learn Debian packaging through hours of study and practice
2. borrow existing packaging from elsewhere, throw a couple of Bazaar branches together and let Launchpad handle the rest
3. Uruguay in both 1930 and 1950.

If you selected either of the first answers you’d be right.

Okay, so, if you want to do it for real — i.e. become an Ubuntu MOTU or otherwise create Debian-style packages from scratch — then you still need to go through the hard work.

However, for everyone else who really just needs to get something out there and working for, say, a group of beta testers, we now have Launchpad’s source package recipes.

How it works, in three steps

It’s almost ridiculously easy to set up a source package build:

And that’s it. Within a few minutes you can set up a daily build direct from your trunk or any other buildable branch in Launchpad.

Watch how it works in our screencast:

An example

Let’s say you’re the developer of a home finance application called Alvin. You track your project’s code using Git and host it on your own server. For the past couple of years Alvin has been packaged in the Ubuntu universe and your trunk has also been imported from Git to a Bazaar branch in Launchpad at lp:alvin.

Just as you’re approaching Alvin’s next release, you want to get some wider testing. In the past, you’ve published a nightly tarball and provided instructions on manual installation. That’s given you a handful of dedicated beta testers but you’re worried that you’re asking too much of people.

With Launchpad’s source package recipes, you write a short recipe that pulls in your trunk branch, adds the packaging from Alvin’s existing Ubuntu package and then builds an installable Ubuntu package in the PPA of your choice:


# bzr-builder format 0.3 deb-version 2.0beta+{revno}
lp:alvin
nest-part packaging lp:ubuntu/alvin debian debian


Paste the recipe into Launchpad and with a couple of clicks you have a daily build of your trunk, that’s published as an Ubuntu package in your PPA.

Now you can ask people to test the latest Alvin code by doing no more than adding your PPA to their system. Launchpad will build a new version of the package on each day it spots a change in your trunk (or the Ubuntu packaging). For your beta testers, any changes will show up just like any other Ubuntu update.

Simple as that!

Here’s a quick recap of how it works: you can take any buildable branch — whether hosted in directly Launchpad or imported from Git, Subversion, CVS or Bazaar hosted elsewhere — merge or nest other branches, add packaging and then leave it to Launchpad to create a daily build that it publishes in your chosen PPA.

Seeing it in action

During the beta, people added a whole range of source package recipes, with a list of more than 350 daily builds as I write this.

Daily builds on Launchpad right now include Project Neon, who have around sixty recipes providing daily builds of KDE and Amarok. There are also daily builds of the Scribus DTP app, Audacity and the scriptable screen reader Gnome Orca.

Try it out

It’s easy to get your own source package recipes and daily builds up and running.

Start at our Getting Started guide and screencast.

I’ll leave the last word to Luke Benstead, who has been using source package recipes while developing a set of game libraries:

I’ve been using LP to develop some small open source game libraries. Because there are quite a few of them, packaging them all is a pain, so the package builds have worked out pretty well for them.

Now I get nightly builds delivered to a PPA, so I know that if I fix a bug it’s reflected to all my machines. And my recipes are only a single line so they’ve been really easy to use. I’m not really sure how they could be easier.

Images:
Beer photo by dearbarbie. CC-BY-SA.
Alvin Hall photo by Phil Guest. CC-BY-SA.

You may start getting “Failed to fetch” error messages when updating your software sources (e.g. through “apt-get update” or “Reload package information” in Synaptic), which may be due to a bug we’ve just cleaned up in Launchpad’s PPAs.

The error looks like this:

  W: Failed to fetch http://ppa.launchpad.net/chromium-daily/ppa/ubuntu/dists/maverick/Release
  Unable to find expected entry  restricted/binary-i386/Packages in Meta-index file (malformed Release file?)

  E: Some index files failed to download, they have been ignored, or old ones used instead.

(more…)

An long-requested feature in Launchpad is to let people see who’s using a PPA. Finally, we’ve implemented this!

Initially, the stats are only available on Launchpad’s webservice API. but we aim to show something useful in the web UI at some point.

If you are already familiar with the webservice API, then you can use the following binary_package_publishing_history object methods to retrieve the information:

• getDailyDownloadTotals
• getDownloadCount
• getDownloadCounts

Fabien Tassin is already using the stats to see how many people are using his daily build PPAs, and wrote an interesting blog post about it.