Cjwatson on Launchpad Blog

Self-service riscv64 builds

Colin Watson — Wed, 22 Nov 2023 14:00:43 +0000

Launchpad has supported building for riscv64 for a while, since it was a requirement to get Ubuntu’s riscv64 port going. We don’t actually have riscv64 hardware in our datacentre, since we’d need server-class hardware with the hypervisor extension and that’s still in its infancy; instead, we do full-system emulation of riscv64 on beefy amd64 hardware using qemu. This has worked well enough for a while, although it isn’t exactly fast.

The biggest problem with our setup wasn’t so much performance, though; it was that we were just using a bunch of manually-provisioned virtual machines, and they weren’t being reset to a clean state between builds. As a result, it would have been possible for a malicious build to compromise future builds on the same builder: it would only need a chroot or container escape. This violated our standard security model for builders, in which each build runs in an isolated ephemeral VM, and each VM is destroyed and restarted from a clean image at the end of every build. As a result, we had to limit the set of people who were allowed to have riscv64 builds on Launchpad, and we had to restrict things like snap recipes to only use very tightly-pinned parts from elsewhere on the internet (pinning is often a good idea anyway, but at an infrastructural level it isn’t something we need to require on other architectures).

We’ve wanted to bring this onto the same footing as our other architectures for some time. In Canonical’s most recent product development cycle, we worked with the OpenStack team to get riscv64 emulation support into nova, and installed a backport of this on our newest internal cloud region. This almost took care of the problem. However, Launchpad builder images start out as standard Ubuntu cloud images, which on riscv64 are only available from Ubuntu 22.04 LTS onwards; in testing 22.04-based VMs on other relatively slow architectures we already knew that we were seeing some mysterious hangs in snap recipe builds. Figuring this out blocked us for some time, and involved some pretty intensive debugging of the “strace absolutely everything in sight and see if anything sensible falls out” variety. We eventually narrowed this down to a LXD bug and were at least able to provide a workaround, at which point bringing up new builders was easy.

As a result, you can now enable riscv64 builds for yourself in your PPAs or snap recipes. Visit the PPA and follow the “Change details” link, or visit the snap recipe and follow the “Edit snap package” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out, including riscv64. This now means that all Ubuntu architectures are fully virtualized and unrestricted in Launchpad, making it easier for developers to experiment.

New domain names for PPAs

Colin Watson — Wed, 16 Feb 2022 18:08:27 +0000

Since they were introduced in 2007, Launchpad’s Personal Package Archives (PPAs) have always been hosted on ppa.launchpad.net. This has generally worked well, but one significant snag became clear later on: it was difficult to add HTTPS support for PPAs due to the way that cookies work on the web.

Launchpad uses a cookie for your login session, which is of course security-critical, and because we use multiple domain names for the main web application (bugs.launchpad.net, code.launchpad.net, and so on), the session cookie domain has to be set to allow subdomains of launchpad.net. We set the “Secure” flag on session cookies to ensure that browsers only ever send them over HTTPS, as well as the “HttpOnly” flag to prevent direct access to it from JavaScript; but there are still ways in which arbitrary JS on an HTTPS subdomain of launchpad.net might be able to exfiltrate or abuse users’ session cookies. As a result, we can never allow any HTTPS subdomain of launchpad.net to publish completely user-generated HTML that we don’t process first.

We don’t currently know of a way to get ppa.launchpad.net to serve arbitrary HTML as Content-Type: text/html, but this is quite a brittle protection as there are certainly ways (used for things like installer uploads) to upload arbitrary files to ppa.launchpad.net under a user-controlled directory structure, and we don’t want the webapp’s security to depend on nobody figuring out how to convince a browser to interpret any of that as arbitrary HTML. The librarian is already on a separate launchpadlibrarian.net domain name for a similar reason.

To resolve this dilemma, we’ve added a new ppa.launchpadcontent.net domain name which supports both HTTP and HTTPS (and similarly private-ppa.launchpadcontent.net for private PPAs, which as before is HTTPS-only). add-apt-repository in Ubuntu 22.04 will use the new domain name by default.

The old names will carry on working indefinitely - we know they’re embedded in lots of configuration and scripts, and we have no inclination to break all of those - but we recommend moving to the new names where possible. ppa.launchpad.net will remain HTTP-only.

Some systems may need to be updated to support the new domain name, particularly things like HTTP(S) proxy configuration files and no_proxy environment variables.

Login regression for users with non-ASCII names

Colin Watson — Thu, 20 Aug 2020 10:01:59 +0000

On 2020-08-13, we deployed an update that caused users whose full names contain non-ASCII characters (which is of course very common) to be unable to log into Launchpad. We heard about this serious regression from users on 2020-08-17, and rolled out a fix on 2020-08-18. We’re sorry about this; it doesn’t meet the standards of both inclusion and quality that we set for ourselves. This post aims to explain what happened, technical details of why it happened, and the steps we’ve taken to avoid it happening again.

Launchpad still runs on Python 2. This is a problem, and we’ve been gradually chipping away at it for the last couple of years. With about three-quarters of a million lines of Python code in the main tree and over 200 dependencies, it’s a big job - but we’re well underway!

Some of those dependencies have been difficult problems in their own right. The one at issue here was python-openid, which we use as part of our login workflow, but which hasn’t been actively maintained for over ten years. Fortunately, in this case we didn’t have to port it ourselves, because there were already a couple of forks featuring Python 3 support while preserving more or less the same interface: we chose python-openid2 on the grounds that it had done a good job of maintaining both Python 2 and 3 support in the same codebase, which we needed in order to arrange a practical transition, and that it was in itself well-maintained. We worked with upstream to fix a couple of issues discovered by the Launchpad test suite that blocked us migrating to it (notably PR #41, although that was fixed as PR #43 instead), and switched Launchpad over once python-openid2 3.2 was released. So far, so good.

One of the major reasons for much of the disruption in the Python 3 transition was to provide a clean separation between the concept of a sequence of bytes and a text string, which was often a problem for code that needed to handle Unicode: it’s all too common in Python 2 to have code that works on the ASCII domain (which can be represented either as str or unicode) but that fails on Unicode strings outside that subset. Launchpad is less prone to that than many Python 2 applications because the ORM we use (Storm) has always been relatively strict about the boundary between bytes and text; nevertheless, having a stricter data model here is a good thing for us in the long term. It might seem ironic that we ran into exactly such a bug as part of porting to Python 3; but then, we aren’t using the new interpreter yet.

Launchpad uses the OpenID Simple Registration Extension in its login workflow. It specifically requests the user’s full name from Canonical’s OpenID provider (login.ubuntu.com, which we generally call “SSO”): this means that if the user has an SSO account but not yet a Launchpad account, we can create a Launchpad account for them without them needing to enter their name again. That full name is encoded as a UTF-8 string, which in turn is URL-encoded using the usual %xx mechanism. This means that if, say, your name is Gráinne Ní Mháille, it will show up in the OpenID response’s query string as openid.sreg.fullname=Gr%C3%A1inne+N%C3%AD+Mh%C3%A1ille.

python-openid2 uses its openid.urinorm module to normalise parts of the response, decoding and re-encoding it to make sure comparisons work as expected; this is built on top of the URL handling code in Python’s standard library. Now, unlike Python 3, Python 2’s urlencode has undocumented restrictions on values in the query argument: if the doseq argument is False (the default), then it converts values using str(v), while if it’s True then it converts Unicode values using v.encode("ASCII", "replace") (potentially losing information!). In this case, doseq is False, and the input given to it is always text (unicode on Python 2): this works fine if the input is within the ASCII subset, but if it’s not:

>>> urlencode({u'openid.sreg.fullname': u'Gráinne Ní Mháille'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/urllib.py", line 1350, in urlencode
    v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

The fix is that on Python 2 one must always pass values to urlencode as bytes rather than text:

>>> urlencode({u'openid.sreg.fullname': u'Gráinne Ní Mháille'.encode('UTF-8')})
'openid.sreg.fullname=Gr%C3%A1inne+N%C3%AD+Mh%C3%A1ille'

We’ve sent PR #47 to python-openid2 to implement this. We’ve also made a temporary local fork of python-openid2 containing this patch and deployed it to Launchpad production.

One thing to be clear about here: though the root cause was a bug in python-openid2, it’s our responsibility to make sure it works correctly when integrated into Launchpad.

We missed this bug because of a gap in testing: although we did test the full login workflow, we only did so with a test user whose full name was entirely ASCII. We’ve closed this gap now, so we’ll catch it if a dependency regresses in future.

Bug emails now use the bug's address in the From: header

Colin Watson — Wed, 20 May 2020 14:43:11 +0000

The From: addresses used by Launchpad’s bug notifications have changed, to improve the chances of our messages being delivered over modern internet email.

Launchpad sends a lot of email, most of which is the result of Launchpad users performing some kind of action. For example, when somebody adds a comment to a bug, Launchpad sends that comment by email to everyone who’s subscribed to the bug.

Most of Launchpad was designed in an earlier era of internet email. In that era, it was perfectly reasonable to take the attitude that we were sending email on behalf of the user - in effect, being a fancy mail user agent or perhaps a little like a mailing list - and so if we generated an email that’s a direct result of something that a user did and consisting mostly of text they wrote, it made sense to put their email address in the From: header. Reply-To: was set so that replies would normally go to the appropriate place (the bug, in the case of bug notifications), but if somebody wanted to go to a bit of effort to start a private side conversation then it was easy to do so; and if email clients had automatic address books then those wouldn’t get confused because the address being used was a legitimate address belonging to the user in question.

Of course, some people always wanted to hide their addresses for obvious privacy reasons, so since 2006 Launchpad has had a “Hide my email address from other Launchpad users” switch (which you can set on your Change your personal details page), and since 2010 Launchpad has honoured this for bug notifications, so if you have that switch set then your bug comments will be sent out as something like “From: Your Name <bug-id@bugs.launchpad.net>”. This compromise worked tolerably well for a while.

But spammers and other bad actors ruin everything, and the internet email landscape has changed. It’s reasonably common now for operators of email domains to publish DMARC policies that require emails whose From: headers are within that domain to be authenticated in some way, and this is incompatible with the older approach. As a result, it’s been getting increasingly common for Launchpad bug notifications not to be delivered because they failed these authentication checks. Regardless of how justifiable our notification-sending practices were, we have to exist within the reality of internet email as it’s actually deployed.

So, thanks to a contribution from Thomas Ward, Launchpad now sends all its bug notifications as if the user in question had the “Hide my email address from other Launchpad users” switch set: that is, they’ll all appear as something like “From: Your Name <bug-id@bugs.launchpad.net>”. Over time we expect to extend this sort of approach to the other types of email that we send, possibly with different details depending on the situation.

Please let us know if this causes any strange behaviour in your email client. We may not be able to fix all of them, depending on how they interact with DMARC’s requirements, but we’d like to be aware of what’s going on.

Launchpad news, March 2019 - July 2019

Colin Watson — Tue, 06 Aug 2019 18:16:20 +0000

Here’s a brief changelog of what we’ve been up to since our last general update.

Bugs

Add basic GitLab bug linking (#1603679)
Expect the upstream bug ID in the “number” field of GitHub issue objects, not the “id” field (#1824728)
Include metadata-only bug changes in Person:+commentedbugs

Build farm

Filter ASCII NUL characters out of build logtails (#1831500)
Encode non-bytes subprocess arguments on Python 2 to avoid crashing on non-ASCII file names under LC_CTYPE=C (#1832072)

Code

Don’t preload recipe data when deleting recipes associated with branches or repositories, and add some more job indexes (#1793266, #1828062)
Fix crash if checkRefPermissions finds that the repository is nonexistent
Add a rescan button to branch merge proposals for failed branch or repository scans
Land parts of the work required for Git HTTPS push tokens, though this is not yet complete (#1824399)
Refactor code import authorisation to be clearer and safer
Set line-height on <pre> elements in Bazaar file views
Work in progress to redeploy Launchpad’s Git backend on more scalable infrastructure

Infrastructure

Upgrade to PostgreSQL 10
Fix make-lp-user, broken by the fix for #1576142
Use our own GPG key retrieval implementation when verifying signatures rather than relying on auto-key-retrieve
Give urlfetch a default timeout, fixing a regression in process-mail (#1820552)
Make test suite pass on Ubuntu 18.04
Retry webhook deliveries that respond with 4xx for an hour rather than a day
Merge up to a current version of Storm
Upgrade to Celery 4.1.1
Move development sites from .dev to .test
Upgrade to Twisted 19.2.1
Upgrade to requests 2.22.0
Use defusedxml to parse untrusted XML
Improve caching of several delegated authorization checks (#1834625)

Registry

Fix redaction in pillar listings of projects for which the user only has LimitedView (#1650430)
Tighten up the permitted pattern for newly-chosen usernames

Snappy

Landed parts of the work required to support private snap builds, though this is not yet complete (#1639975)
Generalise snap channel handling slightly, allowing channel selection for core16 and core18
Add build-aux/snap/snapcraft.yaml to the list of possible snapcraft.yaml paths (#1805219)
Add build-request-id and build-request-timestamp to SNAPCRAFT_IMAGE_INFO
Allow selecting source snap channels when requesting manual snap builds (#1791265)
Push build start timestamps to the store, and use release intents so that builds are more reliably released to channels in the proper sequence (#1684529)
Try to manually resolve symlinks in remote Git repositories when fetching snapcraft.yaml (#1797366)
Consistently commit transactions in SnapStoreUploadJob (#1833424)
Use build request jobs for all snap build requests in the web UI
Honour “base: bare” and “build-base” when requesting snap builds (#1819196)

Soyuz (package management)

Add command-not-found metadata in the archive to the Release file
Check the .deb format using dpkg-deb rather than ar
Add s390x Secure Initial Program Load signing support (#1829749)
Add u-boot Flat Image Tree signing support (#1831942)
Use timeout(1) to limit debdiff rather than using alarm(3) ourselves
Allow configuring the binary file retention period of a LiveFS (#1832477)
Import source packages from Debian bullseye

Launchpad news, February 2019

Colin Watson — Thu, 07 Mar 2019 16:32:09 +0000

Here’s a brief changelog for this month.

Build farm

Allow dispatching builds with base images selected based on the pocket and/or using LXD images instead of chroot tarballs where appropriate (#1811677)

Code

Store bzr-svn’s cache in the import data store
Allow project owners to use the Bazaar branch rescan view
Canonicalise expected rule ordering in GitRepository.setRules (#1815431)
Upgrade to pygit2 0.27.4 (#1815517)

Infrastructure

Use full gpg key fingerprints in rocketfuel-setup (contributed by Andy Brody; #1814206)

Snappy

Support snap bases (#1812985)

Soyuz (package management)

Allow source .changes files to omit the Binary field (#1813037)

Launchpad news, July 2018 - January 2019

Colin Watson — Thu, 21 Feb 2019 23:28:42 +0000

Here’s a brief changelog of what we’ve been up to since our last general update.

Bugs

Parse a few more possible Savane URL formats (#197250)
Compare Bugzilla versions properly when checking whether they support the Bugzilla API (part of #1802798)

Build farm

Configure snap proxy settings for Subversion (#1668358)
Support passing IMAGE_TARGETS, REPO_SNAPSHOT_STAMP, and COHORT_KEY variables into live filesystem builds
Set SNAPCRAFT_BUILD_ENVIRONMENT=host when building snaps (#1791201)
Prevent gathering results of large builds from blocking responses to XML-RPC requests (#1795877)
Add missing indexes on LiveFSFile(libraryfile) and SnapFile(libraryfile)
Direct build failure support to Launchpad Answers rather than to the launchpad-buildd-admins team (#1810001)

Code

Allow proposing merges between different branches of the same personal Git repository
Fix OOPS when trying to look up ~user/project:branch as a unique Git repository name (#1771118)
Optimise GitRepository.fetchRefCommits if there are no commits to fetch
Handle the case where a Bazaar branch and a Git repository have the same identity URL when creating a recipe (#1623924)
Push code imports over bzr+ssh rather than sftp (#1779572)
Fix crash when emailing inline comments on a diff with non-ASCII characters in hunk headers (#1787587)
Percent-encode reference names in GitRef URLs (#1787965)
Add active reviews link to Git-based project pages (#1777102)
Fix handling of non-ASCII ref names
Include the appropriate username in git+ssh:// URLs in the UI
Add instructions on creating personal Git repositories to people’s “View Git repositories” pages (#1590560)
Add available review targets and proposals to the Git repository overview page (#1789847)
Fix incorrect visibility check that broke code imports targeted at private Git repositories (#1789424)
Allow anonymous users to view votes for public merge proposals (#1786474)
Make Git ref scan jobs for repositories with large numbers of refs take much less memory
Tolerate backend timeouts while fetching commit information for GitRef:+index (#1804395)
Add Git per-branch permissions (#1517559)
Add rescan buttons when various kinds of code scanning jobs fail (#1808320)

Infrastructure

Convert all remaining code to use explicit proxy configuration settings rather than picking up a proxy from the environment, making the effective production settings easier to understand
Add support for ECDSA SSH keys (#907675)

Libraries

wadllib 1.3.3: Reimplement MIME multipart/form-data encoding without binary corruption bugs (#1729754)
lazr.restfulclient 0.14.2: Improve compatibility with various httplib2 versions (#1803558)

Registry

Add a suspend-bot-account.py script to suspend an account by email address
Weaken type of key_text in Person.deleteSSHKeysFromSSO so that more existing keys can be deleted (#1780411)
Fix SSHKey.getFullKeyText to not crash on some corrupt keys (#1798046)
Various improvements to the close-account script

Snappy

Extract initial Snap.store_name from snapcraft.yaml for Bazaar as well as Git
Add support for Snapcraft’s architectures keyword (#1770400)
Bump SnapStoreUploadJob.max_retries to 30 to allow for longer store scan times
Include the registered store package name for a snap recipe in its builds’ titles if it exists and differs from the snap recipe name
Move some metadata from SnapStoreUploadJob to SnapBuild, to prevent store upload jobs getting into states that cannot be retried

Soyuz (package management)

Add Archive.getSigningKeyData, currently just proxying through to the keyserver (#1667725)
Add extendedKeyUsage information to kmod signing keys so that they can only be used to sign modules, not boot loaders or kernels (#1774746)

Git per-branch permissions

Colin Watson — Thu, 10 Jan 2019 20:41:08 +0000

We’ve had Git hosting support in Launchpad for a few years now. One thing that some users asked for, particularly larger users such as the Ubuntu kernel team, was the ability to set up per-branch push permissions for their repositories. Today we rolled out the last piece of this work.

Launchpad’s default behaviour is that repository owners may push anything to their own repositories, including creating new branches, force-pushing (rewriting history), and deleting branches, while nobody else may push anything. Repository owners can now also choose to protect branches or tags, either individually or using wildcard rules. If a branch is protected, then by default repository owners can only create or push it but cannot force-push or delete; if a tag is protected, then by default repository owners can create it but cannot move or delete it.

You can also allow selected contributors to push to protected branches or tags, so if you’re collaborating with somebody on a branch and just want to be able to quickly pair-program via git push, or you want a merge robot to be able to land merge proposals in your repository without having to add it to the team that owns the repository and thus give it privileges it doesn’t need, then this feature may be for you.

There’s some initial documentation on our help site, and here’s a screenshot of a repository that’s been set up to give a contributor push access to a single branch:

Launchpad news, June 2018

Colin Watson — Fri, 06 Jul 2018 00:18:52 +0000

Here’s a brief changelog for this month.

Bugs

Handle Bugzilla.time() changes in Bugzilla 5.1.1 (#1774838)
Cope with the comment author field being renamed to creator in recent Bugzilla versions (#1774838)

Build farm

Set the hostname and FQDN of LXD containers to match the host system, though with an IP address pointing to the container (#1747015)
If the extra build arguments include fast_cleanup: True, then skip the final cleanup steps of the build; this can be used when building in a VM that is guaranteed to be torn down after the build
Allow checking out a git tag rather than a branch (#1687078, forum post)
Add a local unauthenticated proxy on port 8222, which proxies through to the remote authenticated proxy; this should allow running a wider range of network clients, since some of them apparently don’t support authenticated proxies very well (#1690834, #1753340, forum post)
Run tar with correct working directory when building source tarballs for snaps

Code

Port the loggerhead (Bazaar code browser) integration to gunicorn, allowing it to be used as an internal API as well
Optimise BuildableDistroSeries.findSeries (#1778732)
Proxy loggerhead branch diffs through the webapp, allowing AJAX MP revision diffs to work for private branches (#904070)

Infrastructure

Convert most code to use explicit proxy configuration settings rather than picking up a proxy from the environment, making the effective production settings easier to understand

Registry

Fix crash while adding an ssh key with unknown type (#1777507)

Miscellaneous

Improve documentation of what deactivating an account does (#993153)

Launchpad news, May 2018

Colin Watson — Sat, 02 Jun 2018 19:44:45 +0000

Here’s a brief changelog for this month.

Build farm

Send fast_cleanup: True to virtualised builds, since they can safely skip the final cleanup steps

Code

Add spam controls to code review comments (#1680746)
Only consider the most recent nine successful builds when estimating recipe build durations (#1770121)
Make updated code import emails more informative

Infrastructure

Upgrade to Twisted 17.9.0
Get the test suite passing on Ubuntu 18.04 LTS
Allow admins to configure users such that unsigned email from them will be rejected, as a spam defence (#1714967)

Snappy

Prune old snap files that have been uploaded to the store; this cleaned up about 5TB of librarian space
Make the snap store client cope with a few more edge cases (#1766911)
Allow branches in snap store channel names (#1754405)

Soyuz (package management)

Add DistroArchSeries.setChrootFromBuild, allowing setting a chroot from a file produced by a live filesystem build
Disambiguate URLs to source package files in the face of filename clashes in imported archives
Optimise SourcePackagePublishingHistory:+listing-archive-extra (#1769979)

Miscellaneous

Disable purchasing of new commercial subscriptions; existing customers have been contacted, and people with questions about this can contact Canonical support
Various minor revisions to the terms of service from Canonical’s legal department, and a clearer data privacy policy