launchpadblog

It’s been a while since we posted much regularly on this team blog, not least because for a while Launchpad was running more or less in maintenance mode.  That’s no longer the case and we’re back to the point where we can do feature development work again, as exemplified by our recent addition of Git code hosting support.

Lots of other things have been happening in the Launchpad world lately, though, and the half-way point in the year seems like a good time to start talking about them.  I’m going to try to do this a bit more regularly, aiming for about once a month when we also update our internal stakeholders.  This post covers roughly the last three months.

Git

Of course, you don’t get to release a major new feature and then have everything be perfect.  In fact, we’d released it pretty much as soon as we had a minimum viable product, so we knew there was plenty more to do.

Now that the basics work reasonably well, we’ve been focusing on tying off loose ends, and on getting to the point where Launchpad itself can be self-hosted in Git: this is partly because most of the things we need to do for that are relevant to many other projects considering a migration, partly because that ensures that any problems with it will affect Launchpad developers directly, and partly because this will allow us to trim a very large amount of database cruft associated with the representation of Launchpad’s own branches.

Here’s a brief changelog since our initial public announcement:

• Leaving the “Target reference path” field empty when proposing a Git-based merge proposal no longer crashes (#1451068)
• Git-based merge proposals can be deleted
• Pushing to private repositories no longer hangs (#1451107)
• The backend now stores reflogs so that we can help users recover from mistakes
• Pushing a new default repository for a project now sets the repository owner to the project owner rather than to the person performing the push
• The project +sharing view now handles Git repositories
• Branches of a project default repository are now shown on the project’s code page, although we expect to be making significant further improvements in this area soon
• Individual Git commits under “Recent Commits” now link directly to a full view of the corresponding commit
• Merge proposal listings now cover Git (#1453020)
• Branches in private repositories now show a privacy banner (#1457553)
• Repositories can now be deleted (#1456583)
• Private repositories are now browseable and show a privacy banner in the browsing interface
• The URLs for Git-based merge proposals have moved to be directly under the source repository rather than under the source branch, so that historical MPs can still be accessed after deleting the branch (#1456589)
• Launchpad now sends mail for Git-based merge proposals
• There are various new Git repository listing views, chiefly for projects, packages, and people
• Repositories now have a complete edit form allowing you to rename them, move them between targets, and set their default branch (“HEAD”) (#1456625)
• We’ve exported a few more bits and pieces on the webservice API to support Git merge robots
• We’ve brought various bits of automation for Git-based merge proposals up to par with their Bazaar equivalents: when a Git branch is updated, Launchpad now automatically updates the preview diffs for any MPs where that is the source branch, and detects whether any MPs where that is the target branch can now be marked as Merged
• The Code settings view for projects now allows you to set the project’s default Git repository and to choose whether the default revision system for the project is Bazaar or Git

We’ve started on support for webhooks, and hope to be able to tell you more about this soon. Git-to-Git repository mirroring and native Git recipes are also frequent requests that we know are important to people, and we hope to get to those soon.

Other code review changes

Many of the changes needed for Git touched on our code review infrastructure, but we’ve been making some other improvements there too.  Launchpad has supported “inline comments” on merge proposals for a year or so now, allowing you to comment on parts of a diff without having to manually quote it; but the feature hadn’t been quite finished and lots of people had nits to pick with it.  We’ve made a couple of improvements here lately:

• Preview diffs in merge proposals now have keyboard navigation to move between files, diff hunks, and inline comments
• Links to diffs now include a per-file diffstat under an expander triangle
• The mails sent when a reviewer adds inline comments now only include relevant parts of the diff context rather than the entire diff, making them much easier to read and making it harder to miss comments (#1334577)

We threw in a couple of bonus features here as well.  You can now see side-by-side diffs for merge proposals as well as unified diffs (#917502): there’s a link at the top of each preview diff allowing you to switch it to side-by-side mode (we may add a user preference for this later).  And resubmitting a merge proposal now automatically preserves the commit message (#676769).

Package build infrastructure

When we first added support for building debug symbol packages (ddebs) in Ubuntu, we handled the build and publication side of things with a temporary hack involving a custom hack to our sbuild fork that stashed the ddebs on the builder for a while, and a job that periodically fetched them to ddebs.ubuntu.com. As is the way of temporary hacks, it proceeded to stay that way for eight and a half years. It mostly worked, but was rather fragile, contributed to tying us to an old sbuild, wasn’t very extensible to PPAs, and couldn’t work with virtualised builders where all the builder’s state is reset before every build.

A couple of years ago we put together the Launchpad changes required to store ddebs straightforwardly in Launchpad instead, but weren’t able to deploy this at the time due to needing better librarian infrastructure. We finally got all this cleared out in April and deployed the new ddeb publishing mechanism. The main user-visible effect of this is that ddebs.ubuntu.com isn’t liable to lose packages any more, and ddebs are also available from Launchpad build pages.

Having done this, we were able to upgrade from our 2004-era fork of sbuild to a modern version, which fixed a number of long-standing bugs.

All architectures are now optional for PPAs (#1244868), so, for example, a PPA that only needs to support armhf can save builder cycles by disabling amd64 and i386. At the moment this can only be done by Launchpad administrators, so ask a question on Launchpad if you think this would be worth doing for any of your PPAs.

We rewrote the way that we install build-dependencies for source package recipes. This allows recipes to handle the “:native” qualifier in build-dependencies which is used in some places as part of supporting cross-building.

The next step in this modernisation programme will be to move all builds into ScalingStack and decommission the corresponding bare-metal builders, which we’ll be starting soon and working on an architecture-by-architecture basis as that cloud gains support for them. In the short term, this will give us better build capacity on the migrated architectures; in the longer term, it will let us support PPA builds on more architectures, and erase the cumbersome distinction between “non-virtualised” and “virtualised” builders. More news on this as it happens.

Performance

We upgraded to significantly more modern database servers earlier this year, which meant that a lot of hitherto difficult timeout problems suddenly either disappeared or became tractable. There are still some bad queries and a few mysterious problems, but as a general trend our timeout rates are very significantly down from where they were six months ago. We have some more general plans in this area, and will continue to spot-fix bad pages as they show up and as time permits.

Epilogue

We have lots more to do across the whole application, but still have a rather limited number of developers. If any of this kind of thing sounds interesting and you’d like to help, you can!

Tags: front-page

This entry was posted by Colin Watson on Thursday, July 9th, 2015 at 1:07 am and is filed under Code, Cool new stuff, General, PPA. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.