Launchpad news, August 2015
Published by Colin Watson September 2, 2015 in General
Here’s a summary of what the Launchpad team got up to in August.
Launchpad news, July 2015
Published by Colin Watson August 2, 2015 in General
Here’s a summary of what the Launchpad team got up to in July.
Improved filtering options for Gmail users
Published by Colin Watson July 29, 2015 in Notifications
Users of some email clients, particularly Gmail, have long had a problem filtering mail from Launchpad effectively. We put lots of useful information into our message headers so that heavy users of Launchpad can automatically filter email into different folders. Unfortunately, Gmail and some other clients do not support filtering mail on arbitrary headers, only on message bodies and on certain pre-defined headers such as Subject. Figuring out what to do about this has been tricky. Space in the Subject line is at a premium – many clients will only show a certain number of characters at the start, and so inserting filtering tags at the start would crowd out other useful information, so we don’t want to do that; and in general we want to avoid burdening one group of users with workarounds for the benefit of another group because that doesn’t scale very well, so we had to approach this with some care.
As of our most recent code update, you’ll find a new setting on your “Change your personal details” page:
If you check “Include filtering information in email footers”, Launchpad will duplicate some information from message headers into the signature part (below the dash-dash-space line) of message bodies: any “X-Launchpad-Something: value” header will turn into a “Launchpad-Something: value” line in the footer. Since it’s below the signature marker, it should be relatively unobtrusive, but is still searchable. You can search or filter for these in Gmail by putting the key/value pair in double quotes, like this:
At the moment this only works for emails related to Bazaar branches, Git repositories, merge proposals, and build failures. We intend to extend this to a few other categories soon, particularly bug mail and package upload notifications. If you particularly need this feature to work for some other category of email sent by Launchpad, please file a bug to let us know.
Launchpad news, April-June 2015
Published by Colin Watson July 9, 2015 in Code, Cool new stuff, General, PPA
It’s been a while since we posted much regularly on this team blog, not least because for a while Launchpad was running more or less in maintenance mode. That’s no longer the case and we’re back to the point where we can do feature development work again, as exemplified by our recent addition of Git code hosting support.
Lots of other things have been happening in the Launchpad world lately, though, and the half-way point in the year seems like a good time to start talking about them. I’m going to try to do this a bit more regularly, aiming for about once a month when we also update our internal stakeholders. This post covers roughly the last three months.
Git code hosting beta
Published by William Grant May 1, 2015 in Code, Cool new stuff, General
Today we’re announcing early support for hosting Git repositories directly on Launchpad, as well as or instead of Bazaar branches. This has been by far the single most commonly requested feature from Launchpad code hosting for a long time; we’ve been working hard on it for several months now, and we’re very happy to be able to release it for general use.
This is distinct from the facility to import code from Git (and some other systems) into Bazaar that Launchpad has included for many years. Code imports are useful to aggregate information from all over the free software ecosystem in a unified way, which has always been one of the primary goals of Launchpad, and in the future we may add the facility to import code into Git as well. However, what we’re releasing today is native support: you can use git push to upload code to Launchpad, and your users and collaborators can use git clone to download it, in the same kind of way that you can with any Git server.
Our support is still in its early stages, and we still have several features to add to bring it up to parity with Bazaar hosting in Launchpad, as well as generally making it easier and more pleasant to use. We’ve released it before it’s completely polished because many people are clamouring to be able to use it and we’re ready to let you all do so. From here on in, we’ll be adding features, applying polish, and fixing bugs using Launchpad’s normal iterative deployment process: changes will be rolled out to production once they’re ready, so you’ll see the UI gradually improving over time.
What’s supported?
You can:
- push Git repositories to Launchpad over SSH
- clone repositories over git://, SSH, or HTTPS
- see summary information on repositories and the branches they contain in the Launchpad web UI
- follow links from the Launchpad web UI to a full-featured code browser (cgit)
- push and clone private repositories, if you have a commercial subscription to Launchpad
- propose merges from one branch to another, including in a different repository, provided that they are against the same project or package
What will be supported later?
Launchpad’s Bazaar support has grown many features over the years, and it will take some time to bring our Git support up to full parity with it and beyond. Git repositories use a somewhat different model from Bazaar branches, which we’ve had to account for in many places, and some facilities will require complete reimplementation before we can support them with Git.
Here’s an incomplete list of some of the features we hope to add:
- useful subscriptions (currently only attribute change notifications work, which are not usually very interesting in themselves)
- RSS feeds
- mirroring
- webhooks
- an integrated code browser
See our help page for more known issues and instructions on using Launchpad with Git.
Helping out
This is a new service, and we welcome your feedback: you can ask questions in #launchpad on freenode IRC, on our launchpad-users mailing list, or on Launchpad Answers, and if you find a bug then please tell us about that too.
Launchpad is free software, licensed under the GNU AGPLv3. We’d be very happy to mentor people who want to help out with parts of this service, or to build things on top of it using our published API. Some preliminary documentation on this is on our developer wiki, and you can always contact us for help.
Launchpad build farm improvements
Published by Colin Watson September 2, 2013 in General
We’ve made a number of improvements to the Launchpad build farm in the last month, with the aim of improving its performance and robustness. This sort of work is usually invisible to users except when something goes wrong, so we thought it would be worth taking some time to give you a summary. Some of this work was on the Launchpad software itself, while some was on the launchpad.net hardware.
(To understand some of the rest of this post, it’s useful to be aware of the distinction between virtualised and devirtualised builders in Launchpad. Virtualised builders are used for most PPAs: they build untrusted code in a Xen guest which is initialised from scratch at the start of each build, and are only available for i386, amd64, and a small number of ARM builds by way of user-mode QEMU. Devirtualised builders run on ordinary hardware with less strict containment, and are used for Ubuntu distribution builds and a few specialised PPAs.)
ARM builders have been a headache for some time. For our devirtualised builders, we were using a farm of PandaBoards, having previously used BeagleBoards and Babbage boards. These largely did the job, but they’re really a development board rather than server-class hardware, and it showed in places: disk performance wasn’t up to our needs and we saw build failures due to data corruption much more frequently than we were comfortable with. We recently installed a cluster of Calxeda Highbank nodes, which have been performing much more reliably.
It has long been possible to cancel builds on virtualised builders: this is easy because we can just reset the guest. However, it was never possible to cancel builds on devirtualised builders: killing the top-level build process isn’t sufficient for builds that are stuck in various creative ways, and you need to make sure to go round and repeatedly kill all processes in the build chroot until they’ve all gone away. We’ve now hooked this up properly, and it is possible for build daemon maintainers to cancel builds on devirtualised builders without operator assistance, which should eliminate situations where we need urgent builds to jump the queue but can’t because all builders are occupied by long-running builds. (People with upload privileges can currently cancel builds too, which is intended mainly to allow cancelling your own builds; please don’t abuse this or we may need to tighten up the permissions.) As a bonus, cancelling a build no longer loses the build log.
Finally, we have been putting quite a bit of work into build farm reliability. A few problems have led to excessively long queues on virtual builders:
- Builders hung for some time when they should have timed out, due to a recent change in su; this is now fixed in the affected Ubuntu series.
- Xen guests often fail to restore for one reason or another, and when this happened builders would fail in ways that required an operator to fix. We had been dealing with this by having our operators do semi-automatic builder fixing runs a few times a day, but in recent months the frequency of failures has been difficult to keep up with in this way, especially at the weekend. Some of this is probably related to our current use of a rather old version of Xen, but the builder management code in Launchpad could also handle this much better by trying to reset the guest again in the same way that we do at the start of each build. As of this morning’s code deployment, we now do this, and the build farm seems to be holding up much more robustly.
This should make things better for everyone, but we aren’t planning to stop here. We’re intending to convert the virtual builders to an OpenStack deployment, which should allow us to scale them much more flexibly. We plan to take advantage of more reliable build cancellation to automatically cancel in-progress builds that have been superseded by new source uploads, so that we don’t spend resources on builds that will be rejected on upload. And we plan to move Ubuntu live file system building into Launchpad so that we can consolidate those two build farms and make better use of our available hardware.
Private Projects and Private Blueprints leave beta
Published by Deryck Hodge December 4, 2012 in General
Today, the Private Projects and Private Blueprints features on Launchpad are leaving beta. These features are available now for use by all Launchpad users. Private Blueprints was started as part of the Private Projects work, with the end goal in mind of truly private projects on Launchpad. Private Projects was described in its beta announcement like this:
When creating a new project on Launchpad, beta testers will have the option to create “Proprietary” or “Embargoed” projects. Embargoed exists for projects that intend to start private but later be revealed publicly. All other private projects should be proprietary. Milestones and series are proprietary or embargoed based on the project setting. To make them public, you will need to make the project itself public.
When you create a proprietary or embargoed project on Launchpad, all of the sharing policies for your project will be set correctly for you. This means that if you start your project as a proprietary project, your branches, bugs, and blueprints will be created proprietary by default. Answers and translations are not available for private projects.
A commercial subscription is required to use private projects, but any user who creates a proprietary or embargoed project on Launchpad will receive a 30 day trial commercial subscription. Launchpad users with existing commercial subscriptions can convert a public project to proprietary or embargoed by changing the information type in the project’s settings. You may have some work to do on your project before you can transition to a private information type — for example, disable answers if you have that app enabled for your project — but Launchpad will block the change and tell you what needs to happen before you can switch to a private information type.
Users should be aware, though, that if your project has been listed on Launchpad publicly until now, then search engines know of its existence already. If you want a proprietary project that no one can learn of its name, you should create a new project on Launchpad. Transitioning a public project to private allows you to keep your series and milestones private going forward, but users may have already been able to discover the existence of the project since it was public already.
We are happy to make truly private projects available for all users on Launchpad. If you run into any issues, please file a bug against Launchpad or ask for help in #launchpad on Freenode.
How Novacut uses Launchpad
Published by Laura czajkowski November 20, 2012 in General
Launchpad has been a key tool used in developing Novacut. I use Launchpad for code hosting, bug tracking, daily builds, and more. For almost two years I’ve been doing monthly stable releases on Launchpad, and Novacut now spans six separate Launchpad projects. To say the least, I’ve learned a lot about Launchpad in the process.
I don’t think Novacut could be where it is today without Launchpad, so I want to pass on some of what I’ve learned the past two years. Here are my five essential Launchpad best practices:
1. Daily Builds
I’m always very thankful that early on Paul Hummer took the time to school me on using Source Package Recipes to do daily builds. This Launchpad service gives you automated package builds across multiple architectures, and multiple Ubuntu releases.
I don’t know how to emphasize this enough, but seriously, you need daily builds. As a point of reference, daily builds are the 3rd item in the famed Joel Test.
These builds are triggered simply by making commits to the appropriate bzr branch on Launchpad (usually your trunk branch). You’ll automatically get up to one build per 24-hour period, and you can manually trigger additional builds when needed.
You can include your debian/ packaging directory in your project source tree, or you can keep debian/ in a separate bzr branch. For the Novacut components, I’ve found it most helpful to keep debian/ in the source trees because it’s handy to be able to land a code change and its corresponding packaging change in a single merge. This works for us because we currently can use the exact same debian/ for all the Ubuntu versions we support. If that’s not true for your project, you’ll need multiple debian/ branches.
For reference, here’s the Novacut Source Package Recipe.
2. Unit Tests
You should run your unit tests during your package builds, and you should fail the build when any unit test fails. This is particularly important for daily builds, because this will prevent a package with broken unit tests from reaching your daily PPA.
The Launchpad build servers are strict and unforgiving environments, which is a good thing when it comes to unit tests. The build servers are also probably quite different from your local development environment. On countless occasions our daily builds have caught failures that only occur on i386 (my workstation is amd64), or only occur on an Ubuntu release other than the one I’m running, etc.
To run your unit tests during the package build, you’ll need to modify your debian/rules file as appropriate. If you’re using debhelper, add an override_dh_auto_test target.
You might also need to add additional packages to the Build-Depends section of your debian/control file, packages that are needed by the unit tests but are otherwise not needed by the build itself.
For reference, here’s the debian/rules file used to run the Dmedia unit tests (which is also a handy Python3 example).
3. Track Ubuntu+1
When a new Ubuntu version opens up for development, I immediately start doing daily builds on the development version, even though I don’t typically upgrade my own computers till around 4 months into the cycle.
I use daily builds on the development release as an early warning system. With no extra effort on my part, these builds give me a heads-up about code or packaging changes that might be needed to make Novacut work well on the next Ubuntu release.
To enable daily builds on the next Ubuntu version, just go to your Source Package Recipe, click on “Distribution series”, and check the box for the newest series. Now you’ll have daily builds on the newest Ubuntu version, in addition to all the versions you were already building for.
For example, I’m currently in the process of enabling daily builds for Raring, as you can see in the Microfiber Source Package Recipe. And I did indeed encounter a build failure on Raring, seemingly caused by a debhelper issue.
For the first month or so in a cycle, I don’t tend to worry much about build failures on the development version. But after the dust has settled a bit, I make sure to keep the builds in working order, and I even do monthly stable releases for the Ubuntu development version. Again, I do all this pro-actively even before I personally start running the newest Ubuntu version.
4. PPAs & Users
Whenever someone asks me why I use Launchpad instead of github, my short answer is always, “PPAs and users”.
Source Package Recipes give you much more than just a build, they give you daily packages that are easily consumable by your testing community and early adopters. This tight feedback loop prevents you from running too far ahead without getting a good reality check from your target users.
Keep in mind that for some products, the early adopters willing to install from a PPA might not be all that representative of your target user. So when it comes to making design decisions, you might need to politely ignore certain feedback from some of these early adopters. In my experience, this wont cause any hard feelings as long as you have clearly communicated who your target user is, and why.
For reference, you might look at the way we’ve defined the Novacut target user.
I recommend creating PPA names that are well-branded and easy to remember. First, create a Launchpad team with the same name as your product. In our case, we have a ~novacut team. Second, I recommend creating a daily and a stable PPA owned by the same team. In our case, that gives us two easy to remember PPAs:
Although none of our target users (professional video editors) currently use Ubuntu to do their job, I’ve been surprised by how many follow Novacut’s development via our stable PPA, and even our daily PPA. This has helped keep us on track, and has helped us build customer loyalty even before we have a finished product.
For me personally, this daily user engagement also makes the design and development process more enjoyable. It’s hard to empathize with an abstract persona; it’s easier to solve specific problems for specific people.
5. Use Apport
Till recently I didn’t realize that you can use Apport for automated crash reporting in unofficial packages delivered through a PPA.
We haven’t had Apport integration for that long, but it’s already provided us with dozens of highly valuable crash reports. Almost immediately some hardware specific issues came to light and were fixed, convincing me that a key benefit of Apport is knowing how your app might misbehave on a larger, more variable pool of hardware.
Apport also helped some rare bugs come to light. I thought Dmedia was basically crash-free, but those one-in-a-thousand bugs pop out quickly when thousands of people are running it. Most of these bugs would have eventually been found by one of our core devs, but the quicker a bug is discovered, the quicker and easier the bug is to fix.
For more info, check out this blog post and this screencast, where I covered our Apport integration in detail.
And for reference, see the merge proposal that added Apport integration in Novacut.
A big thank you to Jason DeRose for sharing how his project uses Launchpad on a daily basis.
The information sharing feature is complete
Published by Curtis Hovey November 6, 2012 in General
Launchpad’s bug and branch privacy features were replaced by information sharing that permits project maintainers to share kinds of confidential information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.
The Disclosure features
Disclosure is a super feature composed on many features that will allow commercial projects to work in private. Untrusted users cannot see the project’s data. Project maintainers can share their project with trusted users to reveal all or just some of the project’s data. The ultimate goal is to create private project in Launchpad, but that feature required several other features to be completed first. The Purple squad worked on Trusted Pickers, Privacy Transitions, Hardened Projects, Social Private Teams, and Sharing.
There was a lot of overlap between each feature the Purple squad worked on. Though we could start each feature independent of one another, we could only complete about 90% of each. When the Sharing UI changes entered beta, we were unblocked and fixes about most of the remaining issues, but fixing all the issues required all projects to switch to Sharing. We did not consider Sharing, or any of the required features complete until we fixed all the bugs.
Disclosure facts
- Planning started in June 2010 to replace the existing privacy mechanisms with something that would scale.
- Early testing revealed that users did not trust Launchpad because the UI could not explain what was confidential, or what the consequences of a change would be — this needed to be fixed too.
- 149 related bugs were identified in Launchpad.
- Work started in June 2011 by the Purple squad.
- Replacing the old privacy mechanisms and addressing the trust and information issues took 16 months.
- About 45,000 lines code were added to support the features.
- About 15% of the lines were for missing JavaScript test coverage.
- More that 700 bugs were fixed in total.
- About 5% of the fixed bugs were caused by the old non-scaling privacy mechanisms.
- About 4% of the fixed bugs were caused by old JavaScript enhancements that broke features for non-JavaScript users.
Lessons learned
- Misrepresentation of what is confidential, or what will be confidential or public is very important to users — more important than supporting private data.
- Privacy/Sharing must be a first-class mechanism beneath all the mechanisms that work with confidential data.
- Privacy was added on top of bugs, and it failed to scale to 100’s of bugs.
- Privacy was added on top of branches, and it failed to scale to 1000’s of branches.
- Filtering private items in code, or in database joins is not fast enough to work with 100,000’s of items.
- Launchpad’s ReSTful object API is not suitable for working with large collections of objects like bugs or branches; a lighter, service-based approach was used to quickly work with large amounts of data.
- Users need to work with confidential data via the API, using a text web browser from servers, using a browser with accessibility tools, as well as the common case of using a JavaScript enabled browser.
- Lots of mock-ups and interactive tests will not predict all the interactions a user will have with real data; test with real code and data early to developer the final design.
Private projects for beta testers
Published by Deryck Hodge October 25, 2012 in General
If you are part of Launchpad’s beta testers team, you can now start trialing private projects on Launchpad. The private projects feature builds on the great sharing work that Launchpad’s Purple Squad has done, allowing Launchpad users to create true private projects now. A commercial subscription is required to use private projects, but any user who creates a private project on Launchpad will receive a 30 day trial commercial subscription.
When creating a new project on Launchpad, beta testers will have the option to create “Proprietary” or “Embargoed” projects. Embargoed exists for projects that intend to start private but later be revealed publicly. All other private projects should be proprietary. Milestones and series are proprietary or embargoed based on the project setting. To make them public, you will need to make the project itself public.
Be warned, this is a large change to Launchpad and there are certainly bugs in our handling of privacy. You can check out our list of known issues, if you’d like. We, as the Launchpad Orange Squad, are committed to fixing all of those before we leave beta. So don’t worry, we’re still actively working on this feature. We did, however, want users to begin using this feature to get early feedback on the work. Don’t trial your super secret project with this feature just yet, but if you have something safe to try out private projects, now is a good time for beta testers to get going with the feature.
Enjoy private projects on Launchpad now, beta testers! And please file any bugs you find.
