launchpadblog

Continuing with the recent improvements to the build farm – Jelmer has made another massive one.

The last major scalability problem that we had was one where the whole farm was blocked when a single builder was ready to upload a build. In the case of large packages, like the kernel, the manager process could block for over a minute while it waited for the upload processor to unpack the package and verify its contents.

Jelmer’s work has decoupled the upload processing from the build farm manager process. What happens now is that the files collected from the builder are thrown into a staging queue area and then the manager process immediately continues with polling the rest of the builders, unblocked. A cron job will then process the builder upload queue at 1 minute intervals.

You can see the dramatic effect this has had on the overall queue for the PPA builders here:

This is quite an incredible improvement as you can see! But we’re not stopping there, we’re currently doing a massive refactoring of the builder dispatching code so it’s all fully asynchronous. When this is all done we’re going to be in superb shape to support an increase in load that’s anticipated from the increasing number of people using the package recipes.

Rob Collins joined the Launchpad team recently. Here’s a bit about him.

Matthew: What do you do on the Launchpad team?

Rob: I’m the Architect for Launchpad – I’m responsible for guiding the development and technology choices we make so that Launchpad can meet the needs of our stakeholders, users, developers and system administrators.

Practically speaking this means that I do a bit of everything: if I don’t, I can’t really feel what’s going on across the system, and being well informed is key to making good decisions / giving good advice.

Matthew: Can we see something that you’ve worked on in Launchpad?

Rob: A few recent examples of my personal code are: email list moderation pages are now batched, so should time out much less often if a list gets a lot of mail to moderate; I’ve been poking at bugs and bug API performance quite a bit, and the /attachments API is significantly faster now; the /messages API is going to be faster soon — and the main bugs page has had several seconds of overhead shaved off it for many bugs. I’ve set up a page with some of the things I think will help us with building and running Launchpad as we move forward.

Matthew: Where do you work?

Rob: I work from home like so many of us do; I recently moved to Rangiora, NZ. Yes, we were 40km or so from the epicentre.

Matthew: What can you see from your office window?

Rob: Grass, fence, roofs and sky; shorly to be grass, hedge and sky 😉

Matthew: What did you do before joining the Launchpad team?

Rob: Most recently I was one of the developers of Bazaar at Canonical, working mainly on the deep guts of the system.

Matthew: You’ve dedicated your Tuesdays to performance issues in Launchpad. Tell us about some of the improvements you’ve made.

Rob: So the ones I’ve listed above are pretty nice.

Here’s a list of specific timeout bug fixes I’ve done. More importantly though, we’ve been working on better diagnostics gathering for timeouts, and that work is making things easier and easier to analyse.

Matthew: What else are you working on right now?

Rob: Release features when they are done — this change to how we develop and rollout launchpad is a key change to speeding up the delivery of features. It’s not the full story, because we don’t (yet) have a plan for dealing with schema changes without downtime — that’s in the slightly longer term pipeline.

Matthew: What’s the next big thing you want to tackle in Launchpad?

Rob: Downtime-free schema changes; this should permit us to deliver new functionality much more rapidly.

Matthew: What experiences from your time in the Bazaar team will help with your new-ish role?

Rob: Ah! I spot a leading question. One of the interesting differences between Bazaar and Launchpad is that nearly every operation in Bazaar exercises an entire stack from scratch: there’s no active DB server with loaded caches ready and primed to go. That really got me thinking, continually, about the overheads involved in the entire stack : I think we need that same thought and consideration in Launchpad as we work on performance and deployment.

Matthew: How can community contributors help with what you’re working on?

Rob: Many, many ways!

Firstly, there’s the current timeout bugs. All of these can be fixed by patches … the exact patch and how hard it is to write will of course vary 🙂

Many timeouts are actually pretty shallow issues — but others will require schema changes and so forth. I’m delighted to help mentor folk working on any part of Launchpad.

The process and deployment changes we’re doing are mainly a sysadmin/deployment issue and not as amenable to contribution (by anyone not one of our sysadmins).

Matthew: How did you get into free software?

Rob: Thats a long story 🙂 Gradual love affair starting with demo code back on an Amiga A500.

Matthew: What’s more important? Principle or pragmatism?

Rob: Case by case basis 😉

Matthew: What free software projects do you contribute to?

Rob: Currently I’m still swimming for the surface in Launchpad 😉

Recently I’ve contributed to:

• Launchpad
• Bazaar
• A variety of python testing projects:
• testtools,testresources,testscenarios,fixtures,testrepository,
  unittest in Python itself
• Storm
• Ubuntu and Debian

The list going way back is longer … but also mainly of historical interest.

Matthew: Tell us something really cool about Launchpad that not enough people know about.

Rob: Recipes. Recipes are cool.

Over the summer, Jono and I have been compiling a list of all the features in Launchpad.

While the help wiki and the heads of various members of the Launchpad team are a pretty good guide to everything that’s in Launchpad, we haven’t had a canonical, comprehensive, list of Launchpad’s features. Obviously having that one page makes it easier to keep track of what’s there and to think about what Launchpad is and what it isn’t.

So, what do we consider to be a feature? Really, it’s anything where someone can interact with Launchpad or where bugs can live. Simple as that.

If you think something’s missing from this list, or needs more explanation, please do go ahead edit the wiki page.

Launchpad’s feature list.

You’ve probably seen the featured projects list on Launchpad’s home page:

As you can see, there’s a wide variety of projects in that list, and each of them is an excellent example of how Launchpad can help free software projects. However, we’ve had pretty much the same projects in that list for a few months now so perhaps time to give some of the other 19,444 Launchpad projects a slot in the list.

So, it’s over to you: leave a comment here nominating which Launchpad-using project(s) you want to see in the featured projects list. If you have time, let us know why.

Jon Sackett joined the Launchpad Registry team a couple of weeks ago. Here’s a quick run-down of who he is.

Matthew: What do you do on the Launchpad team?

Jon: I’m part of the Registry team; we maintain the people, teams and projects bits and pieces used by all the other parts of Launchpad.

Right now I’m mostly helping pay down technical debt, but I’m also helping with features that help those core objects be smarter about the way they use other applications.

Matthew: Can we see something that you’ve worked on?

Jon: Almost everything I’ve done has been internal without a real UI component.

Matthew: Where do you work?

Jon: I work in my home office in an apartment in downtown Durham, NC. Sometimes I change it up and work from my porch.

Matthew: What can you see from your office window?

Jon: The old brickface and industrial windows across the road. On days where I’m working from my porch I get a better view of the downtown

Matthew: What did you do before working at Canonical?

Jon: I worked as a Python/Django developer at a company called MetaMetrics, that does some really neat things in education with natural language processing.

Matthew: How did you get into free software?

Jon: I was introduced to Linux in college as a better environment for coding in my CS classes. Since moving into web programming and Python, I think almost every tool I use has come from free software.

Matthew: What’s more important? Principle or pragmatism?

Jon: In concrete matters (like code), pragmatism. It’s no use to anyone if your principles only prevent you from doing things.

That said, principles are still important; when you opt for the pragmatic approach, your principles can still influence how
that plays out.

Matthew: Do you/have you contribute(d) to any free software projects?

Jon: Sadly, precious little. I have a patch in the Django project, and a couple of my own projects are available under a BSD license. One of the reasons I wanted to work on Launchpad was to do more with and for free software.

Matthew: Tell us something really cool about Launchpad that not enough people know about.

Jon: How completely well it supports the whole development lifecycle — I think a lot of people consider Launchpad just another code hosting service, and it’s so much more than that.

Matthew: Thanks Jon!

Some bugs get reported more than once. That’s why we’ve got the dupe finder.

Some duplicate bugs slip through the dupe finder. Really common issues get quite a few dupes and someone from the relevant project usually goes through and marks them as duplicates of the master bug where the actual discussion and tracking is taking place.

There has been a really annoying bug in the way Launchpad has handled all this, though, and Deryck‘s just fixed it 🙂

Let’s say you’ve got a bug report that has a few duplicates attached to it. You then discover that, actually, there’s an older bug with a more mature discussion and that, really, that’s the master bug. Until now, before you marked your bug as a duplicate of the master, you’d first have to take all the dupes of your bug and manually make them dupes of the master.

Still with me? 🙂

For a busy bug with many dupes, some of which have their own dupes, that’s a real disincentive to clearing up the multiple duplicates and gathering everything together on the one true master bug.

Now, though, simply mark your bug as a duplicate and Launchpad will automatically transfer your bug’s dupes to the new master bug.

Simple 🙂

Some bugs attract many, many comments.

For a while now, Launchpad has displayed only the first 80 comments on any bug report, with the option of viewing the full comment history. That’s been good for speeding up page loads but not so great at offering an accurate view of the current state of discussion about the bug.

Bryce has fixed that. Now, a bug report page still shows only 80 comments, by default. However, to give a better overview of the state of discussion, it now shows the first 40 and the last 40 comments.

So, half way down the comments you’d see something like:

Here's what you see in the middle of the bug comment history

Then, at the bottom of the summarised comment history there’s this:

Comment history summary

With any luck, this should result in new bug comments that take into account the most recent discussion.

One of my favourite things about Launchpad’s bug tracker is the dupe finder: when you report a new bug, it’ll search to see if there’s already a similar bug report. It’s the same for questions in Launchpad Answers, too.

Getting to see possible dupes before you file a bug or question is a great time saver for you and the people on the other end. However, the dupe finder has been timing out a lot lately.

Rob Collins, Launchpad’s new Technical Architect, has introduced some changes that should make the dupe finder more reliable.

Other than fewer timeouts, here’s what you might notice:

• the dupe finder now returns fewer matches — three or four rather than ten or more
• the results should be more relevant.

We want to know how this works in practice. Let us know how you get on with the new dupe finder. Either leave a comment here, mail feedback@launchpad.net or join us on the launchpad-users mailing list.

How Rob did it

The previous dupe finder had a number of problems, not least that the search engine it’s built on is less efficient than we need. We’re planning to replace the search engine but not straight away, so Rob looked for a temporary solution that would work for the next five or six months.

I’ll hand over to Rob to explain what he actually did:

The old search did a pre-pass over every possible hit, which is 400,000 items for Ubuntu bugs and very slow to do. It then did a search matching any document that had a rare search term in it.

So, by rare we mean that the term showed up in less than half of the possible hits.

For example, if you searched for “firefox crashes on <website> in flash” on /ubuntu/+filebug it would search for any bug with any of “firefox” (< 50% of bugs are on firefox), "crash" (<50% of bugs say "crash"), "<can switch this off easily if we have to, so we do want feedback about how people find this.

Improvements, you say?

Very recently we saw the beta release of a new feature on Launchpad: building packages from recipes. Recipes bring Bazaar branches for the upstream source code and a packaging branch together to generate a Debian source package. We informally referred to this internally as “the Wellington project” because we first embarked on this long road of development back in November 2009 with a coding sprint in Wellington, New Zealand.

So, why do we need to use the build farm for this?

Very early on we realised that this would be a challenging project because packaging recipes allow untrusted arbitrary code to run as part of building the package, so we decided that the only place we could do this operation was in the same place that we do one of the other untrusted operations – the PPA build machines.

The PPA build machines work for untrusted code because they run as a virtual machine. After the build job has finished, the machine is simply ripped out and restarted so if any code did something nasty, we throw all the nastiness away!

Why can’t the build farm do this already?

The Launchpad build farm has been around for a while now, but it’s of course only used for building source packages into binary debs. We realised that we’d need a whole new data model on the Launchpad side, changes to the master/slave protocol and changes in the slave code to make it actually run this new job type – a lot of work! It was also about this time I became aware of a further untrusted job type, generation of translation templates for Launchpad Translations, so we began work on making the build farm able to process any kind of job.

What had to change?

By far the biggest change for this was to fix how we store the build/job information in the database. The canonical way to refer to a “job” on the build farm was via “build” record, and this is what all of the history pages in the UI were using. We’ve now re-architected this to use a generic “build farm job” and a bunch of new database tables that re-factor all the information needed for various jobs on the build farm, in fact we found out that the recipe build jobs and the source package build jobs had quite a lot in common which resulted in some merciless code re-factoring indeed.

The other major change was the slave builder code itself. This runs as a Twisted executive that we talk to using XMLRPC from the build manager, which kicks off external processes to run the job and monitors them. The only job it knew about was to run sbuild to build source packages. We’ve changed this to also run bzr builder and a custom script to generate the translations templates.

So, will I see anything different?

A recipe build in action

You won’t see much visually that’s changed in the Build Farm. If you’re observant, you’ll see the occasional job appear in the list that announces it’s a recipe build, but they’re generally pretty quick to complete!

We’ll post a more detailed explanation of how to use recipe builds in the future, but for now it’s remaining in beta until all the problems are ironed out. If you have any feedback, please file a bug or send us an email.

Recently, we changed the way assigning bugs works on Launchpad. It used to be that anyone could assign anyone else to a bug. This was open to abuse as you can imagine. Bug 511269 was filed about the potential problems with this, and we recently changed Launchpad so that only bug supervisors can assign a bug to someone else.

You can still assign a bug to yourself, but this does keep you from assigning someone to a bug to draw their attention to said bug. In the end, this is a good thing, though, as people should only be assigned bugs who are going to be responsible for working on them.

Now there is one issue with this change. Projects that had not established a bug supervisor for the project will find their developers can no longer assign bugs to each other. The easy fix for this is to create a bug supervisor team for your project and have the people working on your bugs assigned to this team. We do realize this might be a bit heavy weight for some projects, so we’ve opened bug 603281 about this issue.  A fix for this — only requiring bug supervisor permissions if bug supervisor is defined — should be appearing on Launchpad soon.