More Build Farm Improvements
Published by Julian Edwards October 4, 2010 in Code, Cool new stuff, PPA
Continuing with the recent improvements to the build farm – Jelmer has made another massive one.
The last major scalability problem that we had was one where the whole farm was blocked when a single builder was ready to upload a build. In the case of large packages, like the kernel, the manager process could block for over a minute while it waited for the upload processor to unpack the package and verify its contents.
Jelmer’s work has decoupled the upload processing from the build farm manager process. What happens now is that the files collected from the builder are thrown into a staging queue area and then the manager process immediately continues with polling the rest of the builders, unblocked. A cron job will then process the builder upload queue at 1 minute intervals.
You can see the dramatic effect this has had on the overall queue for the PPA builders here:
This is quite an incredible improvement as you can see! But we’re not stopping there, we’re currently doing a massive refactoring of the builder dispatching code so it’s all fully asynchronous. When this is all done we’re going to be in superb shape to support an increase in load that’s anticipated from the increasing number of people using the package recipes.
Code hosting/browsing unavailable 14.00 UTC 23rd September 2010
Published by Matthew Revell September 23, 2010 in Notifications
Launchpad’s code hosting and browsing will be offline for 15 minutes today for emergency hardware maintenance.
Going offline: 14.00 UTC 23rd September 2010
Expected back: 14.15 UTC 23rd September 2010
We’re sorry for the short notice of this downtime.
Meet Rob Collins
Published by Matthew Revell September 20, 2010 in Meet the devs
Rob Collins joined the Launchpad team recently. Here’s a bit about him.
Matthew: What do you do on the Launchpad team?
Rob: I’m the Architect for Launchpad – I’m responsible for guiding the development and technology choices we make so that Launchpad can meet the needs of our stakeholders, users, developers and system administrators.
Practically speaking this means that I do a bit of everything: if I don’t, I can’t really feel what’s going on across the system, and being well informed is key to making good decisions / giving good advice.
Matthew: Can we see something that you’ve worked on in Launchpad?
Rob: A few recent examples of my personal code are: email list moderation pages are now batched, so should time out much less often if a list gets a lot of mail to moderate; I’ve been poking at bugs and bug API performance quite a bit, and the /attachments API is significantly faster now; the /messages API is going to be faster soon — and the main bugs page has had several seconds of overhead shaved off it for many bugs. I’ve set up a page with some of the things I think will help us with building and running Launchpad as we move forward.
Matthew: Where do you work?
Rob: I work from home like so many of us do; I recently moved to Rangiora, NZ. Yes, we were 40km or so from the epicentre.
Matthew: What can you see from your office window?
Rob: Grass, fence, roofs and sky; shorly to be grass, hedge and sky 😉
Matthew: What did you do before joining the Launchpad team?
Rob: Most recently I was one of the developers of Bazaar at Canonical, working mainly on the deep guts of the system.
Matthew: You’ve dedicated your Tuesdays to performance issues in Launchpad. Tell us about some of the improvements you’ve made.
Rob: So the ones I’ve listed above are pretty nice.
Here’s a list of specific timeout bug fixes I’ve done. More importantly though, we’ve been working on better diagnostics gathering for timeouts, and that work is making things easier and easier to analyse.
Matthew: What else are you working on right now?
Rob: Release features when they are done — this change to how we develop and rollout launchpad is a key change to speeding up the delivery of features. It’s not the full story, because we don’t (yet) have a plan for dealing with schema changes without downtime — that’s in the slightly longer term pipeline.
Matthew: What’s the next big thing you want to tackle in Launchpad?
Rob: Downtime-free schema changes; this should permit us to deliver new functionality much more rapidly.
Matthew: What experiences from your time in the Bazaar team will help with your new-ish role?
Rob: Ah! I spot a leading question. One of the interesting differences between Bazaar and Launchpad is that nearly every operation in Bazaar exercises an entire stack from scratch: there’s no active DB server with loaded caches ready and primed to go. That really got me thinking, continually, about the overheads involved in the entire stack : I think we need that same thought and consideration in Launchpad as we work on performance and deployment.
Matthew: How can community contributors help with what you’re working on?
Rob: Many, many ways!
Firstly, there’s the current timeout bugs. All of these can be fixed by patches … the exact patch and how hard it is to write will of course vary 🙂
Many timeouts are actually pretty shallow issues — but others will require schema changes and so forth. I’m delighted to help mentor folk working on any part of Launchpad.
The process and deployment changes we’re doing are mainly a sysadmin/deployment issue and not as amenable to contribution (by anyone not one of our sysadmins).
Matthew: How did you get into free software?
Rob: Thats a long story 🙂 Gradual love affair starting with demo code back on an Amiga A500.
Matthew: What’s more important? Principle or pragmatism?
Rob: Case by case basis 😉
Matthew: What free software projects do you contribute to?
Rob: Currently I’m still swimming for the surface in Launchpad 😉
Recently I’ve contributed to:
- Launchpad
- Bazaar
- A variety of python testing projects:
- testtools,testresources,testscenarios,fixtures,testrepository,
unittest in Python itself - Storm
- Ubuntu and Debian
The list going way back is longer … but also mainly of historical interest.
Matthew: Tell us something really cool about Launchpad that not enough people know about.
Rob: Recipes. Recipes are cool.
Everything in Launchpad
Published by Matthew Revell September 17, 2010 in General
Over the summer, Jono and I have been compiling a list of all the features in Launchpad.
While the help wiki and the heads of various members of the Launchpad team are a pretty good guide to everything that’s in Launchpad, we haven’t had a canonical, comprehensive, list of Launchpad’s features. Obviously having that one page makes it easier to keep track of what’s there and to think about what Launchpad is and what it isn’t.
So, what do we consider to be a feature? Really, it’s anything where someone can interact with Launchpad or where bugs can live. Simple as that.
If you think something’s missing from this list, or needs more explanation, please do go ahead edit the wiki page.
What’s your favourite project in Launchpad?
Published by Matthew Revell September 14, 2010 in General
You’ve probably seen the featured projects list on Launchpad’s home page:
As you can see, there’s a wide variety of projects in that list, and each of them is an excellent example of how Launchpad can help free software projects. However, we’ve had pretty much the same projects in that list for a few months now so perhaps time to give some of the other 19,444 Launchpad projects a slot in the list.
So, it’s over to you: leave a comment here nominating which Launchpad-using project(s) you want to see in the featured projects list. If you have time, let us know why.
Recent posts from Planet Launchpad
Published by Matthew Revell September 10, 2010 in General
Just a quick look at some of the recent posts to Planet Launchpad:
- Julian wants to get people using the PPA SFTP upload service
- Deryck’s looking for help with the bug import script
- Brad writes about the new ways that project pages indicate much progress you’ve made in filling out your project’s details
- Curtis writes about remixing Ubuntu using Launchpad
Launchpad unavailable 9th September 2010 08.00 UTC
Published by Matthew Revell September 7, 2010 in Notifications
On Thursday the 9th of September we’re rolling out the latest Launchpad code to our servers. At the same time we’re taking the opportunity to carry out some server maintenance.
This work will take around three hours from 08.00 UTC and will include a 90 minute period of complete downtime, followed by 90 minutes of Launchpad in a read-only* state.
Downtime starts: 08.00 UTC 9th September 2010
Launchpad returns in read-only mode: 09.30 UTC 9th September 2010
Launchpad expected to return to normal: 11.00 UTC 9th September 2010.
We’re sorry for the relatively short notice of this service disruption.
* In read-only mode, Launchpad’s web interface is available for browsing. Other aspects of Launchpad, such as uploading to PPAs and pushing to code branches, are offline.
Code hosting maintenance Friday, September 3
Published by Gary Poster September 2, 2010 in Notifications
Launchpad code hosting will be offline Friday between 8.00 and 9.30 UTC for unexpected hardware maintenance. This means you won’t be able to browse, push to, pull from or otherwise access code hosted on Launchpad.
Going offline: 8.00 UTC 3rd September 2010
Expected back: 9.30 UTC 3rd September 2010
Meet Jon Sackett
Published by Matthew Revell August 24, 2010 in Meet the devs
Jon Sackett joined the Launchpad Registry team a couple of weeks ago. Here’s a quick run-down of who he is.
Matthew: What do you do on the Launchpad team?
Jon: I’m part of the Registry team; we maintain the people, teams and projects bits and pieces used by all the other parts of Launchpad.
Right now I’m mostly helping pay down technical debt, but I’m also helping with features that help those core objects be smarter about the way they use other applications.
Matthew: Can we see something that you’ve worked on?
Jon: Almost everything I’ve done has been internal without a real UI component.
Matthew: Where do you work?
Jon: I work in my home office in an apartment in downtown Durham, NC. Sometimes I change it up and work from my porch.
Matthew: What can you see from your office window?
Jon: The old brickface and industrial windows across the road. On days where I’m working from my porch I get a better view of the downtown
Matthew: What did you do before working at Canonical?
Jon: I worked as a Python/Django developer at a company called MetaMetrics, that does some really neat things in education with natural language processing.
Matthew: How did you get into free software?
Jon: I was introduced to Linux in college as a better environment for coding in my CS classes. Since moving into web programming and Python, I think almost every tool I use has come from free software.
Matthew: What’s more important? Principle or pragmatism?
Jon: In concrete matters (like code), pragmatism. It’s no use to anyone if your principles only prevent you from doing things.
That said, principles are still important; when you opt for the pragmatic approach, your principles can still influence how
that plays out.
Matthew: Do you/have you contribute(d) to any free software projects?
Jon: Sadly, precious little. I have a patch in the Django project, and a couple of my own projects are available under a BSD license. One of the reasons I wanted to work on Launchpad was to do more with and for free software.
Matthew: Tell us something really cool about Launchpad that not enough people know about.
Jon: How completely well it supports the whole development lifecycle — I think a lot of people consider Launchpad just another code hosting service, and it’s so much more than that.
Matthew: Thanks Jon!
Dupes of dupes now become dupes of the master bug
Published by Matthew Revell August 13, 2010 in Bug Tracking, Cool new stuff
Some bugs get reported more than once. That’s why we’ve got the dupe finder.
Some duplicate bugs slip through the dupe finder. Really common issues get quite a few dupes and someone from the relevant project usually goes through and marks them as duplicates of the master bug where the actual discussion and tracking is taking place.
There has been a really annoying bug in the way Launchpad has handled all this, though, and Deryck‘s just fixed it 🙂
Let’s say you’ve got a bug report that has a few duplicates attached to it. You then discover that, actually, there’s an older bug with a more mature discussion and that, really, that’s the master bug. Until now, before you marked your bug as a duplicate of the master, you’d first have to take all the dupes of your bug and manually make them dupes of the master.
Still with me? 🙂
For a busy bug with many dupes, some of which have their own dupes, that’s a real disincentive to clearing up the multiple duplicates and gathering everything together on the one true master bug.
Now, though, simply mark your bug as a duplicate and Launchpad will automatically transfer your bug’s dupes to the new master bug.
Simple 🙂
