3

Continuous deployment in Launchpad

Published by Robert Collins October 5, 2010 in General

It currently takes an average of two weeks for new changes that have been developed for Launchpad to become live on the Launchpad site.

We’re working on changing this and making the way we deploy Launchpad simpler and more reliable at the same time.

In the first generation of this, we are targeting changes that do not alter the data model, and we’re aiming for a delay of 12-16 hours. Longer term we’ll be aiming for a few hours.

If you are a ‘beta‘ user of Launchpad, this has one primary, and very important change: the ‘edge’ site is going to be removed. We now have a process for validating changes that would previously have been validated on edge using a new staging site. The edge site previously received unvalidated updates and would from time to time have issues as a result. If you are not a ‘beta’ user, then nothing should change for you at all, except that you will notice site changes more often, with no downtime, rather than once a month after downtime.

Sometime in the next few weeks the redirect to ‘edge’ will be removed (it only affected beta users). Instead of a redirect to ‘edge’, the main website will offer you any unreleased functionality, and the ‘disable edge redirect’ link will turn off that functionality for a moderate time period. Following that we will put in place a redirect from ‘edge’ to the normal ‘launchpad.net’ across all of the ‘edge’ servers, and move the servers to the main site server farm.


1

Launchpad read-only 08.00 UTC 6th October 2010

Published by Matthew Revell in Notifications

Launchpad’s web interface will be read-only, with other aspects offline, for less than 15 minutes from 08.00 UTC on Wednesday the 6th of October. During this time we’ll be making upgrades to Launchpad’s database servers.

Starts: 08.00 UTC 6th October 2010
Expected back before: 08.15 UTC 6th October 2010

Roughly 24 hours after this read-only time there’ll be another short period where Launchpad is read-only. This will enable us to complete the upgrade work and we’ll announce the exact times separately.


2

More Build Farm Improvements

Published by Julian Edwards October 4, 2010 in Code, Cool new stuff, PPA

Continuing with the recent improvements to the build farm – Jelmer has made another massive one.

The last major scalability problem that we had was one where the whole farm was blocked when a single builder was ready to upload a build. In the case of large packages, like the kernel, the manager process could block for over a minute while it waited for the upload processor to unpack the package and verify its contents.

Jelmer’s work has decoupled the upload processing from the build farm manager process. What happens now is that the files collected from the builder are thrown into a staging queue area and then the manager process immediately continues with polling the rest of the builders, unblocked. A cron job will then process the builder upload queue at 1 minute intervals.

You can see the dramatic effect this has had on the overall queue for the PPA builders here:
Build Farm Queue Size

This is quite an incredible improvement as you can see! But we’re not stopping there, we’re currently doing a massive refactoring of the builder dispatching code so it’s all fully asynchronous. When this is all done we’re going to be in superb shape to support an increase in load that’s anticipated from the increasing number of people using the package recipes.


0

Code hosting/browsing unavailable 14.00 UTC 23rd September 2010

Published by Matthew Revell September 23, 2010 in Notifications

Launchpad’s code hosting and browsing will be offline for 15 minutes today for emergency hardware maintenance.

Going offline: 14.00 UTC 23rd September 2010
Expected back: 14.15 UTC 23rd September 2010

We’re sorry for the short notice of this downtime.


0

Meet Rob Collins

Published by Matthew Revell September 20, 2010 in Meet the devs

Rob Collins joined the Launchpad team recently. Here’s a bit about him.

Matthew: What do you do on the Launchpad team?

Rob: I’m the Architect for Launchpad – I’m responsible for guiding the development and technology choices we make so that Launchpad can meet the needs of our stakeholders, users, developers and system administrators.

Practically speaking this means that I do a bit of everything: if I don’t, I can’t really feel what’s going on across the system, and being well informed is key to making good decisions / giving good advice.

Matthew: Can we see something that you’ve worked on in Launchpad?

Rob: A few recent examples of my personal code are: email list moderation pages are now batched, so should time out much less often if a list gets a lot of mail to moderate; I’ve been poking at bugs and bug API performance quite a bit, and the /attachments API is significantly faster now; the /messages API is going to be faster soon — and the main bugs page has had several seconds of overhead shaved off it for many bugs. I’ve set up a page with some of the things I think will help us with building and running Launchpad as we move forward.

Matthew: Where do you work?

Rob: I work from home like so many of us do; I recently moved to Rangiora, NZ. Yes, we were 40km or so from the epicentre.

Matthew: What can you see from your office window?

Rob: Grass, fence, roofs and sky; shorly to be grass, hedge and sky 😉

Matthew: What did you do before joining the Launchpad team?

Rob: Most recently I was one of the developers of Bazaar at Canonical, working mainly on the deep guts of the system.

Matthew: You’ve dedicated your Tuesdays to performance issues in Launchpad. Tell us about some of the improvements you’ve made.

Rob: So the ones I’ve listed above are pretty nice.

Here’s a list of specific timeout bug fixes I’ve done. More importantly though, we’ve been working on better diagnostics gathering for timeouts, and that work is making things easier and easier to analyse.

Matthew: What else are you working on right now?

Rob: Release features when they are done — this change to how we develop and rollout launchpad is a key change to speeding up the delivery of features. It’s not the full story, because we don’t (yet) have a plan for dealing with schema changes without downtime — that’s in the slightly longer term pipeline.

Matthew: What’s the next big thing you want to tackle in Launchpad?

Rob: Downtime-free schema changes; this should permit us to deliver new functionality much more rapidly.

Matthew: What experiences from your time in the Bazaar team will help with your new-ish role?

Rob: Ah! I spot a leading question. One of the interesting differences between Bazaar and Launchpad is that nearly every operation in Bazaar exercises an entire stack from scratch: there’s no active DB server with loaded caches ready and primed to go. That really got me thinking, continually, about the overheads involved in the entire stack : I think we need that same thought and consideration in Launchpad as we work on performance and deployment.

Matthew: How can community contributors help with what you’re working on?

Rob: Many, many ways!

Firstly, there’s the current timeout bugs. All of these can be fixed by patches … the exact patch and how hard it is to write will of course vary 🙂

Many timeouts are actually pretty shallow issues — but others will require schema changes and so forth. I’m delighted to help mentor folk working on any part of Launchpad.

The process and deployment changes we’re doing are mainly a sysadmin/deployment issue and not as amenable to contribution (by anyone not one of our sysadmins).

Matthew: How did you get into free software?

Rob: Thats a long story 🙂 Gradual love affair starting with demo code back on an Amiga A500.

Matthew: What’s more important? Principle or pragmatism?

Rob: Case by case basis 😉

Matthew: What free software projects do you contribute to?

Rob: Currently I’m still swimming for the surface in Launchpad 😉

Recently I’ve contributed to:

The list going way back is longer … but also mainly of historical interest.

Matthew: Tell us something really cool about Launchpad that not enough people know about.

Rob: Recipes. Recipes are cool.


1

Everything in Launchpad

Published by Matthew Revell September 17, 2010 in General

Over the summer, Jono and I have been compiling a list of all the features in Launchpad.

While the help wiki and the heads of various members of the Launchpad team are a pretty good guide to everything that’s in Launchpad, we haven’t had a canonical, comprehensive, list of Launchpad’s features. Obviously having that one page makes it easier to keep track of what’s there and to think about what Launchpad is and what it isn’t.

So, what do we consider to be a feature? Really, it’s anything where someone can interact with Launchpad or where bugs can live. Simple as that.

If you think something’s missing from this list, or needs more explanation, please do go ahead edit the wiki page.

Launchpad’s feature list.


30

What’s your favourite project in Launchpad?

Published by Matthew Revell September 14, 2010 in General

You’ve probably seen the featured projects list on Launchpad’s home page:

featured-projects

As you can see, there’s a wide variety of projects in that list, and each of them is an excellent example of how Launchpad can help free software projects. However, we’ve had pretty much the same projects in that list for a few months now so perhaps time to give some of the other 19,444 Launchpad projects a slot in the list.

So, it’s over to you: leave a comment here nominating which Launchpad-using project(s) you want to see in the featured projects list. If you have time, let us know why.


0

Recent posts from Planet Launchpad

Published by Matthew Revell September 10, 2010 in General

Just a quick look at some of the recent posts to Planet Launchpad:


0

Launchpad unavailable 9th September 2010 08.00 UTC

Published by Matthew Revell September 7, 2010 in Notifications

On Thursday the 9th of September we’re rolling out the latest Launchpad code to our servers. At the same time we’re taking the opportunity to carry out some server maintenance.

This work will take around three hours from 08.00 UTC and will include a 90 minute period of complete downtime, followed by 90 minutes of Launchpad in a read-only* state.

Downtime starts: 08.00 UTC 9th September 2010
Launchpad returns in read-only mode: 09.30 UTC 9th September 2010
Launchpad expected to return to normal: 11.00 UTC 9th September 2010.

We’re sorry for the relatively short notice of this service disruption.

* In read-only mode, Launchpad’s web interface is available for browsing. Other aspects of Launchpad, such as uploading to PPAs and pushing to code branches, are offline.


0

Code hosting maintenance Friday, September 3

Published by Gary Poster September 2, 2010 in Notifications

Launchpad code hosting will be offline Friday between 8.00 and 9.30 UTC for unexpected hardware maintenance. This means you won’t be able to browse, push to, pull from or otherwise access code hosted on Launchpad.

Going offline: 8.00 UTC 3rd September 2010

Expected back: 9.30 UTC 3rd September 2010


Previous Entries
Next Entries