launchpadblog

Launchpad will be offline for about 90 minutes while we roll out some changes to the database and to code hosting.

Following the SourceForge compromise, the SSL certificate for SourceForge subversion changed.

A bug on their end prevents us from automatically importing from their SVN repositories.

Until that bug is resolved, all SourceForge code imports are disabled.

2011/02/14 update: We re-enabled SF code imports since the problem was fixed on their end.

We’ve made some changes to how we organise the Launchpad team!

We’re no longer divided into application-based teams (Bugs, Code, Foundations, Registry, Soyuz and Translations). Instead, we now have five cross-domain engineering squads: three focused on features and two on
maintenance.

How does this affect you?

When you want to speak to someone about a specific part of Launchpad,
whether for help, to escalate something or about an operational issue,
things have changed a little.

Specifically:

• For help and any other operational issues related to Launchpad, you should either send an email to the launchpad-users mailing list, or file a question on the Launchpad project.  (Reminder: these forums are public.)
• If your request would benefit from interactive discussions, drop by in#launchpad on Freenode where one of the people working on the maintenance squads will be able to help you. If your request isn’t suitable for public consumption, you can reach the same people on the #launchpad-ops channel onthe private Canonical IRC server.
• To prioritize feature requests and to escalate other bugs, you should contact our product strategist.
• You can also always bounce ideas to the developers list.

Short-term, we also expect some churn as people are exposed to areas they weren’t used to before. But down-the-line, we’ll have much more distributed knowledge coverage across the whole application.

How the squads work

The squads will alternate between “development project” and “maintenance” modes.

The three development squads will work on longer term projects, usually resulting in new functionality. Once such a squad has finished a project they’ll swap places with one of the two maintenance squads.

Bugs, operational issues and so on will be taken care of by the maintenance squads.

Deciding which development projects to take on will remain the responsibility of our strategist (Jonathan Lange) in collaboration with the Launchpad Stakeholders group.

You can find a list of who is in each squad on our dev wiki.

Why are we doing this?

Our app-based teams served us well, but were becoming a liability:

• Many parts of Launchpad, such as Blueprint, didn’t have a dedicated team and were basically unmaintained.
• Each team was responsible for both new features within their app as wel as maintenance. That slowed both of these.
• Projects that required cross-app integration suffered from hand-off and coordination problem.

With this new structure, we expect to see:

• Better cohesion across the application, as one squad will be responsible for the implementation of new aspects across the whole application.
• Reduced cycle-time both for bug fixes and for new features as the context-switching will be removed.

It will be my pleasure to answer any questions you might have regarding this reorg.

Due to a hardware failure, bazaar.launchpad.net is offline at the moment. We expect it will be back up and working correctly in about an hour or two from now. (That is, by about 17:00 UTC on the 21st of January.)

This affects all bzr-branch-related services: ssh and http access to branches, and also browsing of branches.

If you saw an “incorrect data check” error or similar from bzr while branching from Launchpad, this is probably the reason why, and the problem should be fixed soon.

Update: Code hosting is now back online. We’re sorry for the inconvenience.

We restored team polls because several Ubuntu teams require their use in their charter. They cannot easily switch to another service because it is not possible to organise the members to vote. We decided that to restore them while we decide what to do.

Option one: Contributors re-invent the UI so that setting up a poll is uncomplicated by silly restrictions and requirements. Many teams are not using Launchpad because they want condorcet polls. Launchpad may require them to keep the feature.

Option two: provide the information teams need to use another poll service. Team members can hide their email addresses, so it is not possible for the team admins to gather the member information to setup a poll. Honouring member privacy may require choosing or setting up a poll service that uses OpenId.

There is an unstated issue that options one and two do not address. It is also not possible to contact every member of team in Launchpad. How do team members ever know when a team admin creates a poll? Regardless of if the poll is in Launchpad, or in another service, members are unlikely to participate unless they happen to see the poll while visiting the team page in Launchpad. How often does that happen? Why would I visit my team’s page? I visit the teams I admin because I review membership proposals, moderate list mail and check on PPAs. I have no reason to visit the page of a team that I am just a member of.

We’re coming to the end of the first week of Launchpad’s 2010 Bug Jam. How’s it gone so far?

At the time of writing:

• Jono’s BugJam-o-Meter shows 77 bugs fixed as part of the jam
• most of those 77 have been fixes, with 19 Fix Released and 24 Fix Committed
• just one bug has been marked with the experimental Opinion status during the bug jam
• the hottest bug is 297505: Warn users that mailing list archives are public unless the team is actually private
• the oldest bug tackled is 33203: poll: dates are not properly validated, which was reported on March 1st 2006

Join us in #launchpad-dev on FreeNode or on the launchpad-dev list.

From today, all Launchpad bugs, code, questions and blueprints are tracked under the one launchpad project.

We’ve already moved everything from the individual projects over to the parent launchpad project. All you need do differently is search/file bugs, questions and blueprints under that parent Launchpad project, rather than Rosetta, for example.

Don’t worry, though, there are redirects in place so that old links will still work.

There are also a couple of one-time steps you may need to take:

• Update your bug subscriptions: if you’re subscribed to individual bugs, you need do nothing. If you’re subscribed to all bugs for a particular project, Malone for example, you’ll now need to subscribe to all Launchpad bugs.
• Check your answer contact status: if you’re an answer contact for one particular application in Launchpad, and want to continue as such, you’ll need to become an answer contact for all of Launchpad.

To start with, bugs that we’ve merged in from one of the old sub-projects will have a tag that shows which project it came from. However, we’re planning to drop those tags once everyone’s settled into using just the one project.

Our code hosting won’t change at all as we’ve always hosted code under the parent Launchpad project.

This new approach will better reflect that Launchpad is one codebase but will also have a big practical benefit: it’ll be easier to find bugs and dupes because everything will be under the same project.

Why we’re doing this

For almost four years now, Canonical’s Launchpad team has been divided along application lines: i.e. we have sub-teams who each look after a particular part of Launchpad. So, Deryck, Abel, Gavin and Graham are currently the Launchpad Bugs team and work on nothing other than Launchpad’s bug tracker.

Reflecting this team structure in our Launchpad projects has made it easier for those sub-teams to plan their work.

It has worked pretty well but we’re about to change the structure of Canonical’s Launchpad team for a couple of reasons:

• we want to focus on releasing features, and fixing problems, wherever they are
• we want all Canonical Launchpad developers to be familiar with the full Launchpad codebase, rather than focusing only on one part.

So, as of February 17th the Canonical Launchpad team will have five squads. At any one time, three of those squads will each be working on a particular feature and the other two will be working on maintenance. Once a feature squad finishes its feature, it’ll switch places with one of the maintenance squads.

This will mean that there’ll always be ten Canonical Launchpad developers dedicated to fixing bugs, dealing with critical issues and generally making sure that Launchpad is serving you well. And of course there’ll be fifteen developers working on new features.

Rather than make this post even longer, I’ll write more soon and in the meantime point you to Rob Collins’ rousing launchpad-developers post in favour of the new structure.

As ever, if you have questions then please join us on the launchpad-developers mailing list or feel free to contact me directly.

TL;DR

This is a pretty long post. Here’s the important bit, if you don’t want to read the rest.

”’For all Launchpad status updates/notices:”’ use our launchpadstatus identi.ca account (RSS feed, Twitter mirror).

”’For notice of major interruptions only:”’ subscribe to the launchpad-announce mailing list or the Launchpad blog’s notification category.

You can also use the Launchpad release schedule Google calendar (ical).

Improving how we announce service disruption

Over the past few months we in the Launchpad community have been moving toward releasing features with no downtime.

We can already push out new features when they’re ready, rather than as part of a monthly code push. However, we still have some service interruption: what was our monthly code release is now a roll-out of changes to the database’s structure. And, of course, there are times when we need to disrupt to Launchpad’s service in order to maintain hardware and so on.

The way that we announce such planned service interruptions has been a topic of discussion lately in the Launchpad community. I wanted to get a feel for what people beyond the Launchpad development community would prefer and so last week I
posted a survey asking how people would rather we make such announcements.

Before I look at what people said, I’ll note that 126 people completed the survey and that the results are really nothing more than a discussion point; the respondents were self-selecting and there’s no way to know if that 126 people were representative.

What people do now

The people who responded to the survey told us that, right now, they get Launchpad service status information in these ways:

Some of you reading may wonder why email isn’t mentioned. For some time we’ve had a launchpad-announce list specifically for announcing Launchpad service interruptions. However, until now that list hasn’t been well publicised.

What people said they want

In the survey, I asked what method people would prefer to use to get information about Launchpad service interruptions. Rather than limit what people could suggest, I gave an open text box.

The responses divide fairly neatly into six categories.

So, what should we do in light of this?

How we’re going to announce service interruptions in future

We have, broadly, two types of service interruption:

• localised disruptions that most people won’t notice: e.g. code browse offline for less than five minutes
• widespread disruptions that could inconvenience thousands of people: e.g. Launchpad read-only for 90 minutes.

Here’s how we’ll announce them from now on:.

Localised disruptions:

• identi.ca/Twitter notice at least 24 hours in advance
• #launchpad irc topic message at least 24 hours in advance

Widespread disruptions:

• One week before:
  • Email to launchpad-announce
  • Post to the Launchpad blog notifications category
  • Announce on identi.ca/Twitter
  • Ensure the Launchpad release schedule calendar (ical) is correct
• 24 hours before:
  • Reminder post to identi.ca/Twitter
  • Message in #launchpad topic
• Five to ten minutes before:
  • Message at the top of Launchpad pages
  • Reminder post to identi.ca/Twitter

We’ll also look at ways of improving the notification system in Launchpad itself, although any changes are still some way off.

Of course, we’re open to suggestions for how to better announce such disruptions. Please do leave your comments here.

I’m running a short survey — four questions — to find out how people want to get system status updates about Launchpad.

If you’ve got an opinion on how you’d prefer to get info about pending and unplanned Launchpad service interruptions, take the survey 🙂

I previously posted about our continuous deployment efforts in Launchpad. Since then the project has come a long way. We can deploy to nearly all our services without downtime. The remaining services are a bit trickier – but we are working on them.

As part of the project we are consolidating the ‘edge’ domain – https://edge.launchpad.net/, https://bugs.edge.launchpad.net/ and other similar domains – into the main launchpad UI. These domains are now deprecated.

The most important thing this means for you is that for members of our beta test program, we will no longer redirect you to https://edge.launchpad.net/ – instead we are serving our beta UI directly from the main website. The edge site is now running exactly the same code as the main Launchpad cluster and is updated at exactly the same time.

We have done this to deliver new features to our users more efficiently and at the same time simplify our production environment. So far the project has been very successful from our perspective – as I write this we have 5 days of inventory – code we’ve written but not deployed. This is down from an average of 2 weeks prior to this initiative starting, and we often sit lower – 1 to 2 days worth.

In the coming months as we refine this process and project we want to remove the edge cluster. As part of this we will start redirecting browser requests to ‘edge’ domains to the main Launchpad domain.

API clients cannot be redirected in this way, so we also ask that anyone writing or using Launchpad API scripts update them to use the primary cluster. We will slowly decrease the cluster size and disable it completely once we see no traffic on it. The main cluster is currently 3 times the size and should perform better for nearly any API script. To do this, use LPNET_SERVICE_ROOT rather than EDGE_SERVICE_ROOT. To get the LPNET_SERVICE_ROOT symbol, import it from launchpadlib.uris:

from launchpadlib.uris import LPNET_SERVICE_ROOT

If you have any questions about any of this we’d be delighted to hear from you – here, on IRC or the launchpad-user mailing list.

Rob Collins
Technical Architect