launchpadblog

Congratulations to everyone in the Ubuntu community on the successful release of Ubuntu 11.04!

Here’s to the next six months of oneiric fun.

Photo by Sharon Terry. Licence: CC-BY-ND 2.0.

Rejoice Blueprint users for Tim has brought the glory of Ajax to the main blueprint page.

He writes:

Now we can update the following without reloading the primary page:

• title – the H1 heading
• summary
• whiteboard
• assignee
• drafter
• approver
• priority
• implementation status
• definition status

Using the new custom events that the page raises when the context object changes (using YUI magic and API PATCH requests), when you change the title of the blueprint, the document title (title bar) and the breadcrumbs also change. When the implementation status is updated, the overall status updates, and the “started by” and “completed by” are shown or hidden as appropriate.

This is work that I’ve wanted to see done for almost a year, and recent other changes I’ve done adding more widget wrappers and javascript goodness have made this possible without adding copious amounts of custom javascript.

A side-effect of these changes is that there are now more fields exported over the API for blueprints.

Photo by Will Scullin. Licence: CC BY 2.0

Last week I asked for suggestions of what should be on the Launchpad front-page featured project list.

Thanks for your suggestions! Based on your comments, I’ve added the following projects to the front page:

• OpenStack: the open source cloud platform
• Gufw: the GUI front-end for the UFW firewall configuration app
• Tomdroid: the read-only Tomboy client for Android
• District Health Information Software: health management software in use in many developing countries
• Unity: the desktop experience

I’m also going to look at better ways of show-casing the projects of Launchpad. I’ll blog with more info soon.

We are now two months away from our next Thunderdome. How are we doing in regards with the objectives set for that milestone? You may recall from my last post the objectives:

1. have no timeouts with a cut-off at 9s;
2. have an empty critical bugs queue;
3. getting a slot free on our ‘Next’ queue.

We practically achieved the first objective! Today, we lowered the hard timeout to 9s and this didn’t increase our number of daily timeouts. We don’t have zero timeouts yet. We still have a fair bunch of timeout bugs to fix. But we get on average 650 requests timing out in a day. That’s less than 0.01% of our traffic.

These remaining timeout bugs are part of our second objective. On that front, we are in a more difficult position. We have 259 critical bugs to close. That went up since last time! What went wrong? Well, we had less people working on critical bugs for once. That’s been fixed this week when the Orange squad rotated back on maintenance. We again have two full squads working on critical bugs. Second, we modified our OOPS reporting to show all timeouts happening, not only the ones occurring the most often. That resulted in about 30 new timeouts filed. (See the hight red bar at the start of the graph). Fortunately for us, the rate of new critical bugs is declining.  We are at about 23 on average in the last two weeks. That’s still high and some of those are related to JS regressions escaping to production because our Windmill test infrastructure is disabled. This means that 51 is now the magic number. We need to close 51 of these critical bugs per week to reach 0 by the Thunderdome. That was the number we closed in our best week, just before the number of people working on criticals was reduced. So we’ll also need to reduce the number of new critical bugs found each week to succeed here.

Finally, on our last objective, like I already mentioned, the Orange squad switched back to maintenance. That’s because they completed the final bits of the project they were working on: Sharing translations between upstreams and Ubuntu. (Expect a blog post about it next week once everything is available to the general public.) That means that we need to complete 5 more projects to free a slot in our Next queue. That will be a stretch, but not impossible since of the two currently in progress, one should be completed in a couple of weeks and the other before the end of next month.  And two of the remaining ones are much simpler than what we have worked on until now.

I’ll post again next month to see how the final stretch looks like.

Photo by Aslak Raanes. Licence: CC BY 2.0.

Have you seen that Launchpad is on Facebook?

If you want to get Launchpad feature and development news through Facebook, like our page.

There are also our Twitter and identi.ca accounts, if you prefer those.

What does “critical” mean? And how is it that the number of critical Launchpad bugs has leapt up to almost 300?

Until January, we reserved the critical importance for near-disastrous issues such as data loss; stuff that just about never happens.

This was fine except that it didn’t fully reflect what we in the Launchpad team consider to be really important. So now we also use critical for bugs where:

• there’s a crash: you see one of Launchpad’s oops reports and can’t do what you wanted
• a timeout occurs: Launchpad takes so long to do what you asked that our timeout system kicks in to prevent rogue requests from sapping resources
• a supported browser can’t run Launchpad’s Javascript
• there’s a regression: we accidentally break an existing feature.

The principle here is: just as Launchpad should never lose your data, nor should it stop you from doing something because of poor engineering.

So, our critical bug count has shot up to around 270 because we’ve expanded what we consider to be critical.

Another reason is that we’re getting tougher on timeouts. Obviously we don’t want Launchpad pages to be so slow that they time out.

So, over the past nine months we’ve repeatedly lowered our timeout limit, meaning that we uncover more and more slow-loading pages. Each time a new page hits the timeout limit, we report a new critical bug and work to fix it. We don’t lower the timeout again until we have the failure rate below 0.01% of requests.

Our aim is to have zero critical bugs at any time. Once we get there, Launchpad will have no oopses and no timeouts. People are already talking about how Launchpad is faster to use and that’s largely thanks to the performance effort that our Technical Architect, Rob Collins, has been driving. There’ll be more about this performance work in coming blog posts.

If there’s a down-side to this focus on critical bugs it’s this: the bug that’s important to you may not get fixed as quickly as you’d like. This isn’t because we don’t consider such bugs important but because we’ve decided that the most important bugs are those that stop you dead in your tracks.

If you’re interested in following the performance effort, or the work to tackle these critical bugs, join the launchpad-dev mailing list.

Thanks to Rob Collins for his help with this post.

Photo by Brian Searle. Licence: CC-BY.

Back in September, I asked for suggestions for featured projects for Launchpad’s front page.

As it’s been more than six months, and we have so many exciting projects in Launchpad, I think it’s time for a refresh. While all the projects featured on Launchpad’s front page are still doing great work, it’s only fair to give some others a turn in the spotlight 🙂

So, which projects do you want to see on the front page and why?

Photo by Adrián Pérez. Licence: CC-BY-SA.

Recently, Ursula and I have been improving the way that we test new Launchpad features.

Already, Launchpad has an extensive test suite but there are some things that automated tests can’t look out for. Rather than just testing the quality of our code, we also want to test the quality of the experience.

To do that, we’ve been doing more exploratory testing. Now, when a feature is getting close to deployment we will try out every part of the feature and make notes of anything in the experience that needs to be fixed before we release it. In particular, we’re interested in the feature’s:

• ease of use and discoverability
• completeness
• quality of implementation
• suitability to the problem it is solving
• conformity to Launchpad’s principles and UI guidelines.

We’re also aiming to timebox the testing; something that takes too long to explore is likely too complex. For now, we’re using a 25 minute limit, as borrowed from The Pomodoro Technique, as it seems like a good starting point.

If you’re interested in what we’re doing, you can follow our progress both on the launchpad-dev mailing list and on the Launchpad dev wiki. Also, I’m hoping that we can get your help in testing beta features. I’ll write more about that soon.

Launchpad has been operating in the squad structure for two months now. From my point of view, things have been going very well.

We are making steady progress on fixing timeouts, OOPSes, regressions and other operational issues.

The blue squad switched to maintenance after completing the ‘daily build’ project. The orange squad should follow-up soon once they complete the ‘sharing translations upstreams translations in Ubuntu’ project.

The cross-domain pollination is paying off. Having the API expert working as part of the squad finishing off the ‘daily builds’ work resulted in a long-due overhaul of our AJAX infrastructure. See the drive-by ajaxification of the blueprint page for the results! We saw similar things happen on the orange squad where the knowledge around the job system was put to good use.

I’ve got a lot of good feedback from developers who enjoy the enhanced focus they get in this new configuration.

As expected, the pace is picking up as the new squads learn to operate better together. The average of bugs closed per week went from 63 to 84.

At the last Thunderdome  (whole Launchpad team in person-meeting) where we kicked off the reorg, I issued a challenge: by the next Thunderdome (scheduled for the last week of June) we should:

• have no timeouts with a cut-off at 9s;
• have an empty critical bugs queue;
• be on our way to delivering new features within 45 days on average

The next Thunderdome is now 3 months away. How are we doing?

Yesterday, Robert announced that the timeout was lowered to 11s. That means we have 2 seconds left to go! Looks like this is going well.

On the second point, we have today 224 open critical bugs on the Launchpad project. We started at 264. So on the surface, it’s not going well. The trick here is that there were a lot of issues that weren’t on the list. So we kept finding new issues as we fixed some, giving an impression of stasis. Over the last two weeks though, we have seen a constant drop in the queue length, so I hope that all issues are now recorded and that we are on are way to burning this queue down. The fact that we are now closing 34 of those per week on average (over 21 per week when we started) gives me comfort. We need to make that number go down by 16 each week to reach our goal. So it seems well within reach!

The last one is more challenging. We have a big structural constraint that would make it hard to achieve a cycle time of 45 days on new features: our DB deployment story. Most feature will require two DB patch iterations and we deploy those every 30 days.  Nonetheless, we could read this goal as getting slots free on our ‘Next’ queue. That means finishing off everything that we have in progress plus some things that we haven’t started yet. We have some big items on there. Things like derived distributions and getting our bug privacy manageable. That’s where our real challenge lie! I’m looking forward to see how this will go.

Ever wanted to know what exactly is happening in Launchpad development, but found our list of in-progress bugs too daunting and detailed? Well, do we have a link and a screenshot for you.

Screenshot:

Link. (Log in: guest@launchpad.net; Password: launchpad).

Thanks to the nice folk at LeanKit Kanban, we’ve now got a guest account set up so that anyone can our kanban board. The board shows all of the high level features that we are working on (those are the green ones), the infrastructure projects we’re doing (those are the blue ones) and all of the community-driven work that we know about (the yellow ones).

The further to the right a card is, the closer it is to being done. The cards in the Next column are the things we’ll start to work on once we’ve got room on the board to start them. “UA” means we’re fixing the final few bugs in a feature before we consider it to be done, and “Deployment” means that we need sysadmins to do something.

Our goal is to keep the amount of work-in-progress down to a small number, and to be able to move things from left to right as quickly as possible. We hope making the board available gives you a better insight into Launchpad development, and maybe even encourages you to join in the fun.

Update: We’ve fixed the log-in credentials, for real this time. Sorry for publishing the wrong ones previously, and previously before that.