launchpadblog

We are now two months away from our next Thunderdome. How are we doing in regards with the objectives set for that milestone? You may recall from my last post the objectives:

1. have no timeouts with a cut-off at 9s;
2. have an empty critical bugs queue;
3. getting a slot free on our ‘Next’ queue.

We practically achieved the first objective! Today, we lowered the hard timeout to 9s and this didn’t increase our number of daily timeouts. We don’t have zero timeouts yet. We still have a fair bunch of timeout bugs to fix. But we get on average 650 requests timing out in a day. That’s less than 0.01% of our traffic.

These remaining timeout bugs are part of our second objective. On that front, we are in a more difficult position. We have 259 critical bugs to close. That went up since last time! What went wrong? Well, we had less people working on critical bugs for once. That’s been fixed this week when the Orange squad rotated back on maintenance. We again have two full squads working on critical bugs. Second, we modified our OOPS reporting to show all timeouts happening, not only the ones occurring the most often. That resulted in about 30 new timeouts filed. (See the hight red bar at the start of the graph). Fortunately for us, the rate of new critical bugs is declining.  We are at about 23 on average in the last two weeks. That’s still high and some of those are related to JS regressions escaping to production because our Windmill test infrastructure is disabled. This means that 51 is now the magic number. We need to close 51 of these critical bugs per week to reach 0 by the Thunderdome. That was the number we closed in our best week, just before the number of people working on criticals was reduced. So we’ll also need to reduce the number of new critical bugs found each week to succeed here.

Finally, on our last objective, like I already mentioned, the Orange squad switched back to maintenance. That’s because they completed the final bits of the project they were working on: Sharing translations between upstreams and Ubuntu. (Expect a blog post about it next week once everything is available to the general public.) That means that we need to complete 5 more projects to free a slot in our Next queue. That will be a stretch, but not impossible since of the two currently in progress, one should be completed in a couple of weeks and the other before the end of next month.  And two of the remaining ones are much simpler than what we have worked on until now.

I’ll post again next month to see how the final stretch looks like.

Photo by Aslak Raanes. Licence: CC BY 2.0.

This entry was posted by Francis J. Lacoste on Friday, April 22nd, 2011 at 2:00 pm and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.