launchpadblog

In my last post, I explained how squads doing maintenance picked the next bug to work on. But how does it work for the other squads?

We have two squad that works on a project which is usually a feature. In this context, a feature means something that will take a team of 4-5 engineers several weeks (or months) to develop. It will usually need a good amount of discussion with the stakeholders to clarify requirements  and we’ll do user testing along the way.

We maintain a short queue of 2 projects to work on next. The items on the queue are determined through the Stakeholders Meeting process. The stakeholders cabal meets once a month (when there is a free slot on the queue) to discuss what’s should be the next items to be developed by Launchpad feature squads. The way it works is that before the meeting interested parties send in their proposal with justifications as why this is the most important piece. People discuss each other proposals. Then the Launchpad Product Strategist (Jonathan Lange) and I will come back with a proposal taking into account the stakeholders discussion. That proposal is discussed during the meeting and a decision on what to add to the Next queue is taken. We try to operate by consensus, but in the end, the Product Strategist has the final say.

The stakeholders group composition which is made of one representative from the major teams within Canonical derives from  the business goal of Launchpad which is to give Canonical a competitive advantage in delivering a high quality OS.

An error that is easily made from that would be to think that only the interests of people within Canonical matter in determining What’s Next? If that would be the case, it would be a very shallow view on our part and we would miss the more general aspect of achieving our business goal which is also to accelerate open-source software development. (Still to give Ubuntu a competitive advantage over proprietary solutions, but hey, it’s a well-known fact that Canonical is a for-profit company.)

The burden of taking into accounts the wide-majority of non-Canonical Launchpad users fall on our Product Strategist’s shoulders. He listens closely to users, in person, on mailing lists, by taking into consideration things reported during user testing sessions. You can always find his long-term plans on the roadmap.

I want something in, who should I talk to?

• If you are within Canonical, talk to the representative working in your unit.
• Otherwise, talk to the product strategist. The easiest way is to start a discussion on the launchpad-users mailing list.
• For Ubuntu folk, we area working with the Ubuntu Technical Board to get a representative who would cater for Ubuntu Project as a whole. But they still haven’t made a decision as to whom they will appoint. Once they are appointed, you should contact them. In the mean-time, talk to the strategist.

Not all features are created equal

Not all feature requests will take several weeks for a squad to implement. For those that can be tackled by one engineer in one or two weeks, they’ll be taken by one of the maintenance squad. These small features can escalated by the stakeholders. (That was the other mechanism by which a bug could be escalated to Critical in my last post.)

When we put items on the Next queue, the feature definition is quite vague. It will specified through a LEP by the product team and the assigned squad once it’s time to work on it. But how that work and what happens after that is probably the topic for another post (as is the answer as to why people write LEPs before we decide to work on the thing).

Hope these posts shed some light on how we decide What’s Next?

Photo by Anonymous. Licence: CC BY 2.0.

To many people, Launchpad development looks like a black box. People asks for thing in various places (the bug tracker, the mailing lists, IRC, directly to developers, or simply wish it out loud) and then, often unexpectedly, Lauchpad will change.

We strive to work in the open, discussing changes on the development mailing list or on the bug tracker, publishing our Kanban board,  announcing stuff here on this blog, often trying to contact concerned users directly. But because Launchpad is a big project catering to the needs of many different type of users, it’s easy to miss out on some people.

So here it is, the Definitive guide on how Launchpad folks decides what to work on next!

Since the squad reorganisation, the principles used to determine what’s next will depend on whether we are looking at a maintenance squad or a feature squad.

What’s next for maintenance?

Let’s start by the easiest case: maintenance. Squads on maintenance works from the bugs in the bug tracker. And then we work on bugs starting from highest importance (Critical currently). Our triage rules explains how we determine bug importance. That’s a very verbose page that in practice means today, treat it as critical if:

• it’s a regression
• it causes an OOPS or a timeout
• creates operational issues

All the other distinctions  between High, Medium and Low at this point are  moot until we clear the backlog of such issues (see my status post to see how we are doing there). That’s not to say that we don’t fix High, Medium or Low bugs. Developers can, at their discretion, decide to work on such bugs if they happen to work in the area, or they find it related to a critical bug, or just because. But developers on maintenance will be generally found fixing critical bugs at this time.

Which one? We usually try to work on the “hot” ones. Those are the ones happening frequently or recently. But after that, it’s really up to developer’s discretion. They’ll usually pick based on their area of competence or area in which they want to acquire competence.

Once we are done with the Critical backlog, then we’ll tackle High bugs.  Again, the same kind of decision process will be used to decide which one to pick next: that is, developer’s discretion. The idea is that time on maintenance rotation should be mostly self-directed for developers.

There is another way that a bug can jump the queue, it can be escalated to Critical importance. Since it’s closely related to the stakeholders’ process, I will cover it in my next post where I’ll discuss on how we determine What’s Next? for  feature squads.

Photo by Crystal. Licence: CC BY 2.0.

Congratulations to everyone in the Ubuntu community on the successful release of Ubuntu 11.04!

Here’s to the next six months of oneiric fun.

Photo by Sharon Terry. Licence: CC-BY-ND 2.0.

Rejoice Blueprint users for Tim has brought the glory of Ajax to the main blueprint page.

He writes:

Now we can update the following without reloading the primary page:

• title – the H1 heading
• summary
• whiteboard
• assignee
• drafter
• approver
• priority
• implementation status
• definition status

Using the new custom events that the page raises when the context object changes (using YUI magic and API PATCH requests), when you change the title of the blueprint, the document title (title bar) and the breadcrumbs also change. When the implementation status is updated, the overall status updates, and the “started by” and “completed by” are shown or hidden as appropriate.

This is work that I’ve wanted to see done for almost a year, and recent other changes I’ve done adding more widget wrappers and javascript goodness have made this possible without adding copious amounts of custom javascript.

A side-effect of these changes is that there are now more fields exported over the API for blueprints.

Photo by Will Scullin. Licence: CC BY 2.0

Last week I asked for suggestions of what should be on the Launchpad front-page featured project list.

Thanks for your suggestions! Based on your comments, I’ve added the following projects to the front page:

• OpenStack: the open source cloud platform
• Gufw: the GUI front-end for the UFW firewall configuration app
• Tomdroid: the read-only Tomboy client for Android
• District Health Information Software: health management software in use in many developing countries
• Unity: the desktop experience

I’m also going to look at better ways of show-casing the projects of Launchpad. I’ll blog with more info soon.

We are now two months away from our next Thunderdome. How are we doing in regards with the objectives set for that milestone? You may recall from my last post the objectives:

1. have no timeouts with a cut-off at 9s;
2. have an empty critical bugs queue;
3. getting a slot free on our ‘Next’ queue.

We practically achieved the first objective! Today, we lowered the hard timeout to 9s and this didn’t increase our number of daily timeouts. We don’t have zero timeouts yet. We still have a fair bunch of timeout bugs to fix. But we get on average 650 requests timing out in a day. That’s less than 0.01% of our traffic.

These remaining timeout bugs are part of our second objective. On that front, we are in a more difficult position. We have 259 critical bugs to close. That went up since last time! What went wrong? Well, we had less people working on critical bugs for once. That’s been fixed this week when the Orange squad rotated back on maintenance. We again have two full squads working on critical bugs. Second, we modified our OOPS reporting to show all timeouts happening, not only the ones occurring the most often. That resulted in about 30 new timeouts filed. (See the hight red bar at the start of the graph). Fortunately for us, the rate of new critical bugs is declining.  We are at about 23 on average in the last two weeks. That’s still high and some of those are related to JS regressions escaping to production because our Windmill test infrastructure is disabled. This means that 51 is now the magic number. We need to close 51 of these critical bugs per week to reach 0 by the Thunderdome. That was the number we closed in our best week, just before the number of people working on criticals was reduced. So we’ll also need to reduce the number of new critical bugs found each week to succeed here.

Finally, on our last objective, like I already mentioned, the Orange squad switched back to maintenance. That’s because they completed the final bits of the project they were working on: Sharing translations between upstreams and Ubuntu. (Expect a blog post about it next week once everything is available to the general public.) That means that we need to complete 5 more projects to free a slot in our Next queue. That will be a stretch, but not impossible since of the two currently in progress, one should be completed in a couple of weeks and the other before the end of next month.  And two of the remaining ones are much simpler than what we have worked on until now.

I’ll post again next month to see how the final stretch looks like.

Photo by Aslak Raanes. Licence: CC BY 2.0.

Have you seen that Launchpad is on Facebook?

If you want to get Launchpad feature and development news through Facebook, like our page.

There are also our Twitter and identi.ca accounts, if you prefer those.

What does “critical” mean? And how is it that the number of critical Launchpad bugs has leapt up to almost 300?

Until January, we reserved the critical importance for near-disastrous issues such as data loss; stuff that just about never happens.

This was fine except that it didn’t fully reflect what we in the Launchpad team consider to be really important. So now we also use critical for bugs where:

• there’s a crash: you see one of Launchpad’s oops reports and can’t do what you wanted
• a timeout occurs: Launchpad takes so long to do what you asked that our timeout system kicks in to prevent rogue requests from sapping resources
• a supported browser can’t run Launchpad’s Javascript
• there’s a regression: we accidentally break an existing feature.

The principle here is: just as Launchpad should never lose your data, nor should it stop you from doing something because of poor engineering.

So, our critical bug count has shot up to around 270 because we’ve expanded what we consider to be critical.

Another reason is that we’re getting tougher on timeouts. Obviously we don’t want Launchpad pages to be so slow that they time out.

So, over the past nine months we’ve repeatedly lowered our timeout limit, meaning that we uncover more and more slow-loading pages. Each time a new page hits the timeout limit, we report a new critical bug and work to fix it. We don’t lower the timeout again until we have the failure rate below 0.01% of requests.

Our aim is to have zero critical bugs at any time. Once we get there, Launchpad will have no oopses and no timeouts. People are already talking about how Launchpad is faster to use and that’s largely thanks to the performance effort that our Technical Architect, Rob Collins, has been driving. There’ll be more about this performance work in coming blog posts.

If there’s a down-side to this focus on critical bugs it’s this: the bug that’s important to you may not get fixed as quickly as you’d like. This isn’t because we don’t consider such bugs important but because we’ve decided that the most important bugs are those that stop you dead in your tracks.

If you’re interested in following the performance effort, or the work to tackle these critical bugs, join the launchpad-dev mailing list.

Thanks to Rob Collins for his help with this post.

Photo by Brian Searle. Licence: CC-BY.

Back in September, I asked for suggestions for featured projects for Launchpad’s front page.

As it’s been more than six months, and we have so many exciting projects in Launchpad, I think it’s time for a refresh. While all the projects featured on Launchpad’s front page are still doing great work, it’s only fair to give some others a turn in the spotlight 🙂

So, which projects do you want to see on the front page and why?

Photo by Adrián Pérez. Licence: CC-BY-SA.

Recently, Ursula and I have been improving the way that we test new Launchpad features.

Already, Launchpad has an extensive test suite but there are some things that automated tests can’t look out for. Rather than just testing the quality of our code, we also want to test the quality of the experience.

To do that, we’ve been doing more exploratory testing. Now, when a feature is getting close to deployment we will try out every part of the feature and make notes of anything in the experience that needs to be fixed before we release it. In particular, we’re interested in the feature’s:

• ease of use and discoverability
• completeness
• quality of implementation
• suitability to the problem it is solving
• conformity to Launchpad’s principles and UI guidelines.

We’re also aiming to timebox the testing; something that takes too long to explore is likely too complex. For now, we’re using a 25 minute limit, as borrowed from The Pomodoro Technique, as it seems like a good starting point.

If you’re interested in what we’re doing, you can follow our progress both on the launchpad-dev mailing list and on the Launchpad dev wiki. Also, I’m hoping that we can get your help in testing beta features. I’ll write more about that soon.