launchpadblog

Congratulations and virtual cupcakes to nigelb and chrisjohnston who got their first changes in to Launchpad, and to the reviewers who helped get them finished off and deployed.

jml said it so well at his recent short talk:

I’m not going to kid you, [changing Launchpad is] not easy but it’s sooo worth it, you get to help all these people and you get thunderous applause like I did earlier, and really should have gone to those five guys.

(update: fixed mysteriously-broken video start-at time.)

Jonathan (our Product Strategist) and I attended UDS-O in Budapest last week. Brad, Ian, Huw and Julian also came along for some part of it. And what great feedback we received from Ubuntu contributors! Working on a big and relatively old project like Launchpad can sometime be hard. From within, we are well aware that development is slower than it could be, that we have a lot of tech-debt and a big pile of Critical bugs, not to mention the hundreds of things that we know we could do to make Launchpad more compelling. Working distributed where you interact with users and fellow developers mostly through IRC and email, and sometime voice for “richer” communication makes it even easier to lose the perspective you get when you speak to real users face-to-face.

For me, UDS brought a nice uplifting fresh air (and that’s not a small feat when you know that this is a conference where you have sometime ~400 people in close proximity in the same room 😉 I don’t have enough fingers to count the number of people who thanked us for our performance work or for some other bug fixes we did! From their perspective, Launchpad development is going well and they want more! The performance improvements we did have been noticed! And people were thrilled by the new bug subscription work.

You can get a feel for the atmosphere by watching Jono’s 5-minutes lightning talk. The audience reaction is very much in line with the individual feedback we received.

It’s always great to know your effort are appreciated by your users. Thank to all the folks at UDS. Keep the love coming!

Photo by yixing h. Licence: CC BY-NC-ND 2.0.

Launchpad has a lot of <a

Launchpad has a lot of critical bugs. A month ago, we had close to 300 of them. At the time of writing, we have 226.

Francis reckons that we can have zero critical bugs by June 27th but I am a little bit sceptical. To that end, I’ve made a wager with the Launchpad developer community:

If we have zero critical bugs by June 27th, then I’ll let one Launchpad contributor shove a cream pie in my face at our next get-together in Dublin.

Here are the ground rules:

• launchpad-project, not launchpad
• The bugs have to be actually closed, and if fixed, actually released
• “Critical” is as defined on our bug triage page (and no cheating by changing the policy)
• Any bugs that are escalated by Canonical stakeholders after the announcement do not count, but any new timeouts, oopses and so forth do count
• I will leave it to others to nominate a pie-er
• A custard pie would also be acceptable

If you want to see me publicly and deliciously embarrassed in the tastiest way possible, then now is the time to start fixing bugs.  I will make sure that the event is filmed, photographed, instagrammed, live-tweeted and made available in whatever ways the cool kids are mainlining their microtainment these days.

Since announcing this on Twitter, Ted Gould and Neil J. Patel have also volunteered to be cream-pied if we meet this goal.

Photo by little blue hen. Licence: CC BY 2.0.

As part of determining What’s Next?, Jonathan (our Product Strategist) and I will be attending UDS Oneiric next week (May 9th – May 13th) in Budapest. If you are running a session where you need input from Launchpad, or working on a Ubuntu project that would require Launchpad infrastructure change, be sure to subscribe us to the blueprint so that we know to attend the session. It’s likely that anything major that would require a full squad probably wouldn’t be able to be completed before late in the cycle. But there are certain small enhancements that could certainly be taken as part of the maintenance squad work.

Feel free also to grab us in the corridors or in the bar to give us praise and tell us how Launchpad is working great for you! We always appreciate those stories. If you want to complain, or tell us how Launchpad could be even greater if it had this extra feature or do this that way, well, that’s ok too, but will be better appreciated if you offer us a drink 😉

Photo by Chris Barella. Licence: CC BY-NC 2.0.

In my last post, I explained how squads doing maintenance picked the next bug to work on. But how does it work for the other squads?

We have two squad that works on a project which is usually a feature. In this context, a feature means something that will take a team of 4-5 engineers several weeks (or months) to develop. It will usually need a good amount of discussion with the stakeholders to clarify requirements  and we’ll do user testing along the way.

We maintain a short queue of 2 projects to work on next. The items on the queue are determined through the Stakeholders Meeting process. The stakeholders cabal meets once a month (when there is a free slot on the queue) to discuss what’s should be the next items to be developed by Launchpad feature squads. The way it works is that before the meeting interested parties send in their proposal with justifications as why this is the most important piece. People discuss each other proposals. Then the Launchpad Product Strategist (Jonathan Lange) and I will come back with a proposal taking into account the stakeholders discussion. That proposal is discussed during the meeting and a decision on what to add to the Next queue is taken. We try to operate by consensus, but in the end, the Product Strategist has the final say.

The stakeholders group composition which is made of one representative from the major teams within Canonical derives from  the business goal of Launchpad which is to give Canonical a competitive advantage in delivering a high quality OS.

An error that is easily made from that would be to think that only the interests of people within Canonical matter in determining What’s Next? If that would be the case, it would be a very shallow view on our part and we would miss the more general aspect of achieving our business goal which is also to accelerate open-source software development. (Still to give Ubuntu a competitive advantage over proprietary solutions, but hey, it’s a well-known fact that Canonical is a for-profit company.)

The burden of taking into accounts the wide-majority of non-Canonical Launchpad users fall on our Product Strategist’s shoulders. He listens closely to users, in person, on mailing lists, by taking into consideration things reported during user testing sessions. You can always find his long-term plans on the roadmap.

I want something in, who should I talk to?

• If you are within Canonical, talk to the representative working in your unit.
• Otherwise, talk to the product strategist. The easiest way is to start a discussion on the launchpad-users mailing list.
• For Ubuntu folk, we area working with the Ubuntu Technical Board to get a representative who would cater for Ubuntu Project as a whole. But they still haven’t made a decision as to whom they will appoint. Once they are appointed, you should contact them. In the mean-time, talk to the strategist.

Not all features are created equal

Not all feature requests will take several weeks for a squad to implement. For those that can be tackled by one engineer in one or two weeks, they’ll be taken by one of the maintenance squad. These small features can escalated by the stakeholders. (That was the other mechanism by which a bug could be escalated to Critical in my last post.)

When we put items on the Next queue, the feature definition is quite vague. It will specified through a LEP by the product team and the assigned squad once it’s time to work on it. But how that work and what happens after that is probably the topic for another post (as is the answer as to why people write LEPs before we decide to work on the thing).

Hope these posts shed some light on how we decide What’s Next?

Photo by Anonymous. Licence: CC BY 2.0.

To many people, Launchpad development looks like a black box. People asks for thing in various places (the bug tracker, the mailing lists, IRC, directly to developers, or simply wish it out loud) and then, often unexpectedly, Lauchpad will change.

We strive to work in the open, discussing changes on the development mailing list or on the bug tracker, publishing our Kanban board,  announcing stuff here on this blog, often trying to contact concerned users directly. But because Launchpad is a big project catering to the needs of many different type of users, it’s easy to miss out on some people.

So here it is, the Definitive guide on how Launchpad folks decides what to work on next!

Since the squad reorganisation, the principles used to determine what’s next will depend on whether we are looking at a maintenance squad or a feature squad.

What’s next for maintenance?

Let’s start by the easiest case: maintenance. Squads on maintenance works from the bugs in the bug tracker. And then we work on bugs starting from highest importance (Critical currently). Our triage rules explains how we determine bug importance. That’s a very verbose page that in practice means today, treat it as critical if:

• it’s a regression
• it causes an OOPS or a timeout
• creates operational issues

All the other distinctions  between High, Medium and Low at this point are  moot until we clear the backlog of such issues (see my status post to see how we are doing there). That’s not to say that we don’t fix High, Medium or Low bugs. Developers can, at their discretion, decide to work on such bugs if they happen to work in the area, or they find it related to a critical bug, or just because. But developers on maintenance will be generally found fixing critical bugs at this time.

Which one? We usually try to work on the “hot” ones. Those are the ones happening frequently or recently. But after that, it’s really up to developer’s discretion. They’ll usually pick based on their area of competence or area in which they want to acquire competence.

Once we are done with the Critical backlog, then we’ll tackle High bugs.  Again, the same kind of decision process will be used to decide which one to pick next: that is, developer’s discretion. The idea is that time on maintenance rotation should be mostly self-directed for developers.

There is another way that a bug can jump the queue, it can be escalated to Critical importance. Since it’s closely related to the stakeholders’ process, I will cover it in my next post where I’ll discuss on how we determine What’s Next? for  feature squads.

Photo by Crystal. Licence: CC BY 2.0.

Congratulations to everyone in the Ubuntu community on the successful release of Ubuntu 11.04!

Here’s to the next six months of oneiric fun.

Photo by Sharon Terry. Licence: CC-BY-ND 2.0.

Rejoice Blueprint users for Tim has brought the glory of Ajax to the main blueprint page.

He writes:

Now we can update the following without reloading the primary page:

• title – the H1 heading
• summary
• whiteboard
• assignee
• drafter
• approver
• priority
• implementation status
• definition status

Using the new custom events that the page raises when the context object changes (using YUI magic and API PATCH requests), when you change the title of the blueprint, the document title (title bar) and the breadcrumbs also change. When the implementation status is updated, the overall status updates, and the “started by” and “completed by” are shown or hidden as appropriate.

This is work that I’ve wanted to see done for almost a year, and recent other changes I’ve done adding more widget wrappers and javascript goodness have made this possible without adding copious amounts of custom javascript.

A side-effect of these changes is that there are now more fields exported over the API for blueprints.

Photo by Will Scullin. Licence: CC BY 2.0

Last week I asked for suggestions of what should be on the Launchpad front-page featured project list.

Thanks for your suggestions! Based on your comments, I’ve added the following projects to the front page:

• OpenStack: the open source cloud platform
• Gufw: the GUI front-end for the UFW firewall configuration app
• Tomdroid: the read-only Tomboy client for Android
• District Health Information Software: health management software in use in many developing countries
• Unity: the desktop experience

I’m also going to look at better ways of show-casing the projects of Launchpad. I’ll blog with more info soon.

We are now two months away from our next Thunderdome. How are we doing in regards with the objectives set for that milestone? You may recall from my last post the objectives:

1. have no timeouts with a cut-off at 9s;
2. have an empty critical bugs queue;
3. getting a slot free on our ‘Next’ queue.

We practically achieved the first objective! Today, we lowered the hard timeout to 9s and this didn’t increase our number of daily timeouts. We don’t have zero timeouts yet. We still have a fair bunch of timeout bugs to fix. But we get on average 650 requests timing out in a day. That’s less than 0.01% of our traffic.

These remaining timeout bugs are part of our second objective. On that front, we are in a more difficult position. We have 259 critical bugs to close. That went up since last time! What went wrong? Well, we had less people working on critical bugs for once. That’s been fixed this week when the Orange squad rotated back on maintenance. We again have two full squads working on critical bugs. Second, we modified our OOPS reporting to show all timeouts happening, not only the ones occurring the most often. That resulted in about 30 new timeouts filed. (See the hight red bar at the start of the graph). Fortunately for us, the rate of new critical bugs is declining.  We are at about 23 on average in the last two weeks. That’s still high and some of those are related to JS regressions escaping to production because our Windmill test infrastructure is disabled. This means that 51 is now the magic number. We need to close 51 of these critical bugs per week to reach 0 by the Thunderdome. That was the number we closed in our best week, just before the number of people working on criticals was reduced. So we’ll also need to reduce the number of new critical bugs found each week to succeed here.

Finally, on our last objective, like I already mentioned, the Orange squad switched back to maintenance. That’s because they completed the final bits of the project they were working on: Sharing translations between upstreams and Ubuntu. (Expect a blog post about it next week once everything is available to the general public.) That means that we need to complete 5 more projects to free a slot in our Next queue. That will be a stretch, but not impossible since of the two currently in progress, one should be completed in a couple of weeks and the other before the end of next month.  And two of the remaining ones are much simpler than what we have worked on until now.

I’ll post again next month to see how the final stretch looks like.

Photo by Aslak Raanes. Licence: CC BY 2.0.