Flacoste on Launchpad Blog

Faster deployments

Francis J. Lacoste — Fri, 03 Feb 2012 12:36:25 +0000

Back in September, we announced our first fastdowntime deployment. That was a new way to do deployment involving DB changes. This meannt less downtime for you the user, but we were also hoping that it would speed up our development by allowing us to deliver changes more often.

How can we evaluate if we sped up development using this change? The most important metric we look at when making this evaluation is cycle time. That’s the time it takes to go from starting to make a change to having this change live in production. So before fastdowntime, our cycle time was about 10 days, and it is now about 9 days. So along the introduction of this new deployment process, we cut 1 day off the average, or a 10% improvement. That’s not bad.

But comparing the cumulative frequency distribution of the cycle time with the old process and the new will give us a better idea of the improvement.

On this chart, the gap between the orange (fastdowntime deployment) and blue (original process) lines shows the improvement to us. We can see that more changes were completed sooner. For example, under the old process about 60% of the changes were completed in less than 9 days whereas about 70% were completed under the same time in the new process. It’s interesting to note that for changes that took less than 4 days to complete or that took more than 3 weeks to complete, there is no practical difference between the two distributions. We can explain that by the fact that things that were fast before are still fast, and things that takes more than 3 weeks would usually have also encountered a deployment point in the past.

That’s looking at the big picture. Looking at the overall cycle time is what gives us confidence that the process as a whole was improved. For example, the gain in deployment could have been lost by increased development time. But the closer picture is more telling.

The cycle time charted in this case is from the time a change is ready to be deployed until it’s actually live. It basically excludes the time to code, review, merge and test the changes. In this case, we can see that 95% of the changes had to wait less than 9 days to go live under the new process whereas it would take 19 days previously to get the same ratio. So an
improvement of 10 days! That’s way more nice.

Our next step on improving our cycle time is to parallelize our test suite. This is another major bottleneck in our process. In the best case, it usually takes about half a day between the time a developer submits their branch for merging until it is ready for QA on qastaging. The time in between is passed waiting and running the test suite. It takes about 6 hours to our buildbot to validate a set of revisions. We have a project underway to run the tests in parallel. We hope to reduce the test suite time to under an hour with it. This means that it now would be possible for a developer to merge and QA a change on the same day! With this we expect to shave another day maybe two from the global cycle time.

Unfortunately, there are no easy silver bullets to make a dent in the time it takes to code a change. The only way to be faster there would be to make the Launchpad code base simpler. That’s also under way with the services oriented architecture project. But that will take some time to complete.

Photo by Martin Heigan. Licence: CC BY NC ND 2.0.

Legacy, performance, testing: 6 months of new critical bugs analyzed

Francis J. Lacoste — Fri, 09 Dec 2011 21:20:52 +0000

The Launchpad maintenance teams have been working since the beginning of the year at reducing our Critical bugs count to 0. Without much success this far. The long term trend keeps the backlog at around 300. And it’s not because we haven’t been fixing these. Since the beginning of the year, more than 800 Critical bugs were fixed, but more than 900 were reported :-(

So I investigated what was the source of all these new critical bugs we were finding. A random sample of 50 critical bugs filed were analyzed to see where and why they were introduced. The full analysis is available as a published Google document.

Here are the most interesting findings from the report:

Most of the new bugs (68%) are actually legacy issues lurking in our code base.
Performance and spotty test coverage represents together more than 50% of the cause of our new bugs. We should refocus maintenance on tackling performance problems, that’s what is going to bring us the most bang for the bucks (even though it’s not cheap).
As a team, we should increase our awareness of testing techniques and testing coverage. Always do TDD, maybe investigate ATDD to increase the coverage and documentation our the business rules we should be supporting.
We also need to pay more attention to how code is deployed, it’s now very usual for scripts to be interrupted, and for the new and ancient version of the code to operate in parallel.

Another way of looking at this is that Launchpad represents a very deep mine of technical debt. We don’t know how exactly deep the mine is, but we are going to find existing Critical issues until we hit the bottom of the mine. (Or until we finish refactoring enough of Launchpad for better testing and performance. That’s what the SOA project is all about.)

In the mean time, we should pay attention to regressions and fallouts, (those are really the new criticals) to make sure that we aren’t extending the mine!

Photo by Brian W. Tobin. Licence: CC BY-NC-ND 2.0.

Speeding up development

Francis J. Lacoste — Fri, 09 Sep 2011 21:38:44 +0000

Today we reached a significant milestone, we completed our first fast down-time deployment. Two obvious reasons for doing this were already mentioned in the announcement and our technical architect’’s post describing the change:

We’ll have less downtime per month (at the cost of more frequent but short interruption).
We’ll be able to deploy fixes and changes involving DB schema more frequently.

But from my perspective, the most important benefit I think we’ll get from this is a speed up in our rate of development, particularly, in terms of completing feature projects. It’s not a secret, our feature squads spends a lot of time to complete their projects. There are multiple reasons for this, but in the end, there usually fall under two broad cateries: the time it takes to actually make the change, and the delays in getting feedback on the change itself.

To help with the first category, you’ll want better and more powerful libraries, better architecture, developers’ training, etc. Think the time difference between developping a database web application in Django vs as a CGI C application using only the standard C library. Launchpad isn’t using the most modern libraries and toolkits, and we could still make a lot of improvements there. But the costs of making changes in this space are compounded by the problems of the other category.

Once you wrote the change, you are far from done. There are lots of hoops you still have to jump through before saying “done-done”: you’ll need to make sure the tests pass, to get your changes reviewed, merged, QAed and then deployed. And finally, you’ll probably want to make sure that it matches the user’s expectations, but until it’s in production, this is hard to assess reliably. All of these steps takes time and introduce delays, some bigger than others. The Launchpad team is always on the look-out to cut in these delays and the new “fast down-time deployments” cut on of the biggest one we had.

Since a picture is worth at least a thousand words, have a look at the chart above to have a better idea of what I’m talking about. It shows the distribution of the time it takes to complete a “change”. (What this plots is the cycle time from coding to deployment of our Kanban cards which roughly map to one logical change.) You’ll see that 50% of our changes are deployed to production in about a week. And the next 45% takes between 1 and 5 weeks. Now, our feature projects are composed of many many of these smaller changes. If those are all relatively small changes, why do they take so long?

One of the big bottleneck was the batch size of our DB deployment. If a change required a DB schema it waited until the next downtime deployment which happened once every month. In theory, that means that on average a change involving the database would wait 2 weeks in the queue before deployment. In practice, it’s more complex than that, because squad leads would often plan around these. So a database change would be hold off onto because it was deemed that it couldn’t be safely completely to be part of the next downtime deployment. So it might be put on hold in favour of other work, and delayed to the next downtime deployment. It’s also frequent to have other changes building on the first also queued up waiting for the next deployment window. Add on top of that, that it’s common for the completion of a feature to require several iterations of DB change based on feedback and you quickly understand how you can be working months on a feature project!

But this major bottleneck is now gone! We’ll be able to land and deploy DB changes reliably within days, giving us much more rapid feedback. I’m looking forward to the change in the cycle time distribution in the coming months. The whole distribution should move toward the left. I’ll write a follow-up in two months to see if this prediction comes true.
Photo by Nathan E Photography. Licence: CC BY 2.0.

Matthew Revell is the new Launchpad Product Manager

Francis J. Lacoste — Fri, 26 Aug 2011 22:04:01 +0000

It’s my pleasure to announce that as of today, Matthew Revell is the new
Launchpad Product Manager replacing Jonathan Lange.

We were seduced by his bold vision for Launchpad along with his data-centric approach that he intends to bring to the role. He also has an already extensive experience interacting with Launchpad developers and users. If you read this blog, you probably read something written by him! Or you might have interacted with him in one of the many user-research sessions he ran. The introduction of user-research helped us release better designed feature. Building on this experience, we hope that his leadership will bring Launchpad to the next level.

Matt will communicate more about his plans for Launchpad shortly. In the
mean time, let’s give him our warmest congratulations!

Echoes from the Dublin Thunderdome

Francis J. Lacoste — Fri, 15 Jul 2011 21:52:33 +0000

I can’t believe it’s been two weeks already! From June 25th to July 1st, the whole Launchpad team was gathered in Dublin for our semi-annual all-hands event. Like usual, we also invited the Bazaar team and one of our friendly sysadmin.

The theme this time around was “UI, UI, UI”. Since the January squad reorganisation, it became evident that a big stumbling block for squads working on features was UI work. Our team is still inexperienced with JavaScript and the YUI3 framework. So each squad has encountered similar problems and sometime found different solutions. We wanted to take advantage of the face-time to come to a common understanding of the best patterns to use.

In the end, it was a week of intense hacking on UI infrastructure, with part of the mornings dedicated on presentation where people shared what they learnt in the past 6 months.

Projects that were worked on during the week:

An asynchronous notification system (extracted from Landscape) which will allow us to update pages as soon as tasks complete on the backend.
We ditched our buggy Windmill-based Javascript integration tests, in favour of a yuitest-based suite. We already use yuitest for JS unit tests, but now we’ll be able to use it for integration tests where we need to make XHR requests to a live server. We’ll even be able to set server-side fixtures from within the test!
Investigate the use of selenium2 for acceptance testing. These are tests ensuring a complete workflow works and that ideally will be able to run against both the staging server as well as within tree for regression testing.
Integrating loggerhead-backed branch data on some Launchpad pages. You should soon be able to get the diff of a particular revision directly on the merge proposal page.
Infrastructure to refresh all the client-side representation of objects related to a page in one request.
Many UI bugs were fixed.

This was also the week we said farewell to old friend: Jonathan our Product Strategist and Ursula Junque who is now working as QA analyst for the Ubuntu Server team. In the end, Jono didn’t get pied as we still had a huge number of Critical bugs left open. We did rejoice in achieving our performance improvements target. Given the trend, I estimated that we have still 6 months before emptying the Critical bugs list. The new target is to reduce to a 100 for October. Let’s see if I was more realistic this time around.

And we also took the traditional team picture.

Great fun it was, and I can’t wait for the next one in January.

Photo by Miss_Colleen. Licence: CC BY-NC-SA 2.0.

Farewell, Jonathan

Francis J. Lacoste — Fri, 17 Jun 2011 21:35:31 +0000

Yesterday, Jonathan Lange our Product Strategist announced that he was leaving Launchpad after four and a half year. The fact that he’s only leaving next door to work on the Ubuntu Developer Programme from within the Platform team is only a small comfort.

Before becoming Launchpad Product Strategist, Jonathan was known for adding support for source package branches to Launchpad. He became Strategist at about the same time I took the interim leadership of Launchpad. It will interesting to see how I manage without him!

He was instrumental in most of the process changes that happened to the team in the last two years. We can thanks Jonathan for bringing us the LEP process which make sure that we focus on the right things before starting coding, the Stakeholders process to make sure that we are working on the right thing. He also brought some clarity on what Launchpad was all about! Recently, he has been working on clarifying the feature development process, so that we can deliver polished changes on a regular cadence to users. I’m probably forgetting also a lot of other changes which his constant quest for improvements suggested to us in one way or another.

Oh, and we can thank him for making sure that we didn’t move on early from daily builds, improved bug subscriptions and other changes without the polish they have today. I (and I’m sure our users) thank you!

Whoever takes that role forward, will have big shoes to fill! If you are interested by the challenge, a job opening should be posted soon!

Thanks a lot Jonathan, it was a pleasure to work with you all these years! You’ll be missed. Good luck with your new project, but I’m glad to think that the future of Ubuntu Application Development is in excellent hands!

Photo by Robin Robokow. Licence: CC BY-NC-SA 2.0.

Bryce Harrington appointed Ubuntu Launchpad stakeholder representative

Francis J. Lacoste — Sat, 04 Jun 2011 17:38:00 +0000

When I explained how we determined what feature to work on next, I mentioned that we were looking to add a stakeholders representative for the Ubuntu project. Well, it’s now done. The Ubuntu Technical Board delegated Bryce Harrington to that position.

So if you are part of the Ubuntu community and need something from Launchpad, you can now talk to Bryce.

I’m really glad to have him on that position. Not only is he a respected member of the Ubuntu community, but he also did a 6 months rotation within the Launchpad team, so has understanding of how our project operates.

He’ll now be in a position to escalate important bugs affecting the Ubuntu community, as well as present the community’s view on what new features should be prioritized. It won’t be an easy job, since the Ubuntu project is big and varied. The people who cares about bugs are often not the same that cares about translations nor the same that cares about package publishing or the Ubuntu distribution development toolchain. And the stakeholder representative job is to help prioritize these needs to make sure that the Launchpad team works on the most important work. Plus then he’ll have to find common grounds with the other stakeholder representatives!

Thanks Bryce for stepping up to these responsibilities!

Photo by Wally Gobetz. Licence: CC BY-NC-ND 2.0.

What does rotation to maintenance really means?

Francis J. Lacoste — Fri, 03 Jun 2011 21:36:30 +0000

I’ve been asked this question privately a few times already. What does it mean for a squad to rotate from feature-mode to maintenance-mode? The question arises as a number of folks noticed that some members of the squad continue working on feature-related bugs after they are on maintenance shift. That happened with the former Blue squad when they completed daily builds, with the Orange squad when they completed sharing translations, and with the Yellow squad and better bug subscriptions. So let me clarify what that boundary means.

In a nutshell, it means that the squad starts new work from a different queue. After the switch, new work will come from the general maintenance queue. Because we do the switch aligned with the calendar week, there is always work related to the feature that is still in-process (code started but not reviewed, or landed but not deployed, etc.) That’s the work that continues after the transition. After the rotation, the squad also becomes on the hook to cater for the user support duties related to maintenance mode.

What are the trade-off related to this boundary definition? The main impact is that we get a dip in maintenance related bug fixes around the rotation. That’s because the new maintenance squad has still some members not ready to start new maintenance-related work. It could be argued that because of the interrupt nature of the maintenance responsibilities, it would take longer to finish off the work related to the feature. I’m not sure that happens in practice, but I make note to check the data…

The alternative would be to only rotate after all the feature-related bugs are done-done. The rest of the squad could fix some lower priority bugs related to the feature or clean-up tech-debt accumulated during their rotation. It would preserve the number of people effectively working on maintenance around the transitions. But it would mean that it would take longer before we start the next feature. And given the poor state of our feature cycle time, it’s not a trade-off that I (and I guess our stakeholders) would be willing to make at this time. We might revisit this in the future though.

Photo by Camilia Hoel. Licence: CC BY-NC-SA 2.0.

Countdown to the pie

Francis J. Lacoste — Fri, 27 May 2011 22:16:45 +0000

We are now one month away from the Dublin Thunderdome and the stakes of the game have increased substantially since last time! At UDS, Jonathan announced he was willing to take a pie in the face if we achieved our 0 critical bugs goal. Two others joined him in the bet.

As of this writing, we have 210 critical bugs left to fix to pie Jono, Ted and Neil. (203 actually, since 7 of those are bugs escalated after the bet announcement.) Unfortunately, their face seems pretty safe :-/ If we look at the trend of the last 4 weeks, we see that we are burning down about 12 bugs per week. At this rate, we would achieve this goal in 18 weeks, so around the end of September. 3 months too late for a pie! This week, the Yellow squad is starting on maintenance, so we’ll see if they can improve on the 12 bugs a week burn-down. I mean, that number is 3 times the long-term rate which is more like 4 a week, so we are definitively improving! Both because we are better at fixing bugs, but also because the number of newly found issues is declining. You can see the combined effect of this on the burndown graph as the slope gets much sharper on the right.

It doesn’t look much better on our third objective which is to have a free slot in the stakeholders Next queue. This week the Teal squad resumed work on the our project to improve Launchpad privacy features. They did an initial break-down of the project and they have work until the next UDS! The Red squad is finishing off derived distributions, probably before the Thunderdome. But even then, at that point we would be starting on adding customizable columns on search results. And the Next queue would still be full at 2 items. On the plus side, we won’t be over our work-in-process limit anymore :-)

I’m looking forward to see how this all turns out in Dublin, but whatever happens, at the very least, we’ll have succeeded early on our 9s timeout objective.

Photo by David Muir. Licence: CC BY-NC-ND 2.0.

Love from Budapest

Francis J. Lacoste — Fri, 20 May 2011 21:42:54 +0000

Jonathan (our Product Strategist) and I attended UDS-O in Budapest last week. Brad, Ian, Huw and Julian also came along for some part of it. And what great feedback we received from Ubuntu contributors! Working on a big and relatively old project like Launchpad can sometime be hard. From within, we are well aware that development is slower than it could be, that we have a lot of tech-debt and a big pile of Critical bugs, not to mention the hundreds of things that we know we could do to make Launchpad more compelling. Working distributed where you interact with users and fellow developers mostly through IRC and email, and sometime voice for “richer” communication makes it even easier to lose the perspective you get when you speak to real users face-to-face.

For me, UDS brought a nice uplifting fresh air (and that’s not a small feat when you know that this is a conference where you have sometime ~400 people in close proximity in the same room ;-) I don’t have enough fingers to count the number of people who thanked us for our performance work or for some other bug fixes we did! From their perspective, Launchpad development is going well and they want more! The performance improvements we did have been noticed! And people were thrilled by the new bug subscription work.

You can get a feel for the atmosphere by watching Jono’s 5-minutes lightning talk. The audience reaction is very much in line with the individual feedback we received.

It’s always great to know your effort are appreciated by your users. Thank to all the folks at UDS. Keep the love coming!

Photo by yixing h. Licence: CC BY-NC-ND 2.0.