Posts Tagged ‘launchpad’

How Novacut uses Launchpad

Tuesday, November 20th, 2012

Launchpad has been a key tool used in developing Novacut. I use Launchpad for code hosting, bug tracking, daily builds, and more. For almost two years I’ve been doing monthly stable releases on Launchpad, and Novacut now spans six separate Launchpad projects. To say the least, I’ve learned a lot about Launchpad in the process.

I don’t think Novacut could be where it is today without Launchpad, so I want to pass on some of what I’ve learned the past two years. Here are my five essential Launchpad best practices:

1. Daily Builds

I’m always very thankful that early on Paul Hummer took the time to school me on using Source Package Recipes to do daily builds. This Launchpad service gives you automated package builds across multiple architectures, and multiple Ubuntu releases.

I don’t know how to emphasize this enough, but seriously, you need daily builds. As a point of reference, daily builds are the 3rd item in the famed Joel Test.

These builds are triggered simply by making commits to the appropriate bzr branch on Launchpad (usually your trunk branch). You’ll automatically get up to one build per 24-hour period, and you can manually trigger additional builds when needed.

You can include your debian/ packaging directory in your project source tree, or you can keep debian/ in a separate bzr branch. For the Novacut components, I’ve found it most helpful to keep debian/ in the source trees because it’s handy to be able to land a code change and its corresponding packaging change in a single merge. This works for us because we currently can use the exact same debian/ for all the Ubuntu versions we support. If that’s not true for your project, you’ll need multiple debian/ branches.

For reference, here’s the Novacut Source Package Recipe.

2. Unit Tests

You should run your unit tests during your package builds, and you should fail the build when any unit test fails. This is particularly important for daily builds, because this will prevent a package with broken unit tests from reaching your daily PPA.

The Launchpad build servers are strict and unforgiving environments, which is a good thing when it comes to unit tests. The build servers are also probably quite different from your local development environment. On countless occasions our daily builds have caught failures that only occur on i386 (my workstation is amd64), or only occur on an Ubuntu release other than the one I’m running, etc.

To run your unit tests during the package build, you’ll need to modify your debian/rules file as appropriate. If you’re using debhelper, add an override_dh_auto_test target.

You might also need to add additional packages to the Build-Depends section of your debian/control file, packages that are needed by the unit tests but are otherwise not needed by the build itself.

For reference, here’s the debian/rules file used to run the Dmedia unit tests (which is also a handy Python3 example).

3. Track Ubuntu+1

When a new Ubuntu version opens up for development, I immediately start doing daily builds on the development version, even though I don’t typically upgrade my own computers till around 4 months into the cycle.

I use daily builds on the development release as an early warning system. With no extra effort on my part, these builds give me a heads-up about code or packaging changes that might be needed to make Novacut work well on the next Ubuntu release.

To enable daily builds on the next Ubuntu version, just go to your Source Package Recipe, click on “Distribution series”, and check the box for the newest series. Now you’ll have daily builds on the newest Ubuntu version, in addition to all the versions you were already building for.

For example, I’m currently in the process of enabling daily builds for Raring, as you can see in the Microfiber Source Package Recipe. And I did indeed encounter a build failure on Raring, seemingly caused by a debhelper issue.

For the first month or so in a cycle, I don’t tend to worry much about build failures on the development version. But after the dust has settled a bit, I make sure to keep the builds in working order, and I even do monthly stable releases for the Ubuntu development version. Again, I do all this pro-actively even before I personally start running the newest Ubuntu version.

4. PPAs & Users

Whenever someone asks me why I use Launchpad instead of github, my short answer is always, “PPAs and users”.

Source Package Recipes give you much more than just a build, they give you daily packages that are easily consumable by your testing community and early adopters. This tight feedback loop prevents you from running too far ahead without getting a good reality check from your target users.

Keep in mind that for some products, the early adopters willing to install from a PPA might not be all that representative of your target user. So when it comes to making design decisions, you might need to politely ignore certain feedback from some of these early adopters. In my experience, this wont cause any hard feelings as long as you have clearly communicated who your target user is, and why.

For reference, you might look at the way we’ve defined the Novacut target user.

I recommend creating PPA names that are well-branded and easy to remember. First, create a Launchpad team with the same name as your product. In our case, we have a ~novacut team. Second, I recommend creating a daily and a stable PPA owned by the same team. In our case, that gives us two easy to remember PPAs:

Although none of our target users (professional video editors) currently use Ubuntu to do their job, I’ve been surprised by how many follow Novacut’s development via our stable PPA, and even our daily PPA. This has helped keep us on track, and has helped us build customer loyalty even before we have a finished product.

For me personally, this daily user engagement also makes the design and development process more enjoyable. It’s hard to empathize with an abstract persona; it’s easier to solve specific problems for specific people.

5. Use Apport

Till recently I didn’t realize that you can use Apport for automated crash reporting in unofficial packages delivered through a PPA.

We haven’t had Apport integration for that long, but it’s already provided us with dozens of highly valuable crash reports. Almost immediately some hardware specific issues came to light and were fixed, convincing me that a key benefit of Apport is knowing how your app might misbehave on a larger, more variable pool of hardware.

Apport also helped some rare bugs come to light. I thought Dmedia was basically crash-free, but those one-in-a-thousand bugs pop out quickly when thousands of people are running it. Most of these bugs would have eventually been found by one of our core devs, but the quicker a bug is discovered, the quicker and easier the bug is to fix.

For more info, check out this blog post and this screencast, where I covered our Apport integration in detail.

And for reference, see the merge proposal that added Apport integration in Novacut.

A big thank you to Jason DeRose for sharing how his project uses Launchpad on a daily basis.


Burning down critical bugs

Tuesday, October 2nd, 2012

I have been analysing Launchpad’s critical bugs to track the Purple squad’s progress while on Launchpad maintenance duty. In January of 2011, the Cloud Engineering team né Launchpad Engineering team was reorganised into squads, where one or more squads would maintain Launchpad while other squads work on features. This change also aligned with a new found effort to enforce the zero-oops policy. The two maintenance squads had more than 332 critical bugs to close before we could consider adding features that the stakeholders and community wanted. By July 2011, the count dropped to its lowest point, 250 known critical bugs. Why did the count stop falling for fifteen months? Why is the count falling again?

Charting and analysing critical bugs

Chart of Launchpad's critical bugs since the formation of Launchpad squads and maintenance duties
The chart above needs some explanation to understand what is happening in Launchpad’s critical bugs over time. (You may want to open the image in a separate window to see everything in detail.) Each iteration is one week. The backlog represent the open critical bugs in launchpad at the start of the iteration. The future bugs are either bugs that are not discovered, not introduced, or reported and fixed within the iteration. The last group is crucial to understand the lines plotting the number of bugs fixed and added during the iteration. We strive to close critical bugs immediately. Most critical bugs are reported and fixed in a few days, so most bugs were not open long enough to be show up in the backlog. The number of bugs fixed must exceed the number added to make the backlog count fall. You can see that the maintenance squads have always been burning down the critical bugs, but if you are just watching the number of open bugs in Launchpad, you get the sense that the squads are running to just stand still.

I use the lp-milestone YUI widget to chart the bugs and analyse the our progress through the critical bugs. It allows me to summarise a set of bugs, or analyse a subset by bug tag.

Launchpad maintenance analysis -- driving critical bugs to zero

Though 22 bugs were fixed this past week, 14 were added, thus the critical count dropped by 8. The last eight iterations are used to calculate the average bugs closed and open per iteration. The relative velocity (velocity – flux) is used to estimate the remaining number of days to drive the count to zero. When the Purple squad started maintenance on September 10th of 2012, the estimated days of effort was more than 1,200. In just three weeks, the number has fallen dramatically. The principle reason the backlog of critical bugs has fallen is that the Purple squad is now giving those bugs their full attention, but that generalisation is unsatisfactory.

Why is the Purple squad so good at closing bugs in the critical backlog?

I do not know the answer to my question. The critical backlog reached its all-time low of 250 bugs with the release of the Purple squad’s maintenance work in July 2011. There was supposition that  Purple fixed the easy bugs, or that the fixes did not address the root cause, so another critical bug was opened. I disagree. The squad had no trouble finding easy bugs, and it too would have been fixing secondary bugs if the first fix was incomplete. I can tell you how the squad works on critical bugs, but not why it is successful.

I was surprised to see the Purple squad were still the top critical bug closers when it returned to maintenance after 15 months of feature work. How could that be?  The squad fixed a lot of old timeout and JavaScript bugs in the last few months through systemic changes — enough to significantly affect the statistics. About 600 critical bugs were closed while Purple squad were on feature work. The squad closed 210 of those bugs. 60 were regressions that were fixed within the iteration, so they never showed up in the backlog. 70 critical bugs were fixed because they blocked the feature, and 80 critical bugs were because Purple was the only squad awake when the issue was reported. The 4 other squads fixed an average of 98 bugs each when they were on maintenance. The Purple squad fixed more bugs then maintenance squads on average even when they were not officially doing maintenance work.  The data, charts, and analysis always includes the Purple squad.

I suspect the Purple squad has more familiarity with bugs in the critical backlog. They never stopped reading the critical bugs when they were on feature work. They saw opportunities to fix critical bugs while solving feature problems. I know some of the squad members are subscribed to all critical bugs and re-read them often. They triage and re-triage Launchpad bugs. This familiarity means that many bugs are ready to code — they know where the problem is and how to fix it before the work is assigned to them. They fixed many bugs in less than a day, often doing exactly what was suggested in the bug comments.

During the first week of their return to maintenance, about 30 critical bugs were discovered to be dupes of other bugs. Though this change does make the backlog count fall, it also revised all the data, so the chart is not showing these 30 bugs as at all now. The decline of backlog bugs does not include dupes. While the squad was familiar enough to find many bugs that they close in a single day, they were not so familiar as to have known that there were 30 duplicate bugs in the backlog when they started.

Most squad have only one person with DB access, but the Purple squad is blessed with 3 people who can test queries against production-level data. This could be a significant factor. It is nigh impossible to fix a timeout bug without proper database testing. Only 13 of the recent bugs closed were timeouts though. The access also helps plan proper fixes for other bugs as well, so maybe 20% of the fixed bugs can be attributed to database access.

Maybe the Purple squad are better maintenance engineers than other squads who work on maintenance. For 28 months, I was the leading bug closer working on Launchpad. I closed 3 times more bugs than the average Launchpad engineer. I am not a great engineer though. My “winning” streak came to a closed shortly after William Grant started working on Launchpad full time; he soundly trounced me over several months. Then he and I were put on the same squad and asked to fix critical bugs. Purple also had Jon Sackett, who was closing almost 2 times the number bugs than the other engineers. I don’t think I need to be humble on this matter. To use the vulgar, we rocked! Ian was the odd man on the Purple squad. He was the slowest bug closer, often going beyond our intended scope to fix an issue. Then Purple switched to feature work…Ian lept to the first rank while the rest of the squad struggled. Ian fixed almost double the number of Disclosure bugs than other squad members. The leading critical bug closer on the squad at the moment though is Steve Kowalik. This is his first time working on maintenance. His productivity has jumped since transitioning to maintenance.

I can only speculate as to why some engineers are better at maintenance, or can just close more bugs than others. A maintenance engineer must be familiar with the code and the rules that guide it. Feature engineers need to analyse issues and create new rules to guide code. I did not gradually become a leading bug closer, it happen in a single day when I realised while solving one issue that the code I was looking at was flawed, it certainly was causing a bug, I knew how to fix it, and with a few extra hours of extra effort, I could close two bugs in a single day. Closing bugs has always been easy since that moment.

I believe the Purple squad values certainty over severity and small scope over large scope when choosing which critical backlog bugs to fix. I created several charts that break the critical bugs into smaller categories. I suggested the squad burn down sub-categories of bugs like regressions, or 404s. The squad members are instead fixing bugs from the entire backlog. They are choosing bugs that they are certain they can fix in a few hours.  I think the squad has tacitly agreed to fix bugs that are less than a day of effort. When this group is exhausted, they will fix issues that require days of effort, but also fix as many bugs. The last bugs to be fixed will be those that require many days to fix a single bug. Fixing the bugs with the highest certainty reduced our churn through the critical bugs, there are fewer to triage, to dupe, to get ready to code.

The Purple squad avoids doing feature-level design and effort to fix critical bugs. Feature-level efforts entail more risk, more planning, and much more time. There is often no guarantee, low certainty, that a feature will fix the issue. A faster change with higher certainty can fix the issue, but leaves cruft in the code that the engineers do not like. Choosing to do feature-level fixes when a more certain fix is available indicates there is tension between the Launchpad users who have a “critical” issue that stops them from using Launchpad, and the engineers who have a “high” issue maintaining mediocre code. I contend it is easier to do feature-level work when you are not interrupted with maintenance issues. When the Purple squad does choose to do feature-level work to fix a critical, they have a list of the bugs they expect to fix, and they cut scope when fixing a single bug delays the fix of the others. The Launchpad Answers email subsystem was re-written when other options were not viable, there we about 20 leading timeouts represented by 5 specific bugs to justify 10 days of effort to fix them.

The Purple squad is not unique

Nothing that I have written explains why the Purple squad are better are closing critical bugs. All squads have roughly the same skills and make decisions like Purple. Maybe the issue is just a matter of degree. If the maintenance squad is not closing enough bugs to burn down the backlog, their time is consumed by triaging and duping new critical bug reports. Familiarity with Launchpad’s 1000’s of bugs is an advantage when triaging bugs and getting a bug ready to code. Being able to test queries yourself on a production-level database takes hours or days off the time needed to fix an issue. Familiarity with the code and the reasoning that guided it increases the certainty of success. The only domain that Purple is not comfortable working with is lp.translations; the squad is comfortable changing 90% of Launchpad’s code. There may be correlation between familiarity with code, and the facts that the squad members participated in the apocalypse that  re-organised the code base, and that some have a LoC credit count in the 1000’s.

Are you a technical writer -Want to work with amazing people and great team- Come join us

Friday, July 6th, 2012

We’re offering a unique opportunity to take part in one of the most exciting changes to affect the technology industry: the move to the cloud.

As Technical Writer in Canonical’s Launchpad team you’ll find the best way to ensure that users and developers of our software understand its benefits and how to use it. Whether it’s traditional documentation, screen casts or blog posts, you’ll find it easy to choose the right medium and you’ll have all the skills necessary to produce compelling, involving and effective content.

You’ll thrive in a rapidly changing environment where you’ll be expected to grasp new concepts quickly, develop an intimate understanding of three or four products simultaneously and determine the day to day shape of your own work.

A skilled writer, you’ll excel at finding the right information from your research and then communicating it with a casual confidence that puts people at ease and, most importantly, leaves them with the understanding they need to be effective.

Reporting to the team’s Product Manager, you’ll work as part of a fun-loving, highly skilled, global development team who produce tools including Launchpad, MAAS and Bazaar. You’ll share our love of hard work and our passion for free software, Ubuntu and the cloud.

  Key responsibilities and accountabilities

  •  Explain our products through traditional documentation, screencasts, podcasts and any other appropriate method.
  • Help ensure community and developer engagement with our platforms by documenting APIs and communicating the benefits of our various offerings.
  • Tell the story of the products we develop, through compelling blog posts and white papers.
  • Speak directly to the communities who use and develop our software in order to plan how you can best cater to their needs.

  Required skills and experience

  •  Your written English is well crafted, compelling and fun. You care about how you write, as much as what you write. You’ve produced end-user documentation, developer documentation, blog posts and white papers. What’s more, you enjoy doing it.
  • You have at least five years’ experience as technical writer, whether that’s professionally or as a consistent contributor to open source projects.
  • You’re smart: you find no problem in learning and owning a new concept.
  • When you speak, you find an instant rapport with your conversational partner or audience, and have no trouble in pitching your message appropriately.
  • When you listen, you ask all the right questions and can use the answers to create content that is appropriate to your audience and the information you need to communicate.
  • You live and breathe open source technology. You know the industry, understand the community and share the ideals. You know your OpenStack from your Intel, your ARM from your aaS and your Bugzilla from your Git.
  • You’re equally comfortable dealing with people in person, by phone, over email or using IRC and other remote communication tools.
  • You are willing to travel internationally, for periods of one or two weeks and occasionally longer, for conferences, developer-oriented meetings and sprints.

Desirable skills and experience

  •  You’ve worked as part of team building cloud-related technologies or developer tools.
  • You use Ubuntu and are familiar with Launchpad and Bazaar.
  • You have a familiarity with one or more of the following:
    • IaaS platforms such as OpenStack, AWS, Eucalyptus
    • Ubuntu Server, particularly in cloud contexts
    • ARM server
    • distributed version control systems
    • a form of Linux packaging, such as .deb or .rpm
    • Python development
  • You have taken an active role in an open source software project and understand the dynamics, demands and constraints of working in a distributed community of volunteer and paid developers.
  • You’ve worked as part of a distributed team and can demonstrate the self-motivation and discipline required in such an environment.

Apply online, if you have any questions drop by #launchpad

Meet Vincent Ladeuil who works in the Blue Squad on Launchpad

Tuesday, July 3rd, 2012


Laura:  What do you do on the Launchpad team?
Vincent: Maintenance. Although I’m eagerly waiting for the sprint with gmb to get some hints on how to handle the beast 🙂  In the mean time, I’m focusing on fixing bugs and making the udd importer more testable.

Laura: Can we see something that you’ve worked on?
Vincent:  and

Laura: Where do you work?
Vincent:  At home

Laura: What can you see from your office window? View from where I sit
Vincent: The venerable Strasbourg post office, lovely old stones.

Laura:  What did you do before working on the Launchpad team?
Vincent:  Developing bzr.

Laura:  What did you do before working at Canonical?
Vincent:Various service/consulting work for > 20 years, including some episodes at software editors.

Laura: How did you get into free software?
Vincent:  With pleasure

I think the most important event was in 1993: I encountered a blocking bug in g++ related to C++ templates (way before it was standardized). That was a roadblock, no work-around and it was Friday afternoon. In despair, I posted a reproducing case in the related newsgroup. When I came back to work on Monday I got an email telling me the bug was known *and* fixed *and* where to get the patch for the compiler.

That was a light-bulb instant: free software support could be far superior to commercial software support !

One week later, I got a second email asking me if I was out of trouble… Amazing, not only did I get a fix faster than I could have dreamed, but the guy *came back* to ensure I got it…

I never looked back.

Laura:  What’s more important? Principle or pragmatism?
Vincent:  Both are important. If you forget one, be prepared to pay the cost. Both are dangerous too if you forget the other:

– being pragmatic only most often means you’re adding to your tech debt or rely on others to finish your work,

– respecting principles excessively means you never deliver anything.

Laura: Do you/have you contribute(d) to any free software projects?
Vincent: bzr is my most important contributions (including a few plugins). I’ve occasionally sent patches to gtk, perl modules and various other bzr upstream projects.

Laura:  Tell us something really cool about Launchpad that not enough people know about.
Vincent:  Pass  🙂

Laura:  Is there anything in particular that you want to change in Launchpad?
Vincent:  Make it easier to test against for all projects that rely on it (I’m probably biased here as the udd importer severely suffer from not being able to properly test interactions with launchpad (read *and* write (branch creation mainly)).

Meet Jelmer Vernooij of the blue squad

Friday, June 29th, 2012

Laura: What do you do on the Launchpad team?
Jelmer: I’m one of the blue haired freaks on the Launchpad blue squad, although my hair isn’t actually blue – I’m sure we can fix this at the next squad sprint. At the moment, we are working on maintenance: fixing
critical bugs in Launchpad and dealing with incidents.

Laura: Can we see something that you’ve worked on?
Jelmer: I’ve contributed quite a bit to the code behind recipe builds. Most of my work has been on the backend though, not directly user-visible.

Laura: Where do you work?
Jelmer: Like most of us I work at home, which in my case is in Utrecht, the Netherlands. Occasionally I cowork with other teleworkers in Utrecht.

Laura: What can you see from your office window?
Jelmer: At the moment, I see just a big sad drapery made out of rain. On brighter days, I look out on a park and a canal.

Laura: What did you do before working on the Launchpad team?
Jelmer: The Blue squad, which I’m currently in, was originally the Bazaar team. Before that, I worked on the Launchpad team too. This was back in the days when there were no squads, but teams – I was in the Soyuz team. The inimitable Matt Revell interviewed me back in 2010:

Laura: How did you get into free software?
Jelmer: A long time ago, in high school, I ended up maintaining a few server machines running FreeBSD and Samba. After hitting some bugs, a dive into the source code followed to see what I could fix. I’ve been involved with various free software projects ever since.

Laura: What’s more important? Principle or pragmatism?
Jelmer: Do I really have to choose? That’s not very pragmatic.

Laura: Do you/have you contribute(d) to any free software projects?
Jelmer: Beside Launchpad, the main free software project I am involved in is Samba. There are several other projects that I have made major contributions to, such as Bazaar, CUPS, Wireshark, OpenChange, BitlBee.

I’m a Debian maintainer and Ubuntu uploader, mostly for projects I am involved in upstream. This knowledge comes in handy when working on the archive side of Launchpad.

Laura: Tell us something really cool about Launchpad that not enough people know about.
Jelmer: lists all the Launchpad builders and
the mischief they are up to.

Laura: Is there anything in particular that you want to change in Launchpad?
Jelmer: It would still be really nice to have dashboards of some kind in
Launchpad. There is even a LEP.

Meet John Meinel – Blue Squad leader and Papa Smurf

Wednesday, June 20th, 2012

Papa Smurf

Laura: What do you do on the Launchpad team?
John: I’m the team lead for the Blue squad. Right now our squad is on Maintenance, so I generally do the coordination work of our team with other teams and the system administrators.

Laura: Can we see something that you’ve worked on?
John: Before we switched to maintenance, our team was focusing on doing Bazaar work. Within the more recent time, we’ve done stuff like fixing up the Ubuntu Distributed Development package importer, and getting translations for Quantal started for Launchpad.

Laura:Where do you work?
John: I work from home in the Netherlands.

Laura: What can you see from your office window?
John: We have a forest near our house, and some of the other neighbors houses.

Laura: What did you do before working on the Launchpad team?
John: As mentioned, we were focused on development of Bazaar, though arguably that was still part of the Launchpad group (not officially, but in spirit).

Laura: What did you do before working at Canonical?
John: I worked for a company developing medical imaging algorithms, mostly for detection and visualization of disease.

Laura: How did you get into free software?
John: Our team wanted to use something better than CVS for development. At the time SVN was pretty hard to set up, and there was just the beginnings of a couple of tools for distributed version control. I got into tla at the beginning, and was happy when Canonical started Bazaar, and I was able to hack on it.

Laura: What’s more important? Principle or pragmatism?
John: I’m a fairly pragmatic engineer. I think it is good to use principle as a guideline, but in the end if the work isn’t in the hands of people using it, it is providing no benefit and is arguably wasted effort.

Laura: Do you/have you contribute(d) to any free software projects?
John: Well, Launchpad and Bazaar are both pretty clear things (and tla a little bit before that). I also developed some other tools while here at Canonical. Such as Meliae for profiling python memory.

Laura: Tell us something really cool about Launchpad that not enough people know about.
John: The UDD package importer turns the changes from debian packages into real VCS branches that you can do lots of nice stuff on. (annotate the history, log the history, see the graph over time, etc.)

I think we are at about 90% coverage, and you can do “bzr log ubuntu:package” to find out the recent history for a package in ubuntu.

Laura: Is there anything in particular that you want to change in Launchpad?
John: My own personal project with launchpad is improving the connection handling when accessing Bazaar branches. Right now it is approximately 3s just to do the ssh handshake and start talking to codehosting. I have some improvements that should decrease that significantly, but we encounter some strange hanging bugs that are only reproducible in production. And LP’s commitment to having minimal user-visible downtime is particularly problematic for SSH connections. A single HTTP request is less than 5s, but an SSH connection can legitimately be active for an hour if accessing a large project.