How Novacut uses Launchpad
Published by Laura czajkowski November 20, 2012 in General
Launchpad has been a key tool used in developing Novacut. I use Launchpad for code hosting, bug tracking, daily builds, and more. For almost two years I’ve been doing monthly stable releases on Launchpad, and Novacut now spans six separate Launchpad projects. To say the least, I’ve learned a lot about Launchpad in the process.
I don’t think Novacut could be where it is today without Launchpad, so I want to pass on some of what I’ve learned the past two years. Here are my five essential Launchpad best practices:
1. Daily Builds
I’m always very thankful that early on Paul Hummer took the time to school me on using Source Package Recipes to do daily builds. This Launchpad service gives you automated package builds across multiple architectures, and multiple Ubuntu releases.
I don’t know how to emphasize this enough, but seriously, you need daily builds. As a point of reference, daily builds are the 3rd item in the famed Joel Test.
These builds are triggered simply by making commits to the appropriate bzr branch on Launchpad (usually your trunk branch). You’ll automatically get up to one build per 24-hour period, and you can manually trigger additional builds when needed.
You can include your debian/ packaging directory in your project source tree, or you can keep debian/ in a separate bzr branch. For the Novacut components, I’ve found it most helpful to keep debian/ in the source trees because it’s handy to be able to land a code change and its corresponding packaging change in a single merge. This works for us because we currently can use the exact same debian/ for all the Ubuntu versions we support. If that’s not true for your project, you’ll need multiple debian/ branches.
For reference, here’s the Novacut Source Package Recipe.
2. Unit Tests
You should run your unit tests during your package builds, and you should fail the build when any unit test fails. This is particularly important for daily builds, because this will prevent a package with broken unit tests from reaching your daily PPA.
The Launchpad build servers are strict and unforgiving environments, which is a good thing when it comes to unit tests. The build servers are also probably quite different from your local development environment. On countless occasions our daily builds have caught failures that only occur on i386 (my workstation is amd64), or only occur on an Ubuntu release other than the one I’m running, etc.
To run your unit tests during the package build, you’ll need to modify your debian/rules file as appropriate. If you’re using debhelper, add an override_dh_auto_test target.
You might also need to add additional packages to the Build-Depends section of your debian/control file, packages that are needed by the unit tests but are otherwise not needed by the build itself.
For reference, here’s the debian/rules file used to run the Dmedia unit tests (which is also a handy Python3 example).
3. Track Ubuntu+1
When a new Ubuntu version opens up for development, I immediately start doing daily builds on the development version, even though I don’t typically upgrade my own computers till around 4 months into the cycle.
I use daily builds on the development release as an early warning system. With no extra effort on my part, these builds give me a heads-up about code or packaging changes that might be needed to make Novacut work well on the next Ubuntu release.
To enable daily builds on the next Ubuntu version, just go to your Source Package Recipe, click on “Distribution series”, and check the box for the newest series. Now you’ll have daily builds on the newest Ubuntu version, in addition to all the versions you were already building for.
For example, I’m currently in the process of enabling daily builds for Raring, as you can see in the Microfiber Source Package Recipe. And I did indeed encounter a build failure on Raring, seemingly caused by a debhelper issue.
For the first month or so in a cycle, I don’t tend to worry much about build failures on the development version. But after the dust has settled a bit, I make sure to keep the builds in working order, and I even do monthly stable releases for the Ubuntu development version. Again, I do all this pro-actively even before I personally start running the newest Ubuntu version.
4. PPAs & Users
Whenever someone asks me why I use Launchpad instead of github, my short answer is always, “PPAs and users”.
Source Package Recipes give you much more than just a build, they give you daily packages that are easily consumable by your testing community and early adopters. This tight feedback loop prevents you from running too far ahead without getting a good reality check from your target users.
Keep in mind that for some products, the early adopters willing to install from a PPA might not be all that representative of your target user. So when it comes to making design decisions, you might need to politely ignore certain feedback from some of these early adopters. In my experience, this wont cause any hard feelings as long as you have clearly communicated who your target user is, and why.
For reference, you might look at the way we’ve defined the Novacut target user.
I recommend creating PPA names that are well-branded and easy to remember. First, create a Launchpad team with the same name as your product. In our case, we have a ~novacut team. Second, I recommend creating a daily and a stable PPA owned by the same team. In our case, that gives us two easy to remember PPAs:
Although none of our target users (professional video editors) currently use Ubuntu to do their job, I’ve been surprised by how many follow Novacut’s development via our stable PPA, and even our daily PPA. This has helped keep us on track, and has helped us build customer loyalty even before we have a finished product.
For me personally, this daily user engagement also makes the design and development process more enjoyable. It’s hard to empathize with an abstract persona; it’s easier to solve specific problems for specific people.
5. Use Apport
Till recently I didn’t realize that you can use Apport for automated crash reporting in unofficial packages delivered through a PPA.
We haven’t had Apport integration for that long, but it’s already provided us with dozens of highly valuable crash reports. Almost immediately some hardware specific issues came to light and were fixed, convincing me that a key benefit of Apport is knowing how your app might misbehave on a larger, more variable pool of hardware.
Apport also helped some rare bugs come to light. I thought Dmedia was basically crash-free, but those one-in-a-thousand bugs pop out quickly when thousands of people are running it. Most of these bugs would have eventually been found by one of our core devs, but the quicker a bug is discovered, the quicker and easier the bug is to fix.
For more info, check out this blog post and this screencast, where I covered our Apport integration in detail.
And for reference, see the merge proposal that added Apport integration in Novacut.
A big thank you to Jason DeRose for sharing how his project uses Launchpad on a daily basis.
The information sharing feature is complete
Published by Curtis Hovey November 6, 2012 in General
Launchpad’s bug and branch privacy features were replaced by information sharing that permits project maintainers to share kinds of confidential information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.
The Disclosure features
Disclosure is a super feature composed on many features that will allow commercial projects to work in private. Untrusted users cannot see the project’s data. Project maintainers can share their project with trusted users to reveal all or just some of the project’s data. The ultimate goal is to create private project in Launchpad, but that feature required several other features to be completed first. The Purple squad worked on Trusted Pickers, Privacy Transitions, Hardened Projects, Social Private Teams, and Sharing.
There was a lot of overlap between each feature the Purple squad worked on. Though we could start each feature independent of one another, we could only complete about 90% of each. When the Sharing UI changes entered beta, we were unblocked and fixes about most of the remaining issues, but fixing all the issues required all projects to switch to Sharing. We did not consider Sharing, or any of the required features complete until we fixed all the bugs.
Disclosure facts
- Planning started in June 2010 to replace the existing privacy mechanisms with something that would scale.
- Early testing revealed that users did not trust Launchpad because the UI could not explain what was confidential, or what the consequences of a change would be — this needed to be fixed too.
- 149 related bugs were identified in Launchpad.
- Work started in June 2011 by the Purple squad.
- Replacing the old privacy mechanisms and addressing the trust and information issues took 16 months.
- About 45,000 lines code were added to support the features.
- About 15% of the lines were for missing JavaScript test coverage.
- More that 700 bugs were fixed in total.
- About 5% of the fixed bugs were caused by the old non-scaling privacy mechanisms.
- About 4% of the fixed bugs were caused by old JavaScript enhancements that broke features for non-JavaScript users.
Lessons learned
- Misrepresentation of what is confidential, or what will be confidential or public is very important to users — more important than supporting private data.
- Privacy/Sharing must be a first-class mechanism beneath all the mechanisms that work with confidential data.
- Privacy was added on top of bugs, and it failed to scale to 100’s of bugs.
- Privacy was added on top of branches, and it failed to scale to 1000’s of branches.
- Filtering private items in code, or in database joins is not fast enough to work with 100,000’s of items.
- Launchpad’s ReSTful object API is not suitable for working with large collections of objects like bugs or branches; a lighter, service-based approach was used to quickly work with large amounts of data.
- Users need to work with confidential data via the API, using a text web browser from servers, using a browser with accessibility tools, as well as the common case of using a JavaScript enabled browser.
- Lots of mock-ups and interactive tests will not predict all the interactions a user will have with real data; test with real code and data early to developer the final design.
Private projects for beta testers
Published by Deryck Hodge October 25, 2012 in General
If you are part of Launchpad’s beta testers team, you can now start trialing private projects on Launchpad. The private projects feature builds on the great sharing work that Launchpad’s Purple Squad has done, allowing Launchpad users to create true private projects now. A commercial subscription is required to use private projects, but any user who creates a private project on Launchpad will receive a 30 day trial commercial subscription.
When creating a new project on Launchpad, beta testers will have the option to create “Proprietary” or “Embargoed” projects. Embargoed exists for projects that intend to start private but later be revealed publicly. All other private projects should be proprietary. Milestones and series are proprietary or embargoed based on the project setting. To make them public, you will need to make the project itself public.
Be warned, this is a large change to Launchpad and there are certainly bugs in our handling of privacy. You can check out our list of known issues, if you’d like. We, as the Launchpad Orange Squad, are committed to fixing all of those before we leave beta. So don’t worry, we’re still actively working on this feature. We did, however, want users to begin using this feature to get early feedback on the work. Don’t trial your super secret project with this feature just yet, but if you have something safe to try out private projects, now is a good time for beta testers to get going with the feature.
Enjoy private projects on Launchpad now, beta testers! And please file any bugs you find.
Launchpad Workshop at UDS-R
Published by Laura czajkowski October 23, 2012 in General
After the success of the Launchpad clinics at the last UDS-Q we’ve decided to run some more! This time removed the sterile name of clinic and called them workshops.
If you want to get involved, scratch that itch, learn how to fix that irksome bug that has been bugging you’re not alone. Everyone probably has at least one that they’d like to see fixed. The problem is now knowing how to fix them or maybe they don’t know how to set up the Launchpad development environment, well lucky for you we have a lot of Launchpad developers at UDS-R and we’d like to help you help get bugs fixed!
The idea being if you have a bug you would like to fix, or pointed in the right direction that we’ll be there to help you get on the road to offer advice on every step of the Launchpad development process from Lines of code, to branch reviews to getting things done. We’ll have EC2 instances ready for you to develop on, so if you haven’t already gone through the process of setting up local Launchpad development on your machine, you don’t need to worry.
I have created a wiki page on which you should register if you’re going to be attending either of the clinics. Just list your name and the ID of the bug(s) you want to work on on that page. We’ll check the bugs out and get in touch with you if we think they’re too big to work on in the clinics – in which case we’ll try and work with you to get them fixed over a longer period. We’ve added the event to summit schedule, for Tuesday and Thursday of UDS so why not sign up and come along!
If you’ve never contributed before, Graham Binns has written a useful guide to contributing to Launchpad. He has also done up a screencast on fixing a bug in Launchpad.
Burning down critical bugs
Published by Curtis Hovey October 2, 2012 in General
I have been analysing Launchpad’s critical bugs to track the Purple squad’s progress while on Launchpad maintenance duty. In January of 2011, the Cloud Engineering team né Launchpad Engineering team was reorganised into squads, where one or more squads would maintain Launchpad while other squads work on features. This change also aligned with a new found effort to enforce the zero-oops policy. The two maintenance squads had more than 332 critical bugs to close before we could consider adding features that the stakeholders and community wanted. By July 2011, the count dropped to its lowest point, 250 known critical bugs. Why did the count stop falling for fifteen months? Why is the count falling again?
Charting and analysing critical bugs
The chart above needs some explanation to understand what is happening in Launchpad’s critical bugs over time. (You may want to open the image in a separate window to see everything in detail.) Each iteration is one week. The backlog represent the open critical bugs in launchpad at the start of the iteration. The future bugs are either bugs that are not discovered, not introduced, or reported and fixed within the iteration. The last group is crucial to understand the lines plotting the number of bugs fixed and added during the iteration. We strive to close critical bugs immediately. Most critical bugs are reported and fixed in a few days, so most bugs were not open long enough to be show up in the backlog. The number of bugs fixed must exceed the number added to make the backlog count fall. You can see that the maintenance squads have always been burning down the critical bugs, but if you are just watching the number of open bugs in Launchpad, you get the sense that the squads are running to just stand still.
I use the lp-milestone YUI widget to chart the bugs and analyse the our progress through the critical bugs. It allows me to summarise a set of bugs, or analyse a subset by bug tag.
Though 22 bugs were fixed this past week, 14 were added, thus the critical count dropped by 8. The last eight iterations are used to calculate the average bugs closed and open per iteration. The relative velocity (velocity – flux) is used to estimate the remaining number of days to drive the count to zero. When the Purple squad started maintenance on September 10th of 2012, the estimated days of effort was more than 1,200. In just three weeks, the number has fallen dramatically. The principle reason the backlog of critical bugs has fallen is that the Purple squad is now giving those bugs their full attention, but that generalisation is unsatisfactory.
Why is the Purple squad so good at closing bugs in the critical backlog?
I do not know the answer to my question. The critical backlog reached its all-time low of 250 bugs with the release of the Purple squad’s maintenance work in July 2011. There was supposition that Purple fixed the easy bugs, or that the fixes did not address the root cause, so another critical bug was opened. I disagree. The squad had no trouble finding easy bugs, and it too would have been fixing secondary bugs if the first fix was incomplete. I can tell you how the squad works on critical bugs, but not why it is successful.
I was surprised to see the Purple squad were still the top critical bug closers when it returned to maintenance after 15 months of feature work. How could that be? The squad fixed a lot of old timeout and JavaScript bugs in the last few months through systemic changes — enough to significantly affect the statistics. About 600 critical bugs were closed while Purple squad were on feature work. The squad closed 210 of those bugs. 60 were regressions that were fixed within the iteration, so they never showed up in the backlog. 70 critical bugs were fixed because they blocked the feature, and 80 critical bugs were because Purple was the only squad awake when the issue was reported. The 4 other squads fixed an average of 98 bugs each when they were on maintenance. The Purple squad fixed more bugs then maintenance squads on average even when they were not officially doing maintenance work. The data, charts, and analysis always includes the Purple squad.
I suspect the Purple squad has more familiarity with bugs in the critical backlog. They never stopped reading the critical bugs when they were on feature work. They saw opportunities to fix critical bugs while solving feature problems. I know some of the squad members are subscribed to all critical bugs and re-read them often. They triage and re-triage Launchpad bugs. This familiarity means that many bugs are ready to code — they know where the problem is and how to fix it before the work is assigned to them. They fixed many bugs in less than a day, often doing exactly what was suggested in the bug comments.
During the first week of their return to maintenance, about 30 critical bugs were discovered to be dupes of other bugs. Though this change does make the backlog count fall, it also revised all the data, so the chart is not showing these 30 bugs as at all now. The decline of backlog bugs does not include dupes. While the squad was familiar enough to find many bugs that they close in a single day, they were not so familiar as to have known that there were 30 duplicate bugs in the backlog when they started.
Most squad have only one person with DB access, but the Purple squad is blessed with 3 people who can test queries against production-level data. This could be a significant factor. It is nigh impossible to fix a timeout bug without proper database testing. Only 13 of the recent bugs closed were timeouts though. The access also helps plan proper fixes for other bugs as well, so maybe 20% of the fixed bugs can be attributed to database access.
Maybe the Purple squad are better maintenance engineers than other squads who work on maintenance. For 28 months, I was the leading bug closer working on Launchpad. I closed 3 times more bugs than the average Launchpad engineer. I am not a great engineer though. My “winning” streak came to a closed shortly after William Grant started working on Launchpad full time; he soundly trounced me over several months. Then he and I were put on the same squad and asked to fix critical bugs. Purple also had Jon Sackett, who was closing almost 2 times the number bugs than the other engineers. I don’t think I need to be humble on this matter. To use the vulgar, we rocked! Ian was the odd man on the Purple squad. He was the slowest bug closer, often going beyond our intended scope to fix an issue. Then Purple switched to feature work…Ian lept to the first rank while the rest of the squad struggled. Ian fixed almost double the number of Disclosure bugs than other squad members. The leading critical bug closer on the squad at the moment though is Steve Kowalik. This is his first time working on maintenance. His productivity has jumped since transitioning to maintenance.
I can only speculate as to why some engineers are better at maintenance, or can just close more bugs than others. A maintenance engineer must be familiar with the code and the rules that guide it. Feature engineers need to analyse issues and create new rules to guide code. I did not gradually become a leading bug closer, it happen in a single day when I realised while solving one issue that the code I was looking at was flawed, it certainly was causing a bug, I knew how to fix it, and with a few extra hours of extra effort, I could close two bugs in a single day. Closing bugs has always been easy since that moment.
I believe the Purple squad values certainty over severity and small scope over large scope when choosing which critical backlog bugs to fix. I created several charts that break the critical bugs into smaller categories. I suggested the squad burn down sub-categories of bugs like regressions, or 404s. The squad members are instead fixing bugs from the entire backlog. They are choosing bugs that they are certain they can fix in a few hours. I think the squad has tacitly agreed to fix bugs that are less than a day of effort. When this group is exhausted, they will fix issues that require days of effort, but also fix as many bugs. The last bugs to be fixed will be those that require many days to fix a single bug. Fixing the bugs with the highest certainty reduced our churn through the critical bugs, there are fewer to triage, to dupe, to get ready to code.
The Purple squad avoids doing feature-level design and effort to fix critical bugs. Feature-level efforts entail more risk, more planning, and much more time. There is often no guarantee, low certainty, that a feature will fix the issue. A faster change with higher certainty can fix the issue, but leaves cruft in the code that the engineers do not like. Choosing to do feature-level fixes when a more certain fix is available indicates there is tension between the Launchpad users who have a “critical” issue that stops them from using Launchpad, and the engineers who have a “high” issue maintaining mediocre code. I contend it is easier to do feature-level work when you are not interrupted with maintenance issues. When the Purple squad does choose to do feature-level work to fix a critical, they have a list of the bugs they expect to fix, and they cut scope when fixing a single bug delays the fix of the others. The Launchpad Answers email subsystem was re-written when other options were not viable, there we about 20 leading timeouts represented by 5 specific bugs to justify 10 days of effort to fix them.
The Purple squad is not unique
Nothing that I have written explains why the Purple squad are better are closing critical bugs. All squads have roughly the same skills and make decisions like Purple. Maybe the issue is just a matter of degree. If the maintenance squad is not closing enough bugs to burn down the backlog, their time is consumed by triaging and duping new critical bug reports. Familiarity with Launchpad’s 1000’s of bugs is an advantage when triaging bugs and getting a bug ready to code. Being able to test queries yourself on a production-level database takes hours or days off the time needed to fix an issue. Familiarity with the code and the reasoning that guided it increases the certainty of success. The only domain that Purple is not comfortable working with is lp.translations; the squad is comfortable changing 90% of Launchpad’s code. There may be correlation between familiarity with code, and the facts that the squad members participated in the apocalypse that re-organised the code base, and that some have a LoC credit count in the 1000’s.
Mercurial imports will end on October 5th
Published by Matthew Revell September 27, 2012 in Code, Notifications
On the 5th of October we’ll be ending our beta of Mercurial imports in Launchpad. On that day your existing Mercurial imports will cease and you won’t be able to create new ones.
This doesn’t affect Bazaar, Git, Subversion or CVS imports.
You’re probably wondering why. During the beta, we found that not many people wanted to import Mercurial branches into Launchpad. Today there are only around forty people using the facility. It’s also fair to say that our importer wasn’t of the quality we want for Launchpad.
So, with low demand for the feature we decided to focus engineering effort elsewhere rather than continue to maintain, or fix up, a less than satisfactory feature.
I’m sorry if you currently rely on Launchpad to import code from Mercurial into Bazaar. You can, though, still use the bzr-hg plugin locally.
Launchpad JavaScript now combo loaded and faster than ever.
Published by Richard Harding September 25, 2012 in Cool new stuff, General, Performance
Updated network graph
Back in January a side project was started to update the JavaScript used in Launchpad. Launchpad has been using YUI 3.3.0 for a long time, very successfully, however recent advances in YUI 3.5 and higher have added some great tools for development that Launchpad can take advantage of. In order to facilitate easier upgrades our YUI library version Launchpad has been moved to using a combo loader for serving out JavaScript.
This means, that instead of a single launchpad.js file that can be upwards of 3MB in size, each request builds a list of JavaScript modules needed for the current page to work, and the combo loader only sends down those modules. This drastically cuts down on the download size of the JavaScript for users. These combo loaded JavaScript files are also cached for speedy serving to other users of Launchpad.
The combo loader also allows us to specify which YUI version to load via a tweak to the url. In this way we can easily test new version of YUI side by side with the current stable version as they come out. This allows Launchpad to keep with future YUI released much faster.
We’re excited that today Launchpad has moved from YUI 3.3.0 to 3.5.1 and is now served by the combo loader. This change provides a faster experience for users along with easier maintenance and new JavaScript library features for developers.
We’ve still got more to do though. YUI just released version 3.7 and we aim to push that into production faster than ever before. Please let us know how these changes work for you.
Launchpad also wants to thank the folks over at YUI for continuing the great work on a tool that Launchpad heavily depends on.
Parallel testing is live
Published by Laura czajkowski September 24, 2012 in Performance
One of the projects the Launchpad Squads (yellow) have been working on has been the Parallel Testing during the last cycle, this has now been completed and is now in operation. WebOps have today finished setting up parallel testing in buildbot. Buildbot-poll has been updated to know about the new builders, and the developers have confirmed that [testfix] and automatic stable merging etc. work fine. Nothing should have changed except that builds now take 35 minutes rather than 6.5 hours.
If something goes wrong, http://lpbuildbot.canonical.com/waterfall and https://dev.launchpad.net/yellow/ParallelTestingTroubleshooting may be helpful. The buildbot master is still praseodymium, but the slaves are new: sluagh for devel, and radande for db-devel. If you need packages upgraded on the slaves, poke WebOps as before.
If you would like to follow the discussion on this topic you’ll find more on the Launchpad development mailing list
Privacy for blueprints enabled for beta testers
Published by Deryck Hodge September 17, 2012 in General
To go along with recent work to enable information sharing for bugs and branches, we are now enabling privacy for blueprints for beta testers. This means that blueprints now support some of the different information types that bugs and branches also support. For projects with a commercial subscription on Launchpad, this means blueprints can now be set to proprietary or embargoed. Project owners can also manage sharing for blueprints from their project’s sharing details page. For more on how sharing itself works, see Curtis’ blog post that announced that Information sharing is now in beta for everyone.
We have some minor fit-n-finish issues to complete, like nicer UI elements, and of those, we have one last known bug in progress — we know that blueprints don’t currently honor the sharing policy default when new blueprints are created. However, we thought it was worth getting this work to beta testers now to start getting feedback on this as we turn to finishing off the privacy work that is left to do.
Enjoy privacy for blueprints, beta testers! And please file bugs on any issues you find.
Launchpad Builders update
Published by Laura czajkowski September 16, 2012 in Performance
We have recovered some of the affected builders, more will be coming back on later this week with the remainder to come in a few weeks when we have new hardware. Until then launchpad builders will be at a reduced build capacity. l
We apologise for the inconvenience and we’re sorry for the disruption to your service.
