That Juju that you do (Part I: Bring the pain)
Published by Graham Binns March 30, 2012 in General
Benji’s blog post earlier this week gave you all some insight into what the Launchpad Yellow Squad has been doing recently in its attempt to parallelise the Launchpad test suite. One of the side effects of this is that we’ve been making quite a lot of use of Juju, and we thought it’d be nice to actually spell out what we’ve been doing.
The problem
We’re working to parallelise Launchpad’s test suite so that it doesn’t take approximately one epoch to get a branch from being approved for merging until it lands. A lofty goal, sure, and one that presents some interesting problems from the perspective of building an environment to test our work in. You see, Launchpad’s build infrastructure is a pretty complicated beast. It’s come a long way since the time when submitting a branch for merging meant sending an email to our PQM bot, which would then run the test suite and kick the branch out if it failed, but now it’s something of a behemoth.
Time for some S&M
We use Buildbot as our continuous integration system. There are two parts to Buildbot: the master and the slave. Broadly put, the slave is the part of Buildbot that is responsible for doing the actual work of compilation and running tests and the master is responsible for telling the slave when to do things. Each master can be responsible for several slaves. When it became obvious that we were going to need to essentially replicate our existing setup in order to test our parallelisation work, we considered asking Canonicals system administrators, in our sweetest tones, to give us a box upon which to do our testing work, but we spotted two reasons that this would be problematic:
- We didn’t actually know at the outset what the best architecture was for our project.
- Asking for a machine without knowing what you actually need is likely to earn you a look so old it could have come from an ammonite, at least if you have sensible sysadmins.
So instead, the obvious solution: use Amazon EC2. After all, that would allow us to play with different architectures without there being any huge cost in terms of physical resources. Moreover, we’d be able to have root access on the instances on which we were testing, which makes debugging such a complicated process so much easier.
However…
There was still a problem. How to actually set up the test instances, given that there are five of us spread between three timezones, that it takes a significant amount of time to set up a machine for Launchpad development, and finally that we don’t really want to leave EC2 instances running overnight if we don’t have to (because it’s expensive).
The sequence of steps we’d have to take to up an instance tends to look something like this:
- Launch a new EC2 instance (this happens pretty quickly, thanks, Amazon)
- Make sure that everyone’s public SSH keys are usable on that instance
- Run our Launchpad setup script(s) (this takes about an hour, usually).
- Install buildbot.
- Configure buildbot correctly as master or slave.
- Run buildbot (or buildslave, if this is a slave) and make sure it’s hooked up correctly to the other type of buildbot.
- Get some code into buildbot and make it run the test suite.
Parallelising the Unparallelisable
Published by Benji York March 27, 2012 in General
Launchpad has a lot of tests, almost 20,000. There are tests that make sure the internals work as expected, that verify the Javascript works in web browsers, and everything in between. In a perfect world those tests would only take seconds to run. In this world they take hours; six hours on our current continuous integration machines, for instance.
These long-running tests severely impact the time it takes to develop and deploy changes to Launchpad. We would like to improve the situation.
Given that the test cases are theoretically independent of one another, the obvious thing to do is to run the tests in parallel on a multi-core machine. Unfortunately many of the tests interact with the environment (databases, memcached, temporary directories, etc.) and conflict if run simultaneously.
Enter LXC
What we need is a way to isolate the test processes from each another. Virtual machines would allow us to do that, but the overhead and heavy-weight setup makes them unappealing. That’s where LXC (Linux Containers) comes in handy. LXC
allows the easy creation of “containers” that are isolated from the “host” machine without the performance overhead of VMs.
For example, to create a new container use lxc-create:
lxc-create -n test -t ubuntu
The container can then be started:
lxc-start -n test -d
And we can connect to it via SSH (using the default username and password shown during creation, if applicable):
ssh ubuntu@test
There are many options for customising the containers, including mounting a portion of the host’s file system in the container so sharing files between the two is easy.
Getting Ephemeral
All this is very nice for running isolated, parallel test runs but setting up and managing eight or more containers (one per core) is
off-putting, so we have used (and improved) a new LXC feature, “ephemeral” containers (created with lxc-start-ephemeral).
Ephemeral containers are “clones” of a base container and can have a temporary file system that reflects the contents of the base container but any writes are stored in-memory and are not written to disk. This allows us to install Launchpad on a single base container and then spawn many ephemeral containers, each with their own list of tests to run.
The ephemeral containers can then write to their local file systems without interfering with the others running simultaneously. The
containers may also benefit from faster IO because of the file system changes being stored in memory.
Results
We are still working out the kinks in our approach and wrestling with the occasional LXC bug as well as bugs in the Launchpad test suite itself. Even so we have already shortened a full test run on an eight-core EC2 instance down to 45 minutes; a substantial improvement over the current six hours.
(Image by Tolka Rova, Creative Commons license)
Why there is always time
Published by Dan Harrop-Griffiths March 21, 2012 in General
One of the main obstacles I come across when putting forward ideas for user testing for a project, is time:
“We’re on a very tight deadline, we can’t fit in testing,” “Can we leave it until the next release? There isn’t enough time at the moment.” “We just haven’t added in the time for all that user testing stuff.”
But the good news is – there is always time.
User testing, or usability testing (which is what we should really call it as we’re testing if things are usable, not testing the users themselves) can be extremely flexible. It can range from a detailed study of hundreds of painstakingly selected users, conducted in specially constructed labs with hidden screens, video recording devices and microphones, costing thousands of credits, with months to analyse and report the results. On the other end of the scale, it can simply be asking someone you pass in the corridor to look at a quick sketch of a wireframe you’ve made on the back of a napkin.
User testing can be both of these things, and everything in between, and yes, this can depend on time, and of course the other buzzword that sits so closely next to it – money. The thing is, it’s always better to do something, rather than nothing, however tight a deadline is – even if that is just asking a few users to try out a particular feature or function that you’re developing – whether this be with a flat mock-up or a working prototype.
Setting up some basic tests with a handful of users, running them and then writing up the results doesn’t need to take more than a day or two. The results will be pretty simple, and depending on the tests, will more likely be useful as a sense-check than a source of detailed information on user behaviour or working patterns, but this is still valuable stuff that can make or break a new feature. The results will broadly have one of three outcomes – user’s just didn’t ‘get it’ and there are big problems to be fixed; there are smaller problem’s that have slipped everyone’s mind but the user’s found fairly quickly; or (rarely, almost never) everything was perfect and the users had a seamless, faultless experience.
After I’ve reached this point in the discussion, I sometimes come across another potential user research blocker…
“But there’s no point in finding this out, we don’t have enough time to change things before our deadline.”
It may be true that there’s no time to redesign a feature based on recommendations from user testing results in your current cycle – but it’s better to go into the next phase of a project already knowing at least a bit about what user’s think. If you’re in the final stage of a project, these kind of problems can be treated as bugs and ticked off one at a time.
It’s easy to become blinkered with a project, working with the same concepts, terminology and use patterns day after day – it can become hard to think – “if I was looking at all this stuff for the first time, would it make sense?” User testing in its quickest and simplest form aims to answer this question. And that’s something there’s always time for.
Meet Laura Czajkowski
Published by Dan Harrop-Griffiths March 16, 2012 in General, Meet the devs
Dan: What’s your role on the Launchpad team?
Laura: I’m the Launchpad Support Specialist, so my job is pretty varied each day. Launchpad is rather larger than I first ever thought or had experience in using but it’s great to see so many people use it on a daily basis.
My role is to help people via email or IRC with their queries or point them in the right direction of where they can get more information or submit a bug or help them achieve something. I also look after Launchpad bugs and questions each day and it’s fascinating to see the varying questions we get on there so it’s a great way to learn and also see what interesting projects are on Launchpad and the communities that use it.
Dan: You’ve been working on Launchpad as a community member for a while though yeah?
I’ve been using it in the Ubuntu community in the past for blueprints, reporting bugs and and tracking issues and the odd time if I can help out in translations.
Dan: What’s been the biggest challenge in your new role so far?
Laura: Bazaar and PPAs both of which are bizarre to me at present, but the folks in the Launchpad and Bazaar teams have been really helpful to me and really made working with them easy.
Dan: Where do you work, and what can you see from your window?
Laura: I live in London, and work from home four days a week so when I look out the window I see the reflection of the London Eye. The other day a week I head into Canonical HQ.
Dan: If time/money was not an issue, what would you change about Launchpad?
Laura: Oh I’d love to make Launchpad translatable as I do know many people who love to get more involved, having it translated would help here. I’d also love to get more of the developer community involved in Launchpad, and where Launchpad isn’t doing what they’d like get them to submit patches and get them more involved in the process. It’s open source after all 🙂
Dan: How did you first start to get involved in the open source community?
Laura:I got involved when I was in college where I was roped into joining our computer society Skynet. Soon I became treasurer and event organiser and then eventually president of the society and got involved running open source conferences. Never looked back since!
Work items in blueprints
Published by Matthew Revell March 15, 2012 in Cool new stuff
At its most basic, a blueprint in Launchpad is a statement of intent: by creating a blueprint, you’re saying, “Here’s an idea for how this project should progress.”
How do you track the detailed steps to get from having the idea to seeing it implemented?
The Ubuntu development community came up with a solution: they break a blueprint down into items of work that can be assigned to an individual. That makes it easier to track who’s doing what and how they’re getting on, while the blueprint itself tracks the status of the big idea.
Until now, these work items weren’t recognised by Launchpad. Instead, Ubuntu and Linaro tracked them in the blueprint’s whiteboard. Using the Launchpad API, they pulled the whiteboard contents out and used them to generate burn-down charts for status.ubuntu.com and status.linaro.org.
Now, thanks to work by some of the team at Linaro, each blueprint has a text box where you can add, update and delete its related work items.
How you make use of work items will depend on the project you’re working on. Both Ubuntu and Linaro already make extensive use of work items and, if you’re working as part of those communities, you should follow their lead.
Using work items
Work items are pretty simple to use.
Here’s an example of how a set of work items might look:
Design the user interaction: DONE
Visual design: INPROGRESS
Test the UI: TODO
Decide on an API format: POSTPONED
Bootstrap the dev environment: TODO
Set up Jenkins CI: TODO
As you can see, each work item sits on its own line and you indicate its status after a colon.
As you can see, each work item sits on its own line and you indicate its status after a colon. The statuses you can use are:
- TODO
- INPROGRESS
- POSTPONEDDONE
Sometimes a blueprint can be targeted to one milestone but you’ll have a individual work items that need to be targeted to a different milestone. That’s fine, you can tell Launchpad that a bunch of work items are for some other milestone.
In the work items text box, simply use a header like this:
Work items for <milestone>:
If you enter an invalid milestone you’ll get an error message and the opportunity to correct the text field.
A milestone picker will of course be more useful and we’ve submitted a bug to keep track of that.
What’s next?
Soon, you’ll be able get a view of all the work items and bugs assigned to a particular individual or team for a particular milestone.
Of course, if you have other ideas for how Launchpad can make use of work items, we’d love to hear from you.
Photo by Courtney Dirks. Licence: CC BY 2.0
Reimagining the nature of privacy in Launchpad (part 3)
Published by Curtis Hovey February 21, 2012 in Bug Tracking, Code, Coming changes
We are reimagining the nature of privacy in Launchpad. The goal of the disclosure feature is to introduce true private projects, and we are reconciling the contradictory implementations of privacy in bugs and branches.
Launchpad will use policies instead of roles to govern who has access to a kind of privacy
We are implementing three kinds of policies, proprietary, embargoed security, and user data. The maintainer is the default member of these policies. The maintainer can share a kind or private data by adding a user or team to a policy.
For proprietary projects, the maintainer can add their organisational teams to the proprietary policy to share all the project information with the team members.
For Ubuntu, the maintainer will set the apport bot to be the only user in the user data policy; user data is only shared with a bot that removes personal data so that the bug can be made public. The Ubuntu security team will be the only users in the security policy.
Most maintainers will want to add project drivers to the policies if they use drivers. Bug supervisors can be added as well, though the team must be exclusive (moderated or restricted).
You can still subscribe a user or team to a private bug or branch and Launchpad will also permit the user to access it without sharing everything with the user. The existing behaviour will continue to work but it will be an exception to the normal rules.
Polices replace the bug-subscription-on-privacy-change rules. If you have every had to change the bug supervisor for a project with many private bugs, you can rejoice. You will not need to manually update the subscriptions to the private bugs to do what Launchpad implied would happen. Subscriptions are just about notification. You will not be notified of proprietary changes is proprietary information is not shared with you. Sharing kinds or information via policy means that many existing private bugs without subscribers will finally be visible to project members who can fix the issue.
Reimagining the nature of privacy in Launchpad (part 2)
Published by Curtis Hovey February 15, 2012 in API, Bug Tracking, Code, Coming changes, General
We are reimagining the nature of privacy in Launchpad. The goal of the disclosure feature is to introduce true private projects, and we are reconciling the contradictory implementations of privacy in bugs and branches.
We are adding a new kind of privacy called “Proprietary” which will work differently than the current forms of privacy.
The information in proprietary data is not shared between projects. The conversations, client, customer, partner, company, and organisation data are held in confidence. proprietary information is unlikely to every be made public.
Many projects currently have private bugs and branches because they contain proprietary information. We expect to change these bugs from generic private to proprietary. We know that private bugs and branches that belong to projects that have only a proprietary license are intended to be proprietary. We will not change bugs that are public, described as security, or are shared with another project.
This point is a subtle change from what I have spoken and written previously. We are not changing the current forms of privacy. We do not assume that all private things are proprietary. We are adding a new kind of privacy that cannot be shared with other projects to ensure the information is not disclosed.
Launchpad currently permits projects to have default private bugs and branches. These features exist for proprietary projects. We will change the APIs to clarify this. eg:
project.private_bugs = True => project.default_proprietary_bugs = True
project.setBranchVisibilityTeamPolicy(FORBIDDEN) => project.default_proprietary_branches = True
Projects with commercial subscriptions will get the “proprietary” classification. Project contributors will be able to classify their bugs and branches as proprietary. The maintainers will be able to enable default proprietary bugs and branches.
Next part: Launchpad will use policies instead of roles to govern who has access to a kind of privacy.
Contacting teams is easier and more reliable
Published by Curtis Hovey February 14, 2012 in Cool new stuff
Two changes to the contact team email feature were recently released that make communication more reliable.
Non-team members now see the “Contact this team’s admins” link. Previously, non-members saw a link to contact the team owner. The owner is often one person, and some team owners delegate the running of the teams to the team admins. There are often more admins then there are owners. So the message is more likely to reach someone who is involved in the day-to-day team management.
Team members see the “Contact this team’s members” link. Previously, members might have seen a link to contact the team, but that email when to the team email address that the team might not respond to. Many teams still use bogus email addresses to avoid emails from the days before Launchpad had great bug subscription filters. Launchpad has an anti-feature that prevents team mailing-lists from contacts all the team members. Team admins found it impossible to notify the member about issue ranging from policy changes, polls, meetings, to security issues. So members can now trust that their messages will be sent to the team members.
The feature is a convenient way to contact a user or team. Sometimes the feature is the only way you can contact a user or team that does not have a public email address. A user may use the contact user/team feature 3 times each day. The limit ensures no one can spam Launchpad users. The limit also means the feature is not a substitute for mailing lists.
Reimagining the nature of privacy in Launchpad (part 1)
Published by Curtis Hovey February 13, 2012 in Bug Tracking, Code, Coming changes, General
We are reimagining the nature of privacy in Launchpad. The goal of the disclosure feature is to introduce true private projects, and we are reconciling the contradictory implementations of privacy in bugs and branches.
We must change the UI to accommodate the a kind of privacy, and we must change some existing terms because to avoid confusion.
We currently have two checkboxes, Private and Security that create 4 combined states:
- Public
- Public Security
- Private Security
- Private something else
Most private bugs in Launchpad are private because they contain user data. You might think at first that something that is just private is proprietary. This is not the so. Ubuntu took advantage of defects in Launchpad’s conflation of subscription and access to address a kind of privacy we did not plan for. Most private bugs in Launchpad are owned by Ubuntu. They were created by the apport bug reporting process and may contain personal user data. These bugs cannot be made public until they are redacted or purged of user data. We reviewed a sample of private bugs that belong to public projects and discovered more than 90% were made private because they contained user data. Since project contributors cannot hide or edit bug comments, they chose to make the bug private to protect the user. Well done. Launchpad needs to clarify when something contains user data so that everyone knows that it cannot
be made public without removing the personal information.
Public and private security bugs represent two states in a workflow. The goal of every security bug is to be resolved, then made public so that users are informed. People who work on these issues do not use “public” and “private”, they use “unembargoed” and “embargoed”.
Also, when I view something that is private, Launchpad needs to tell me why. The red privacy banner shown on Launchpad pages must tell me why something is private. Is it because the page contains user data, proprietary information, or an embargoed security issue? This informs me if the thing could become public.
When I want to change somethings visibility, I expect Launchpad to show me a choice that clearly states my options. Launchpad’s pickers currently shows me a term without an explanation, yet Launchpad’s code does contain the term’s definition. Instead of making me search help.launchpad.net (in vain), the picker must inform me. Given the risks of disclosing personal user data or proprietary information, I think an informative picker is essential. I expect to see something like this when I open the visibility picker for a bug:
Branches require a similar, if not identical way of describing their kind of information. I am not certain branches contain user data, but if one did, it would be clear that the branch should not be visible to everyone and should not be merged until the user data is removed.
Next post: We are adding a new kind of privacy called “Proprietary” which will work differently than the current forms of privacy.
Faster deployments
Published by Francis J. Lacoste February 3, 2012 in General
Back in September, we announced our first fastdowntime deployment. That was a new way to do deployment involving DB changes. This meannt less downtime for you the user, but we were also hoping that it would speed up our development by allowing us to deliver changes more often.
How can we evaluate if we sped up development using this change? The most important metric we look at when making this evaluation is cycle time. That’s the time it takes to go from starting to make a change to having this change live in production. So before fastdowntime, our cycle time was about 10 days, and it is now about 9 days. So along the introduction of this new deployment process, we cut 1 day off the average, or a 10% improvement. That’s not bad.
But comparing the cumulative frequency distribution of the cycle time with the old process and the new will give us a better idea of the improvement.
On this chart, the gap between the orange (fastdowntime deployment) and blue (original process) lines shows the improvement to us. We can see that more changes were completed sooner. For example, under the old process about 60% of the changes were completed in less than 9 days whereas about 70% were completed under the same time in the new process. It’s interesting to note that for changes that took less than 4 days to complete or that took more than 3 weeks to complete, there is no practical difference between the two distributions. We can explain that by the fact that things that were fast before are still fast, and things that takes more than 3 weeks would usually have also encountered a deployment point in the past.
That’s looking at the big picture. Looking at the overall cycle time is what gives us confidence that the process as a whole was improved. For example, the gain in deployment could have been lost by increased development time. But the closer picture is more telling.
The cycle time charted in this case is from the time a change is ready to be deployed until it’s actually live. It basically excludes the time to code, review, merge and test the changes. In this case, we can see that 95% of the changes had to wait less than 9 days to go live under the new process whereas it would take 19 days previously to get the same ratio. So an
improvement of 10 days! That’s way more nice.
Our next step on improving our cycle time is to parallelize our test suite. This is another major bottleneck in our process. In the best case, it usually takes about half a day between the time a developer submits their branch for merging until it is ready for QA on qastaging. The time in between is passed waiting and running the test suite. It takes about 6 hours to our buildbot to validate a set of revisions. We have a project underway to run the tests in parallel. We hope to reduce the test suite time to under an hour with it. This means that it now would be possible for a developer to merge and QA a change on the same day! With this we expect to shave another day maybe two from the global cycle time.
Unfortunately, there are no easy silver bullets to make a dent in the time it takes to code a change. The only way to be faster there would be to make the Launchpad code base simpler. That’s also under way with the services oriented architecture project. But that will take some time to complete.
Photo by Martin Heigan. Licence: CC BY NC ND 2.0.
