launchpadblog

Benji’s blog post earlier this week gave you all some insight into what the Launchpad Yellow Squad has been doing recently in its attempt to parallelise the Launchpad test suite. One of the side effects of this is that we’ve been making quite a lot of use of Juju, and we thought it’d be nice to actually spell out what we’ve been doing.

The problem

We’re working to parallelise Launchpad’s test suite so that it doesn’t take approximately one epoch to get a branch from being approved for merging until it lands. A lofty goal, sure, and one that presents some interesting problems from the perspective of building an environment to test our work in. You see, Launchpad’s build infrastructure is a pretty complicated beast. It’s come a long way since the time when submitting a branch for merging meant sending an email to our PQM bot, which would then run the test suite and kick the branch out if it failed, but now it’s something of a behemoth.

Time for some S&M

We use Buildbot as our continuous integration system. There are two parts to Buildbot: the master and the slave. Broadly put, the slave is the part of Buildbot that is responsible for doing the actual work of compilation and running tests and the master is responsible for telling the slave when to do things. Each master can be responsible for several slaves. When it became obvious that we were going to need to essentially replicate our existing setup in order to test our parallelisation work, we considered asking Canonicals system administrators, in our sweetest tones, to give us a box upon which to do our testing work, but we spotted two reasons that this would be problematic:

1. We didn’t actually know at the outset what the best architecture was for our project.
2. Asking for a machine without knowing what you actually need is likely to earn you a look so old it could have come from an ammonite, at least if you have sensible sysadmins.

So instead, the obvious solution: use Amazon EC2. After all, that would allow us to play with different architectures without there being any huge cost in terms of physical resources. Moreover, we’d be able to have root access on the instances on which we were testing, which makes debugging such a complicated process so much easier.

However…

There was still a problem. How to actually set up the test instances, given that there are five of us spread between three timezones, that it takes a significant amount of time to set up a machine for Launchpad development, and finally that we don’t really want to leave EC2 instances running overnight if we don’t have to (because it’s expensive).

The sequence of steps we’d have to take to up an instance tends to look something like this:

1. Launch a new EC2 instance (this happens pretty quickly, thanks, Amazon)
2. Make sure that everyone’s public SSH keys are usable on that instance
3. Run our Launchpad setup script(s) (this takes about an hour, usually).
4. Install buildbot.
5. Configure buildbot correctly as   master or slave.
6. Run buildbot (or buildslave, if this is a slave) and make sure it’s hooked up correctly to the other type of buildbot.
7. Get some code into buildbot and make it run the test suite.

Launchpad has a lot of tests, almost 20,000. There are tests that make sure the internals work as expected, that verify the Javascript works in web browsers, and everything in between. In a perfect world those tests would only take seconds to run. In this world they take hours; six hours on our current continuous integration machines, for instance.

These long-running tests severely impact the time it takes to develop and deploy changes to Launchpad. We would like to improve the situation.

Given that the test cases are theoretically independent of one another, the obvious thing to do is to run the tests in parallel on a multi-core machine. Unfortunately many of the tests interact with the environment (databases, memcached, temporary directories, etc.) and conflict if run simultaneously.

Enter LXC

What we need is a way to isolate the test processes from each another. Virtual machines would allow us to do that, but the overhead and heavy-weight setup makes them unappealing. That’s where LXC (Linux Containers) comes in handy. LXC

allows the easy creation of “containers” that are isolated from the “host” machine without the performance overhead of VMs.

For example, to create a new container use lxc-create:

lxc-create -n test -t ubuntu

The container can then be started:

lxc-start -n test -d

And we can connect to it via SSH (using the default username and password shown during creation, if applicable):

ssh ubuntu@test

There are many options for customising the containers, including mounting a portion of the host’s file system in the container so sharing files between the two is easy.

Getting Ephemeral

All this is very nice for running isolated, parallel test runs but setting up and managing eight or more containers (one per core) is
off-putting, so we have used (and improved) a new LXC feature, “ephemeral” containers (created with lxc-start-ephemeral).

Ephemeral containers are “clones” of a base container and can have a temporary file system that reflects the contents of the base container but any writes are stored in-memory and are not written to disk. This allows us to install Launchpad on a single base container and then spawn many ephemeral containers, each with their own list of tests to run.

The ephemeral containers can then write to their local file systems without interfering with the others running simultaneously. The
containers may also benefit from faster IO because of the file system changes being stored in memory.

Results

We are still working out the kinks in our approach and wrestling with the occasional LXC bug as well as bugs in the Launchpad test suite itself. Even so we have already shortened a full test run on an eight-core EC2 instance down to 45 minutes; a substantial improvement over the current six hours.

(Image by Tolka Rova, Creative Commons license)

One of the main obstacles I come across when putting forward ideas for user testing for a project, is time:

“We’re on a very tight deadline, we can’t fit in testing,” “Can we leave it until the next release? There isn’t enough time at the moment.” “We just haven’t added in the time for all that user testing stuff.”

But the good news is – there is always time.

User testing, or usability testing (which is what we should really call it as we’re testing if things are usable, not testing the users themselves) can be extremely flexible. It can range from a detailed study of hundreds of painstakingly selected users, conducted in specially constructed labs with hidden screens, video recording devices and microphones, costing thousands of credits, with months to analyse and report the results. On the other end of the scale, it can simply be asking someone you pass in the corridor to look at a quick sketch of a wireframe you’ve made on the back of a napkin.

User testing can be both of these things, and everything in between, and yes, this can depend on time, and of course the other buzzword that sits so closely next to it – money. The thing is, it’s always better to do something, rather than nothing, however tight a deadline is – even if that is just asking a few users to try out a particular feature or function that you’re developing – whether this be with a flat mock-up or a working prototype.

Setting up some basic tests with a handful of users, running them and then writing up the results doesn’t need to take more than a day or two. The results will be pretty simple, and depending on the tests, will more likely be useful as a sense-check than a source of detailed information on user behaviour or working patterns, but  this is still valuable stuff that can make or break a new feature. The results will broadly have one of three outcomes – user’s just didn’t ‘get it’ and there are big problems to be fixed; there are smaller problem’s that have slipped everyone’s mind but the user’s found fairly quickly; or (rarely, almost never) everything was perfect and the users had a seamless, faultless experience.

After I’ve reached this point in the discussion, I sometimes come across another potential user research blocker…

“But there’s no point in finding this out, we don’t have enough time to change things before our deadline.”

It may be true that there’s no time to redesign a feature based on recommendations from user testing results in your current cycle – but it’s better to go into the next phase of a project already knowing at least a bit about what user’s think. If you’re in the final stage of a project, these kind of problems can be treated as bugs and ticked off one at a time.

It’s easy to become blinkered with a project, working with the same concepts, terminology and use patterns day after day – it can become hard to think – “if I was looking at all this stuff for the first time, would it make sense?” User testing in its quickest and simplest form aims to answer this question. And that’s something there’s always time for.

Dan: What’s your role on the Launchpad team?

Laura: I’m the Launchpad Support Specialist, so my job is pretty varied each day. Launchpad is rather larger than I first ever thought or had experience in using but it’s great to see so many people use it on a daily basis.

My role is to help people via email or IRC with their queries or point them in the right direction of where they can get more information or submit a bug or help them achieve something. I also look after Launchpad bugs and questions each day and it’s fascinating to see the varying questions we get on there so it’s a great way to learn and also see what interesting projects are on Launchpad and the communities that use it.

Dan: You’ve been working on Launchpad as a community member for a while though yeah?

I’ve been using it in the Ubuntu community in the past for blueprints, reporting bugs and and tracking issues and the odd time if I can help out in translations.

Dan: What’s been the biggest challenge in your new role so far?

Laura: Bazaar and PPAs both of which are bizarre to me at present, but the folks in the Launchpad and Bazaar teams have been really helpful to me and really made working with them easy.

Dan: Where do you work, and what can you see from your window?

Laura: I live in London, and work from home four days a week so when I look out the window I see the reflection of the London Eye. The other day a week I head into Canonical HQ.

Dan: If time/money was not an issue, what would you change about Launchpad?

Laura: Oh I’d love to make Launchpad translatable as I do know many people who love to get more involved, having it translated would help here. I’d also love to get more of the developer community involved in Launchpad, and where Launchpad isn’t doing what they’d like get them to submit patches and get them more involved in the process. It’s open source after all 🙂

Dan: How did you first start to get involved in the open source community?

Laura:I got involved when I was in college where I was roped into joining our computer society Skynet. Soon I became treasurer and event organiser and then eventually president of the society and got involved running open source conferences. Never looked back since!

We are reimagining the nature of privacy in Launchpad. The goal of the disclosure feature is to introduce true private projects, and we are reconciling the contradictory implementations of privacy in bugs and branches.

We are adding a new kind of privacy called “Proprietary” which will work differently than the current forms of privacy.

The information in proprietary data is not shared between projects. The conversations, client, customer, partner, company, and organisation data are held in confidence. proprietary information is unlikely to every be made public.

Many projects currently have private bugs and branches because they contain proprietary information. We expect to change these bugs from generic private to proprietary. We know that private bugs and branches that belong to projects that have only a proprietary license are intended to be proprietary. We will not change bugs that are public, described as security, or are shared with another project.

This point is a subtle change from what I have spoken and written previously. We are not changing the current forms of privacy. We do not assume that all private things are proprietary. We are adding a new kind of privacy that cannot be shared with other projects to ensure the information is not disclosed.

Launchpad currently permits projects to have default private bugs and branches. These features exist for proprietary projects. We will change the APIs to clarify this. eg:

    project.private_bugs = True  => project.default_proprietary_bugs = True
    project.setBranchVisibilityTeamPolicy(FORBIDDEN) => project.default_proprietary_branches = True

Projects with commercial subscriptions will get the “proprietary” classification. Project contributors will be able to classify their bugs and branches as proprietary. The maintainers will be able to enable default proprietary bugs and branches.

Next part: Launchpad will use policies instead of roles to govern who has access to a kind of privacy.

We are reimagining the nature of privacy in Launchpad. The goal of the disclosure feature is to introduce true private projects, and we are reconciling the contradictory implementations of privacy in bugs and branches.

We must change the UI to accommodate the a kind of privacy, and we must change some existing terms because to avoid confusion.

We currently have two checkboxes, Private and Security that create 4 combined states:

• Public
• Public Security
• Private Security
• Private something else

Most private bugs in Launchpad are private because they contain user data. You might think at first that something that is just private is proprietary. This is not the so. Ubuntu took advantage of defects in Launchpad’s conflation of subscription and access to address a kind of privacy we did not plan for. Most private bugs in Launchpad are owned by Ubuntu. They were created by the apport bug reporting process and may contain personal user data. These bugs cannot be made public until they are redacted or purged of user data. We reviewed a sample of private bugs that belong to public projects and discovered more than 90% were made private because they contained user data. Since project contributors cannot hide or edit bug comments, they chose to make the bug private to protect the user. Well done. Launchpad needs to clarify when something contains user data so that everyone knows that it cannot
be made public without removing the personal information.

Public and private security bugs represent two states in a workflow. The goal of every security bug is to be resolved, then made public so that users are informed. People who work on these issues do not use “public” and “private”, they use “unembargoed” and “embargoed”.

Also, when I view something that is private, Launchpad needs to tell me why. The red privacy banner shown on Launchpad pages must tell me why something is private. Is it because the page contains user data, proprietary information, or an embargoed security issue? This informs me if the thing could become public.

When I want to change somethings visibility, I expect Launchpad to show me a choice that clearly states my options. Launchpad’s pickers currently shows me a term without an explanation, yet Launchpad’s code does contain the term’s definition. Instead of making me search help.launchpad.net (in vain), the picker must inform me. Given the risks of disclosing personal user data or proprietary information, I think an informative picker is essential. I expect to see something like this when I open the visibility picker for a bug:

Branches require a similar, if not identical way of describing their kind of information. I am not certain branches contain user data, but if one did, it would be clear that the branch should not be visible to everyone and should not be merged until the user data is removed.

Next post: We are adding a new kind of privacy called “Proprietary” which will work differently than the current forms of privacy.

Back in September, we announced our first fastdowntime deployment. That was a new way to do deployment involving DB changes. This meannt less downtime for you the user, but we were also hoping that it would speed up our development by allowing us to deliver changes more often.

How can we evaluate if we sped up development using this change? The most important metric we look at when making this evaluation is cycle time. That’s the time it takes to go from starting to make a change to having this change live in production.  So before fastdowntime, our cycle time was about 10 days, and it is now about 9 days. So along the introduction of this new deployment process, we cut 1 day off the average, or a 10% improvement. That’s not bad.

But comparing the cumulative frequency distribution of the cycle time with the old process and the new will give us a better idea of the improvement.

On this chart, the gap between the orange (fastdowntime deployment) and blue (original process) lines shows the improvement to us.  We can see that more changes were completed sooner. For example, under the old process about 60% of the changes were completed in less than 9 days whereas about 70% were completed under the same time in the new process. It’s interesting to note that for changes that took less than 4 days to complete or that took more than 3 weeks to complete, there is no practical difference between the two distributions. We can explain that by the fact that things that were fast before are still fast, and things that takes more than 3 weeks would usually have also encountered a deployment point in the past.

That’s looking at the big picture. Looking at the overall cycle time is what gives us confidence that the process as a whole was improved. For example, the gain in deployment could have been lost by increased development time. But the closer picture is more telling.

The cycle time charted in this case is from the time a change is ready to be deployed until it’s actually live. It basically excludes the time to code, review, merge and test the changes. In this case, we can see that 95% of the changes had to wait less than 9 days to go live under the new process whereas it would take 19 days previously to get the same ratio. So an
improvement of 10 days! That’s way more nice.

Our next step on improving our cycle time is to parallelize our test suite. This is another major bottleneck in our process. In the best case, it usually takes about half a day between the time a developer submits their branch for merging until it is ready for QA on qastaging. The time in between is passed waiting and  running the test suite. It takes about 6 hours to our buildbot to validate a set of revisions. We have a project underway to run the tests in parallel. We hope to reduce the test suite time to under an hour with it. This means that it now would be possible for a developer to merge and QA a change on the same day! With this we expect to shave another day maybe two from the global cycle time.

Unfortunately, there are no easy silver bullets to make a dent in the time it takes to code a change. The only way to be faster there would be to make the Launchpad code base simpler. That’s also under way with the services oriented architecture project. But that will take some time to complete.

Photo by Martin Heigan. Licence: CC BY NC ND 2.0.

Recently the Launchpad Red Squad and Product Team started working on a new cloud project. As part of that project we’ll be using Juju, a tool that helps you easily deploy services on the cloud.

As an opportunity to learn more about how Juju works, I wrote a charm to deploy oops-tools, an open source Django application that helps visualize and aggregate error reports from Launchpad, on Amazon’s EC2 and Canonical’s private Openstack cloud.

You might be asking, what’s a charm? Charms are basically instructions on how to deploy services, that can be shared and re-used.

Assuming you already have a bootstrapped juju environment, deploying oops-tools using this charm is as easy as:

$ juju deploy --repository=. local:oops-tools
$ juju deploy --repository=. local:postgresql
$ juju add-relation postgresql:db oops-tools
$ juju expose oops-tools


That’s it! With just a few commands, I have an instance of oops-tools up and running in a few minutes.

Under the hood, the oops-tools charm:

• starts two Ubuntu instances in the chosen cloud provider, one for the webserver and another for the database server
• downloads the latest trunk version of oops-tools and its dependencies from Launchpad
• configures oops-tools to run under Apache’s mod_wsgi
• configures oops-tools to use the database server

There’s still work to do, like add support for RabbitMQ (oops-tools uses rabbit to provide real-time error reports), but this initial iteration proved useful to learn about Juju and how to write a charm. As it is, it can be used by developers who want to hack on oops-tools and can be easily changed to deploy oops-tools in a production environment.

If you’d like to give it a try, you can get the charm here: https://code.launchpad.net/~charmers/charms/oneiric/oops-tools/trunk

Enjoy!

We’re making changes to the way that bug heat is calculated and displayed in Launchpad.  From 6^th February, bug heat will no longer age/degrade, and the active flame counter symbol will be replaced by a number, next to a static flame.  Here’s why.

Bug heat ageing is the cause of a wide range of timeouts in Launchpad. Every bug’s heat has to be recalculated and updated every day using a series of complex calculations, and when there are around 1 million bugs reports to track, that’s a lot of pressure on the system, consuming a significant chunk of resources.  Turning off bug aging is the simplest way to solve this issue.

 Display

The flame counter symbol, although adding some visual flair (and flare), also needs to update every time the bug age recalculations are made.  The continual stream of updates to the bug rows also results in poor search index performance.

We’ll still have a flame symbol, however it’ll be static, with the bug heat number next to it. Although not as visually dynamic, it’ll be easier to work out bug heat scores more exactly, at a glance.

Although I’m sure some of us will miss this little Launchpad feature, less timeouts is good news for everyone.

Custom Bug Listings

Over the past few months the Launchpad Orange Squad has been working to make it easier to get the information that matters to you from bug listings.

A lot of you have said in the past that you’d like to be able to filter bugs in a way that works best for you. Hopefully this new feature, with its customisable functionality should help with this goal, filling your screen with more of what you want to see.

Features

You can now sort bugs by criteria such as name, importance, status and age. You can switch on the criteria that you use most and turn off criteria that you don’t use. So if you always like to see bug age, but aren’t interested in heat, you can switch on age and switch off heat, and so on.

Display

We’ve also redesigned how bug listings are displayed – fitting more information into each bug listing, and adding sort options such as bug age, assignee, reporter, and tags.

You can put your results into ascending or descending order without having to reload the page, and you’ll be able to save your preferred layout, so your settings will be saved for the next time you need to look over your bugs.

User research

This was my first main project since joining the Launchpad team back in November as the new Usability & Communications Specialist. User research has played an important part in how we’ve defined the feature and the decisions the team has made to improve the display, wording and functionality.

A number of you took part in one to one interviews, at group sessions at UDS-P and by taking part in an online survey. Thanks to everyone involved – what you told us has really helped to make this feature a more user-friendly experience. Some of our user research results (link) are already available online, with more being added soon. We’ll be carrying out some further tests in the weeks ahead, so please get in touch if you’d like to get involved.

Bugs

Every new feature has teething problems, and custom bug listings is no different. We still have a number of bugs that need tweaking, so please bear with us, and file any bugs if you spot anything that’s still out there.