Parallelising the Unparallelisable

Benji York — Tue, 27 Mar 2012 12:15:58 +0000

Launchpad has a lot of tests, almost 20,000. There are tests that make sure the internals work as expected, that verify the Javascript works in web browsers, and everything in between. In a perfect world those tests would only take seconds to run. In this world they take hours; six hours on our current continuous integration machines, for instance.

These long-running tests severely impact the time it takes to develop and deploy changes to Launchpad. We would like to improve the situation.

Given that the test cases are theoretically independent of one another, the obvious thing to do is to run the tests in parallel on a multi-core machine. Unfortunately many of the tests interact with the environment (databases, memcached, temporary directories, etc.) and conflict if run simultaneously.

Enter LXC

What we need is a way to isolate the test processes from each another. Virtual machines would allow us to do that, but the overhead and heavy-weight setup makes them unappealing. That’s where LXC (Linux Containers) comes in handy. LXC

allows the easy creation of “containers” that are isolated from the “host” machine without the performance overhead of VMs.

For example, to create a new container use lxc-create:

lxc-create -n test -t ubuntu

The container can then be started:

lxc-start -n test -d

And we can connect to it via SSH (using the default username and password shown during creation, if applicable):

ssh ubuntu@test

There are many options for customising the containers, including mounting a portion of the host’s file system in the container so sharing files between the two is easy.

Getting Ephemeral

All this is very nice for running isolated, parallel test runs but setting up and managing eight or more containers (one per core) is
off-putting, so we have used (and improved) a new LXC feature, “ephemeral” containers (created with lxc-start-ephemeral).

Ephemeral containers are “clones” of a base container and can have a temporary file system that reflects the contents of the base container but any writes are stored in-memory and are not written to disk. This allows us to install Launchpad on a single base container and then spawn many ephemeral containers, each with their own list of tests to run.

The ephemeral containers can then write to their local file systems without interfering with the others running simultaneously. The
containers may also benefit from faster IO because of the file system changes being stored in memory.

Results

We are still working out the kinks in our approach and wrestling with the occasional LXC bug as well as bugs in the Launchpad test suite itself. Even so we have already shortened a full test run on an eight-core EC2 instance down to 45 minutes; a substantial improvement over the current six hours.

(Image by Tolka Rova, Creative Commons license)

Benji on Launchpad Blog

Parallelising the Unparallelisable