launchpadblog

We’ve just upgraded Launchpad’s builder machines to Bazaar 2.4. Most importantly, this means that recipe builds of very large trees will work reliably, such as the daily builds of the Linaro ARM-optimized gcc. (This was bug 746822 in Launchpad).

We are going to do some further rollouts over the next week to improve supportability of recipe builds, support building non-native packages, handle muiltiarch package dependencies, improve the buildd deployment story etc.

bzr-svn, bzr-git and bzr-hg are plugins for bzr that make Subversion, Git and Mercurial branches first class citizens in the Bazaar world by allowing you to access them in the same way that you would access native Bazaar branches.

Bazaar has supported multiple file formats from its early days. Both its model and its implementation allow this:

• Revisions are not identified by the checksum of their layout on disk (as they are in systems like Git or Mercurial) but by a (pseudo-)random string. This means that copying data to a different file format does not affect the revision id.
• Repositories are accessed through a well defined interface. Other parts of the codebase are ignorant about the structure of the files on disk.

This has made it easy to introduce better and experimental repository formats without having to break old repositories or render them unusable for previous versions of Bazaar by forcing upgrades. Initially new formats were introduced at a very high pace (perhaps even a too high pace?), but fortunately this has slowed down nowadays: the last default format change before the 2.0 release was in 2007.

Having grown interested in Bazaar through Martin‘s talk at Linux.Conf.Au 2005 and his blog posts I started looking into Bazaar in 2005. Since Samba (the main FOSS project I work on) had just switched to Subversion, I was interested in ways to interact with Subversion using Bazaar, in particular so I could do off-line commits. On the Bazaar wiki Aaron had suggested implementing the well defined interface for repository formats for other version control systems (such as Subversion) as well. This sounded very neat, so I decided to see how far I could get and looked into learning Python and becoming more familiar with the Bazaar API.

Now, four years later, 700 bug reports and about 4400 revisions later, we have released bzr-svn 1.0. The models of Subversion and Bazaar have significant differences, and bzr-svn has to take care of mapping between the semantics of both. Perhaps the best example of this is the fact that a Subversion repository is basically a versioned filesystem; there may be some directories that are commonly used as containers for branches or tags, but there are a lot of exceptions to this convention. In Bazaar on the other hand, a branch is a primary object.

In 2006 Rob and Aaron created a simple plugin for accessing local Git repositories in 2006 called bzr-git. Originally it was based on “stgit”, a tool which (among other things) exposed a Python wrapper around the git executables. Following a switch from Samba to Git I took over in 2008 and changed bzr-git to use a new native implementation of Git in Python, based on a project by James. bzr-git now supports accessing remote repositories, working trees and merging changes back into Git.

At the moment I am working on the bzr-hg plugin, again based on an initial proof of concept by Rob. Last month ago the first version (0.1) was released, providing sufficient support for cloning local and remote Mercurial repositories and accessing working trees. There are still some problems to work out — memory usage is excessive, commit and push do not yet work — but there should be a stable plugin in a few months.

The Bazaar project released their version 2.0.0 this week. I spoke to Martin Pool, the project’s lead, about the release and Bazaar generally.

Matthew: Congratulations on the release of Bazaar 2.0.0! If you had to come up with a headline for this release, what would it be?

Martin: “Harder, better, stronger, faster” — we made our new 2a format the default and it’s considerably smaller and faster. Ian‘s recent benchmarks show repositories in this format are substantially smaller than for Mercurial, and roughly the same size as for Git. Of course results do vary but it does correlate, and determines how much data we have to transfer from local disk or across the network.

The other cool thing about this release is that it’s the start of a stable series of 2.0.1 releases, where we’ll be landing only bugfixes and (as much as we can manage it) no new bugs or features, no API compatibility breaks, and no format changes. We’ve heard from users that in some situations they find our monthly releases too much, so we’re now going to give them the choice of a more stable series, or to keep getting new features every month with the 2.1betas.

Matthew: In the release announcement you mention that the new repository format is “substantially smaller and faster for many operations”. What in particular should I expect to notice as a Bazaar user?

Martin: The thing I actually notice most, living on the other side of the world from the Canonical (London) data centre is that pushes and pulls from Launchpad are dramatically faster — for some operations the dominant factor is the time it takes to open an ssh connection.

Matthew: Is it easy to get the new repository format up and running for my existing branches?

Martin: Yes, basically you just need to run upgrade — but you might like to read the Upgrade Guide first.

That reminds me of something else that changed recently — Ian converted our documentation to use Sphinx, so we get nicer HTML and also native Windows help files.

Matthew: The Bazaar community has done some really interesting work on repository formats. Is this new format the culmination of that or is there more you want to do?

Martin: I think there is more we could possibly do: for instance Alexander Belchenko, a user with some machines on older Windows releases, has the mantra that “OS locks must die” – we should rely on a smaller filesystem feature set, so that Bazaar works better there. Robert Collins has been sketching out a ‘dirstate2’ format for managing the working tree, that can be still smaller and faster.

But personally before we do a new public format release, I think we need to take a hard look at the user experience of a format change, especially when you have multiple developers. It is not nearly as seamless as we’d like. I think 2a gives us a good checkpoint that will work well for a while.

Matthew: What are you most excited about in this release?

Martin: I’m just thrilled it came together for such a good release, that 2a is performing so well, and that it’s into Karmic as a foundation for future stable releases.

Matthew: What makes Bazaar stand out right now from other version control systems?

Martin: I think it’d be that the transition to it can be much easier. Bazaar can be used in either a totally distributed or a centralized way, or any point in between, and that works in very well for people used to working on CVS or Subversion, or teams where some contributors are less technical. We aim to keep the user interface simple and the documentation clear, and every important feature can now be reached through the Explorer gui or through the command line. Also, Bazaar can directly interact with projects in svn, git or hg through foreign-branch plugins, so people can gradually transition.

Matthew: What advice would you offer to a project considering switching to Bazaar from another VCS?

Martin: Don’t hesitate to talk to us on IRC or the lists about how you’re planning to use it or any questions about the transition.

Matthew: What’s your favourite Bazaar plugin?

Martin: Probably bzr-explorer and qbzr, they’re coming along very quickly as a complete graphical interface.

Matthew: What’s next for Bazaar?

Martin: For the next while we’re going to especially focus on helping Ubuntu switch to bzr-based distributed development.

Matthew: Thanks Martin!

As you might have heard already, Launchpad can now import code from Git repositories. You can then create Bazaar branches of those Git repos.

For example:

bzr branch lp:git

Thanks to Jelmer for bzr-git and Michael & Paul for tying it into our rock-solid import system.

Here on the Launchpad team we work pretty closely with the Bazaar community.

This week, Canonical‘s London HQ has been host to a Bazaar sprint. It’s been great to meet up with members of the Bazaar community from Australia, the USA, Argentina, France, the Netherlands, Hungary and the UK. I’ve been working with them on plans for a new Bazaar administrator’s guide.

I’ve also recorded a short chat with Martin Albisetti who is leading the new Bazaar IDE integration team. Martin talks about the technical and advocacy approaches they plan to take.

Listen to the interview with Martin.

Launchpad is kinda hard to describe. When I’m asked what it is, I normally use three or more of the words “open source free software support project Q&A code registration bugs management listing bazaar planning hosting”. Most people make comparisons to Sourceforge, Savannah, Berlios and Google code hosting, and while those are useful, it’s only a part of the picture. The other part, which is perhaps not as well understood, is that it’s also a public project registration service, similar to Freshmeat. Well, except for a twist.

Launchpad’s source code directory

In addition to providing a project registration service open to the public for free (with probably the best Google juice out there; this has caught some people off-guard before!), Launchpad takes this to a next step, and actually provides a unified interface for interacting with each project registered. The most obvious unified service that you can use today is our code directory, which I want to introduce here.

Now Launchpad provides some key features related to source code for free software projects:

• Project registration: you can register any free software project on Launchpad (and separately, of course, have it hosted here — if you own it and want to).
• Bazaar hosting: you can host Bazaar branches for any project, and you can fetch code using the bzr client.
• Code imports: we allow you to request code imports for any externally-hosted project that uses CVS or Subversion.

There’s a really cool thing that falls out of the combination of code imports, branch mirrors and Bazaar: you can use bzr and Launchpad to fetch any piece of software we have registered code for. And Bazaar even provides a little shortcut that makes it even easier to grab the default branch for any project: bzr branch lp:<projectname>.

An open source supermarket

The effect is that you can, right now, pull a /lot/ of software in less than 30 keystrokes, without knowing or caring about what its native version-control system is, or where it’s hosted. Want to grab the Python source code? Just do bzr branch lp:python. How about Storm? bzr branch lp:storm. In fact, there are thousands of native Bazaar branches and over one thousand imported branches. Here are just a few examples of major projects you can pull right now:

• Ruby on Rails:
  bzr branch lp:rails
• Blender:
  bzr branch lp:blender
• F-Spot:
  bzr branch lp:f-spot
• Grub:
  bzr branch lp:grub
• Twisted:
  bzr branch lp:twisted
• Bazaar itself:
  bzr branch lp:bzr
• GCC:
  bzr branch lp:gcc

Couldn’t find a branch listed for the project you want? We can sort this out for you, too. If it’s in CVS or Subversion elsewhere, you can just follow the instructions for setting up an import. If it’s a Bazaar branch, just register it and we’ll hook it up with the project’s mainline series record.

More on project branches

If you visit https://code.launchpad.net/ you’ll notice that it has an abbreviated project cloud, which lists all the projects with branches in Launchpad. The project’s name is rendered in different sizes and intensities according to how active the actual project is; the size of the name in the cloud is defined by the number of branches that the project has, and the intensity of the tag in the cloud is determined by how recent the last commit to any active branch is. And green indicates that there’s a default branch for the project, which means that the bzr branch lp:foo abbreviation works for it. There’s a also a page with the full code cloud.

So each of those projects has active source code branches that you can pull from Launchpad using Bazaar. For instance, to check the source code for Apport, you could click on its entry in the listing and getting there you could inspect the branches available and select one of them for pulling. For instance, if you chose Will Woods’ Fedora support branch you get instructions on how to pull it: bzr branch http://bazaar.launchpad.net/~wwoods/apport/fedora or even just
bzr branch lp:~wwoods/apport/fedora

Imports and today’s deliveries

Import requests are handled in a queue by Launchpad code ninjas; normally this it Michael Hudson’s responsibility but starting today I’ll be helping out too. Of the import requests I handled today, the following succeeded and are ready for grabbing in the great bzr-get-lp-colon fashion:

• bzr branch lp:gtk-qt-engine
• bzr branch lp:lshw
• bzr branch lp:tsclient
• bzr branch lp:dohickey
• bzr branch lp:meshlab
• bzr branch lp:rasmol

I’ll keep you posted on new imports as they come online. Meanwhile, go grab some branches and post some comments telling me what you think. If you have any problems or questions about our code hosting service, feel free to ask on the Launchpad code section in answers.launchpad.net.

The release of Bazaar 0.17, a couple of weeks ago, got me thinking.

Before I joined the Launchpad team, I knew that version control was important but it wasn’t something I thought about a great deal.

In previous jobs, I’d used Subversion to manage documentation and, although I could see the benefits, it often seemed more hassle than it was worth, particularly to my colleagues.

Then I started looking more closely at Bazaar. I saw that it didn’t need a central server, so getting set-up was pain-free and I could work anywhere. It also allowed me to just get on with organising my work the way I wanted to: if I decided that, after all, I didn’t need a directory called “drafts” but instead wanted two directories called “in-progress” and “awaiting-review”, then Bazaar was happy to accommodate me.

But as a non-developer, I rarely even take Bazaar out of second gear. I’d love to hear your stories of using Bazaar. What one thing about Bazaar really makes your life easier? What five, ten, twenty things about Bazaar get you excited? Post your comments here or mail me – feedback@launchpad.net.