Archive for the ‘General’ Category

Writing code for Launchpad

Wednesday, July 29th, 2009

William Grant was the first person outside Canonical to submit a patch to the newly open sourced Launchpad. I asked him a few questions to see how he got on.

Matthew: What did your patch change?

William: Launchpad users may elect to receive the bug’s description and status in the footer of each bug change notification email. This helps keep track of context when dealing with lots of different bugs.

Previously, bug emails that got to you through a team wouldn’t respect the setting — email coming through some teams would have the description, but for other teams it would be omitted. My fix ensures that the user’s selection is respected even for team emails.

Matthew: Why did you take the time to contribute to the Launchpad codebase?

William: I use Launchpad Bugs a lot, and get a lot of bug mail. This is an issue that has been irritating me for some time, and the fix seemed simple enough, so I decided it might be a good first task.

Matthew: How was it finding your way around the source tree?

William: It was easier than I expected. Most of the Bugs code now conveniently lives in lp.bugs, so I wasn’t overwhelmed with having to deal with the whole tree at first. The extensive test suite (particularly some good doctests) pointed me in the right direction, and developers and grep showed me the rest.

Matthew: How easy/hard was it to make your changes and stick to the Launchpad coding guidelines?

William: I didn’t get it quite right at first, but once pointed at the guidelines, it was simple to make my code compliant. It ended up looking a lot cleaner.

Matthew: What about the review process?

William: Remarkably easy. I found a reviewer (Graham Binns) on IRC, and submitted a merge proposal for my branch. A couple of hours and a few test changes later, my branch was approved. Graham landed it as soon as PQM reopened.

Matthew: What advice do you have for other people considering writing a patch for Launchpad?

William: Do it — it’s great to be able to fix those issues that have been bugging you for ages, and it’s not as hard as it looks. The test suite is a great guide through much of the codebase, and there are always developers on IRC to point you in the right direction if you get lost. The reviews are quick, developers are friendly, code style guidelines are clear, releases are frequent… all up it’s a very pleasant experience.

Matthew: Thanks William!

More power to the release manager

Tuesday, July 28th, 2009

Last month we made a small change to the series page as a commitment to making a distinction between the driver for a project and the driver of a series. The drivers of a project have the power to make the decision of what features go into a release. But the driver of a series is special. Often a select number of individuals are delegated the awesome responsibility to define the intent of a series, manage all the milestones necessary to meet the goals, and to create the release. Series drivers are release managers.

When developers talk about the trusted persons who are making the release happen, they use the term release manager. While Launchpad recognised the role, it required the person also be a project owner, which is not always suitable for large projects. Release managers do not need the power to edit the project information, they need the power to edit the series information, create milestones, and release them. That is the power they now have.

A project owner can set a user, or a team as the series release manager from the series page. Project drivers also have the power to create a series; they have the power to start planning to be make themselves the release manger of the series they create.

What’s next

Distribution have release managers too, they have the same responsibilities as project release managers, but giving them power is a bit tricky. For distributions other than Ubuntu, release managers will be able to create series and milestones, and edit their details just as a project release manager.

Ubuntu must be handled as an exception; the gift of Soyuz and Translations is also a curse. Many special tasks must be performed before an Ubuntu series can be register a series in Launchpad. The release managers for an Ubuntu series will have more power to edit the series details and milestones.

Answer contacts can assign questions

Thursday, July 23rd, 2009

Launchpad has supported assigning questions to users for several years, but the privilege was limited to project owners. This meant the feature was rarely used. Since the feature was also not visible, answer contacts often requested that we develop the feature. Question listing now include the assignee column. Answer contacts can assign a question to a user via the edit page. The assigned user will receive a notification about the assigned question. An assigned question will never expire; the assignee is obligated to answer the question.

The launchpad team had considered removing this feature two year ago because it was not popular. There were a few users who explained the need to assign questions to knowledgeable or privileged answer contacts. Simply put, the problem was not that assignment was unwanted, but that it could not be used by the people who needed the feature. The Launchpad team did not really understand how users were trying to use Launchpad until we decided to take turns answering every question asked to the launchpad project. We soon understood the need to assign questions to users. There are many questions that can only be answered by one or two people. The assignment must be visible to everyone, otherwise you would spend an hour reviewing open questions that were already assigned to someone. The cruelest part of assignment that that the assignee was never told that he has a task to complete.

This situation was especially frustrating for me because Answers is the application I started working on when I joined the Launchpad team. I knew that I could fix the issues in a few hours of work. Answers however, is not an application we are developing at the moment, I could not work on it during work hours. So I decided to fix the assignee feature on a Saturday. I could do this because the Launchpad on-call reviewer cannot easily say no to a merge request for a branch. I was also pretty certain my branch would be accepted because the reviewer is also answer contact who has experienced the assignee problems too.

The on-call reviewer rule is in place to ensure every patch is reviewed and given an opportunity to merge. This rule applies to everyone. Some Launchpad users have submitted patches for Launchpad pages and scripts and we have reviewed and merged them. Launchpad is now open source. You too can submit a branch knowing that someone must review it. Many of the Launchpad team members hack on Launchpad on our own time because we love Launchpad. Yet we still cannot fix every bug, or implement every feature. There are a lot of bugs that can be solved in a few hours work. If you want help to close a issue that you care about, we can help. Again the on-call reviewer is obligated try to advise on implementation issues. Also, Launchpad developers prefer to have pre-implementation plans to ensure that when someone decides to start a branch, it can be completed in less than two days and will be accepted by a reviewer. Someone on the #launchpad-dev channel on freenode can help.

Focusing on the Launchpad UI

Wednesday, July 22nd, 2009

Now that we’ve released Launchpad’s source code, our next couple of months of work are going to be mostly focused on our page layouts.
Launchpad has been around for quite a few years now, and tight release schedules packed with ever-changing features have had the side effect of us ending up with a lot of pages with different layouts.
In the next 2 months, we plan to fix that, and make sure every single page in Launchpad (452 templates!) has our new “3.0 look n’ feel”.
What does that look like, you may ask yourself?

Well, we’re still working on it, as we’re going to change the UI for the navigation (as well as tweak it’s functionality a bit, more on navigation in a future post). We do have rough draft which we’re starting to work towards, figure out what works and what doesn’t with real data, things we didn’t think about, etc.

The first major page we’re converting is the project overview page, currently being worked on by the world famous Curtis Hovey, and the initial draft should look similar to this:

It’s important to note we’re still working on the UI, so the image above is our starting point rather than the end product.

Since it will take some time to make all the changes, we’re most likely not going to make a Launchpad release in August, and jump straight to September. Roll-outs to our edge server will continue to happen daily, and we’ll need your feedback on the changes more than ever. If you’re interested in helping us, just join the beta testers team.

Exciting times!  🙂

Launchpad is now open source.

Tuesday, July 21st, 2009

This is a post I’ve been looking forward to for a long time:

Launchpad is now open source!

We released it today under the GNU Affero General Public license, version 3. Note that although we had previously announced that we’d be holding back two components (codehosting and soyuz), we changed our minds: they are included — all the code is open.

Big congratulations (and thanks) to the Canonical Launchpad team, who worked overtime to make this happen sooner rather than later, and to Mark Shuttleworth, whose decision it was to open source Launchpad in the first place.

Rather than repeat the various release announcements, I’ll just point to them:

The Canonical Launchpad developers will be on IRC in channel #launchpad-dev on irc.freenode.net. That’s the place to go for real time development discussion and questions. For usage issues, #launchpad is still the place, as before.

The mailing list is launchpad-dev {AT} lists.launchpad.net, which you can subscribe to by joining the ~launchpad-dev team. Again, that’s the development mailing list; user questions should still go to launchpad-users {AT} lists.launchpad.net.

Please bear with us as we learn how to be an open source team. Many of the Launchpad developers have open source experience already, of course, but as a team we’ve been working on Launchpad in-house for some years. This is a big change. We’ve been looking forward to it, though, and are ready and eager.

That’s all. Happy hacking :-).

-Karl Fogel

Launchpad’s YouTube channel

Wednesday, May 27th, 2009

I’ve set up a Launchpad channel on YouTube. Along with other members of the Launchpad team, I’ll be posting tutorials, demos of new features and more.

If you have a request for a video you’d like to see, file a bug in the Launchpad Documentation project! Or make it yourself and get in touch.

To kick things off, there’s a video showing the new in-line tag editing on bugs pages.

Update: I neglected to mention that all our videos are also online in Ogg Theora format.

Launchpad’s web service code released as standalone libraries

Wednesday, April 29th, 2009

The last time I posted to this blog I was describing a sprint in Montreal, where we were working on refactoring the web service code and moving it from Launchpad to a standalone library. Several weeks of work later, we’re done. You can now create your own web service using the same techniques we use in Launchpad.

The library is lazr.restful, and there’s a companion client library called lazr.restfulclient. They’re also on PyPI (restful, restfulclient). The place to discuss these libraries is the lazr-users mailing list.

I’ll show you lazr.restful first. If you just want to use the package, and you have easy_install, try “easy_install lazr.restful“. You can also download the tarball and use “python setup.py install“ on it. Or you can get check out the source code:

$ bzr branch lp:lazr.restful mybranch
$ cd mybranch
$ python bootstrap
$ python bin/buildout
$ python bin/test

The test command starts up the web service defined in src/lazr/restful/example and puts it through its paces with simulated HTTP requests. The example web service shows how to use Python decorators to turn Zope interfaces into a web service with the same features as Launchpad’s web service. Just as an example, here’s what a normal Zope interface might look like:

class IPerson(Interface):
    first_name = TextLine(title=u"First name", required=True)
    last_name = TextLine(title=u"Last name", required=True)

Here’s what it would look like with lazr.restful annotations:

class IPerson(Interface):
    export_as_webservice_entry(plural_name='people')
    first_name = exported(TextLine(title=u"First name", required=True))
    last_name = exported(TextLine(title=u"Last name", required=True))

lazr.restful will inspect that interface and know that you’re defining an ‘entry’ type resource, with each entry containing two bits of data. It’ll publish a WADL file describing that interface. Clients will be able to make GET, PUT, and PATCH requests to the ‘person’ resources to manipulate your application data.

lazr.restful provides annotations for every web service feature you see in Launchpad: entries, collections, hosted files, named operations, and so on. It even has some features that aren’t used by Launchpad yet, like the ability to make entries respond to DELETE requests.

lazr.restfulclient is the client side. We took almost all the code out of launchpadlib and made a generic client that will run against any lazr.restful service.

The same bootstrapbuildouttest cycle that worked for lazr.restful also works for lazr.restfulclient. The test command will start up the web service from lazr.restful, and manipulate it with Pythonic launchpadlib-like commands instead of fake HTTP requests. Given a ‘person’ resource like the one defined in the example above, you could run this code in lazr.restfulclient.

person_resource.first_name = "new first name"
person_resource.lp_save()

Before long we’ll be removing almost all the code from launchpadlib leaving only the OAuth handling and some Launchpad-specific information. We’re also working to make it easier to use lazr.restful with standard WSGI applications, so you don’t have to know as much Zope.

Karma — where did mine go?

Thursday, April 16th, 2009

A couple of days ago I posted that Launchpad had been assigning too much karma for certain types of translation work.

Basically, if you imported translation files — PO and POT files — into Launchpad from an Ubuntu package or a Bazaar branch then you got vastly more karma than if you simply translated directly through Launchpad’s web interface or uploaded PO files as ‘user uploads’.

Of course, that was unfair so we decided to adjust the inflated karma that Launchpad had awarded. The way Launchpad calculates karma is pretty complex, so the manual reduction we made to karma awarded for those translation imports has also affected karma given for other types of work — such as bugs, code, or blueprints — even for people who never did anything in Translations.

So, what’s going on?

Launchpad treats all its applications as equal in terms of the karma they can give out. Let’s call that the “total karma pool per application”. This means that if the total sum of all karma assigned in Blueprints to all users is X, the total karma assigned in Bugs or Translations will also be X.

However, X is determined as the largest of the actual sums of karma values all applications have. So, if Translations is handing out the most karma in Launchpad, all karma values for other applications will be scaled up to match.

There’s more in our explanation of karma calculation on the help pages.

So, what does this mean? After we fixed bug #286359 to send import notifications to Ubuntu packagers, we also started giving out translation import karma for them. An unproportionally large amount of karma was given out compared to other stuff in Translations, and that caused the total pool of Translations karma to grow, so X started growing for all applications. People noticed that their Translations karma was getting bigger and bigger, and reported bugs such as bug #337645. We also got a few comments on IRC. However, nobody noticed — or at least, they didn’t complain 🙂 — that their other karma was growing along with the growing total karma pool for Translations. Suddenly (or not that suddenly, the scaling factor increased over time), people had about 2–3 times more karma then they were supposed to.

Finally, 6 weeks after we introduced the problem, we’ve normalized translations karma so uploaders do not have an unfair advantage over actual translators. That meant that the total pool of Translations karma went down, and with it, X went down and everybody’s karma across the system went down, regardless of the application. Note that everybody’s non-Translations karma got proportionally scaled down, so relative karma scores are still the same (i.e. if you had more karma than your friend Terry a week ago, you’ve still got more karma than Terry today, unless Terry has subsequently gone on a Launchpad work rampage).

What can we do to lessen the consequences of any karma normalization in the future? We’ve got to be careful when introducing or modifying existing actions. We’ve got to normalize karma points before we work on a feature which actively gives them out. And we’ve got to better communicate changes like this.

So, what actually happened is: karma that was never meant to exist disappeared. Many parallels to the economic situation are easy to envision. Launchpad was The Karma Bank. 🙂

Inside the Launchpad AJAX Sprint: A Week with Widgets and YUI 3

Friday, March 13th, 2009

Recently ten people from Launchpad and other parts of Canonical came together in Berlin to hack on Launchpad’s new YUI 3 JavaScript interface.  The sprint was tremendously successful, producing four fully functioning YUI 3 Widgets, complete with test suites and live demo pages.  This post offers a look inside the event, and some thoughts about what made it so successful.

The sprint brought together one member from each of Launchpad’s teams, as well as Martin Albisetti, for his design insight, Francis Lacoste, for his Zope3 and Launchpad architecture knowledge, and Stuart Langridge, who, to our surprise and fortune, had much jQuery, DOM, and CSS experience.  This wasn’t the team’s first encounter with the YUI 3 JavaScript library (we had spent two weeks in the fall reviewing JavaScript, YUI 3, and client-side development), but this would be our first time taking YUI 3, and Launchpad’s JavaScript support, up to full speed.

Martin kicked off the morning of day one with a walk through some video proof-of-concepts that we would bring to life.  After some wide-ranging discussion and dealing over implementation details, we spent the afternoon looking at the technical side of YUI 3 Widgets.  At the end of the day eight of the sprinters split into pairs, each pair picking a widget to work on.

Day two saw the four teams plan their attack, then dive into the code.  The sprint took on the shape of a massive parallel experiment, with each team choosing a different development approach: one wrote tests first, diving into Test Driven Development; another executed a two-pronged assault, with one person writing the widget while the other worked on Lauchpad integration.  A third team went for functionality first, aiming for a working wireframe before styling the page, or writing tests.  In the end none of the approaches was a clear winner, but it’s worth noting that quickly getting a working prototype up on the projector gave everyone a huge moral boost, and looks to be worth the cost of that team writing their tests last.

One thing that helped every team get up and running quickly with the YUI 3 Widget framework was having an already-running, fully integrated widget in Launchpad: the inline-text editor for Bug and Project titles.  Building the inline editor had cleared paths for coding styles, directory structure, test harnesses, and all of the Zope3 integration code.  A month or two of work had already gone into supporting JavaScript and JavaScript testing in Launchpad — now that work paid off, giving all eight people a very fast ramp-up to code that could land in mainline.  Along with the very well-written YUI 3 docs, everyone was writing code by noon.

Day three saw more code, tests, and our first working widget.  We just needed the fancy frame to have a pixel-perfect movie match.  The teams kept the momentum high all day while Martin, Francis and I criss-crossed the room, lending design decisions, on the spot integration plans, and YUI advice and council.  Stuart Langridge also lent his scripting and CSS experience to the effort.  For any sprint, when the team is blocked on an implementation question, it helps to have someone with the authority to make final, on the spot decisions.  We discovered that, when doing top-to-bottom JavaScript widget implementation, it helps to have a whole team of them.

By noon of day four most of the remaining widgets were functional, if not fully styled.  For me, much of the day was spent pair programming, tackling a subclass of the YUI Overlay widget.  We had trouble getting it to accept the stylish drop-shadowed frame given to us by our designer, and we used YUI 3’s JavaScript unit testing library, yuitest, to work around it.  Our test suite allowed us to spot the Firefox DOM bugs our layout triggered, and assured us that our hackish workarounds unbroke the DOM tree.

Most of the sprinters were qualified code reviewers, so day four’s afternoon was spent in a team review of the fully functional fancy Overlay widget: two authors presenting with Emacs, one scribe, and seven people contributing questions and comments.  The meeting helped immensely in building a culture around JavaScript code in Launchpad, and left us with a draft JavaScript style guide, and a coding standard we could take to the wider team.  I highly recommend such a review, along with the surrounding discussion, to anyone bringing a new language into a programming group.

Everyone had great momentum going into day five.  All four widgets were fully functional, with only a few rough spots remaining.  One nasty Friday surprise was that the original inline editor didn’t work in Konqueror 4.1 — in fact, it made editing Launchpad Bug titles impossible.  This is where our adoption of YUI’s Graded Browser Support and Graceful Degredation really shone.  We quickly made the decision to give KHTML browsers a C-Grade experience: for them, the site behaves as if it were not enhanced with JavaScript: you can still edit a bug title, but you are taken to a new page to do so.  The decision was fast and simple, and with a clear C-Grade verdict of “No JavaScript”, no unnecessary time was spent searching for other fun issues the browser may have presented.

Day five’s afternoon was spent with everyone in a Review Jam.  We mixed up the programming pairs, with one person from each team reviewing someone else’s code, using the guidelines we had drafted the previous day.  Martin did some final user interface design reviews, and everyone left that Friday with a list of changes before final submission to mainline.

The sprint was very successful, one of our best yet, producing four YUI 3 widgets in only three and-a-half days of coding, and we plan to have another sprint like this in the future.  Here’s some of what made this week-long coding sprint work for us:

  • You need to have a good mix of skills at the sprint, including experienced people from each discipline.  And that experience needs to be available to unblock sprinters as quickly as possible.  At this sprint, two of the ten people were dedicated to roaming the room, and at least two more had deep experience with the problems we were tackling.
  • Someone at the sprint needs to be a project stakeholder, with desicion-making authority.  They’ll be called upon as the sprint progresses.  Martin filled this role for us, having final authority on what the Launchpad user interface would look like.
  • If sprinters are building something new, but understood, come prepared with a template, plan, or example that everyone can follow.  Use that template to spend less time at the sprint understanding known problems, and more time tackling new ones.  The completely integrated and running inline editor widget had already covered all aspects of widget development for us.
  • Sprinters do very well with self-selected goals and self-made plans.  We didn’t hold any of the teams to any particular approach (aside from some good-hearted chiding about unwritten tests), which let everyone adapt their style to their own problem.  Two of the teams even picked goals that were more ambitious than the original scope.

A final word on YUI 3: the library performed very well, especially given its pre-release status.  Experienced scripters may find working with YUI 3 a little odd at first, because you are not working with the raw DOM: you have to get used to using YUI Node instances, getters, and setters, instead of DOM object properties (most of the DOM methods are passed right through).  However, once past the initial bump, you can quickly see the advantages of having features like object property getters and setters, a robust and advanced object construction system, and cross-browser APIs.  Stuart also pointed out one big, open question about YUI itself: the only major libraries to tackle JavaScript widget development “in the large” are YUI and Dojo.  We’ll see if YUI’s large approach, jQuery’s small tools approach, or YUI 3’s mix of both is best.

New privacy features for commercial subscribers

Wednesday, February 25th, 2009

Launchpad is set up to make it easy for groups to come together spontaneously around tasks big or small. Take this bug as an example: it affects three projects that share the same code. Launchpad brings them together to find a fix.

But there are times when collaboration needs to be done privately. It could be because of security concerns, or confidentiality obligations to a client, or to coordinate the timing of events for marketing purposes, etc.

Launchpad already enables individual bugs to be marked as private but, until now, that’s been all. However, we’re running an interesting beta program at the moment that allows projects to default to both private bugs and private branches. As this will be of most interest to projects who aren’t ready to share their code, we’ll offer it by commercial subscription.

We’ve already been offering basic commercial subscriptions for a few months now. While Launchpad is always free of charge for free software projects, it also hosts some projects with non-free licenses, and charges them a fee to offset the costs of running the site. Until now, these commercial subscriptions have just meant access to Launchpad with no extra features. But following our beta of private branches and private bugs, we hope to offer a wider range of features to commercial subscribers, as they are likely to have different privacy needs from free software projects. If you’re a commercial subscriber and want to take part in the beta, or if you’re considering a commercial subscription, please contact us.

Of course, commercial subscriptions don’t take anything away from the free software projects that use Launchpad. They remain the focus of our development work, particularly as we move to open sourcing Launchpad itself.