Archive for the ‘General’ Category

Karma — where did mine go?

Thursday, April 16th, 2009

A couple of days ago I posted that Launchpad had been assigning too much karma for certain types of translation work.

Basically, if you imported translation files — PO and POT files — into Launchpad from an Ubuntu package or a Bazaar branch then you got vastly more karma than if you simply translated directly through Launchpad’s web interface or uploaded PO files as ‘user uploads’.

Of course, that was unfair so we decided to adjust the inflated karma that Launchpad had awarded. The way Launchpad calculates karma is pretty complex, so the manual reduction we made to karma awarded for those translation imports has also affected karma given for other types of work — such as bugs, code, or blueprints — even for people who never did anything in Translations.

So, what’s going on?

Launchpad treats all its applications as equal in terms of the karma they can give out. Let’s call that the “total karma pool per application”. This means that if the total sum of all karma assigned in Blueprints to all users is X, the total karma assigned in Bugs or Translations will also be X.

However, X is determined as the largest of the actual sums of karma values all applications have. So, if Translations is handing out the most karma in Launchpad, all karma values for other applications will be scaled up to match.

There’s more in our explanation of karma calculation on the help pages.

So, what does this mean? After we fixed bug #286359 to send import notifications to Ubuntu packagers, we also started giving out translation import karma for them. An unproportionally large amount of karma was given out compared to other stuff in Translations, and that caused the total pool of Translations karma to grow, so X started growing for all applications. People noticed that their Translations karma was getting bigger and bigger, and reported bugs such as bug #337645. We also got a few comments on IRC. However, nobody noticed — or at least, they didn’t complain 🙂 — that their other karma was growing along with the growing total karma pool for Translations. Suddenly (or not that suddenly, the scaling factor increased over time), people had about 2–3 times more karma then they were supposed to.

Finally, 6 weeks after we introduced the problem, we’ve normalized translations karma so uploaders do not have an unfair advantage over actual translators. That meant that the total pool of Translations karma went down, and with it, X went down and everybody’s karma across the system went down, regardless of the application. Note that everybody’s non-Translations karma got proportionally scaled down, so relative karma scores are still the same (i.e. if you had more karma than your friend Terry a week ago, you’ve still got more karma than Terry today, unless Terry has subsequently gone on a Launchpad work rampage).

What can we do to lessen the consequences of any karma normalization in the future? We’ve got to be careful when introducing or modifying existing actions. We’ve got to normalize karma points before we work on a feature which actively gives them out. And we’ve got to better communicate changes like this.

So, what actually happened is: karma that was never meant to exist disappeared. Many parallels to the economic situation are easy to envision. Launchpad was The Karma Bank. 🙂

Inside the Launchpad AJAX Sprint: A Week with Widgets and YUI 3

Friday, March 13th, 2009

Recently ten people from Launchpad and other parts of Canonical came together in Berlin to hack on Launchpad’s new YUI 3 JavaScript interface.  The sprint was tremendously successful, producing four fully functioning YUI 3 Widgets, complete with test suites and live demo pages.  This post offers a look inside the event, and some thoughts about what made it so successful.

The sprint brought together one member from each of Launchpad’s teams, as well as Martin Albisetti, for his design insight, Francis Lacoste, for his Zope3 and Launchpad architecture knowledge, and Stuart Langridge, who, to our surprise and fortune, had much jQuery, DOM, and CSS experience.  This wasn’t the team’s first encounter with the YUI 3 JavaScript library (we had spent two weeks in the fall reviewing JavaScript, YUI 3, and client-side development), but this would be our first time taking YUI 3, and Launchpad’s JavaScript support, up to full speed.

Martin kicked off the morning of day one with a walk through some video proof-of-concepts that we would bring to life.  After some wide-ranging discussion and dealing over implementation details, we spent the afternoon looking at the technical side of YUI 3 Widgets.  At the end of the day eight of the sprinters split into pairs, each pair picking a widget to work on.

Day two saw the four teams plan their attack, then dive into the code.  The sprint took on the shape of a massive parallel experiment, with each team choosing a different development approach: one wrote tests first, diving into Test Driven Development; another executed a two-pronged assault, with one person writing the widget while the other worked on Lauchpad integration.  A third team went for functionality first, aiming for a working wireframe before styling the page, or writing tests.  In the end none of the approaches was a clear winner, but it’s worth noting that quickly getting a working prototype up on the projector gave everyone a huge moral boost, and looks to be worth the cost of that team writing their tests last.

One thing that helped every team get up and running quickly with the YUI 3 Widget framework was having an already-running, fully integrated widget in Launchpad: the inline-text editor for Bug and Project titles.  Building the inline editor had cleared paths for coding styles, directory structure, test harnesses, and all of the Zope3 integration code.  A month or two of work had already gone into supporting JavaScript and JavaScript testing in Launchpad — now that work paid off, giving all eight people a very fast ramp-up to code that could land in mainline.  Along with the very well-written YUI 3 docs, everyone was writing code by noon.

Day three saw more code, tests, and our first working widget.  We just needed the fancy frame to have a pixel-perfect movie match.  The teams kept the momentum high all day while Martin, Francis and I criss-crossed the room, lending design decisions, on the spot integration plans, and YUI advice and council.  Stuart Langridge also lent his scripting and CSS experience to the effort.  For any sprint, when the team is blocked on an implementation question, it helps to have someone with the authority to make final, on the spot decisions.  We discovered that, when doing top-to-bottom JavaScript widget implementation, it helps to have a whole team of them.

By noon of day four most of the remaining widgets were functional, if not fully styled.  For me, much of the day was spent pair programming, tackling a subclass of the YUI Overlay widget.  We had trouble getting it to accept the stylish drop-shadowed frame given to us by our designer, and we used YUI 3’s JavaScript unit testing library, yuitest, to work around it.  Our test suite allowed us to spot the Firefox DOM bugs our layout triggered, and assured us that our hackish workarounds unbroke the DOM tree.

Most of the sprinters were qualified code reviewers, so day four’s afternoon was spent in a team review of the fully functional fancy Overlay widget: two authors presenting with Emacs, one scribe, and seven people contributing questions and comments.  The meeting helped immensely in building a culture around JavaScript code in Launchpad, and left us with a draft JavaScript style guide, and a coding standard we could take to the wider team.  I highly recommend such a review, along with the surrounding discussion, to anyone bringing a new language into a programming group.

Everyone had great momentum going into day five.  All four widgets were fully functional, with only a few rough spots remaining.  One nasty Friday surprise was that the original inline editor didn’t work in Konqueror 4.1 — in fact, it made editing Launchpad Bug titles impossible.  This is where our adoption of YUI’s Graded Browser Support and Graceful Degredation really shone.  We quickly made the decision to give KHTML browsers a C-Grade experience: for them, the site behaves as if it were not enhanced with JavaScript: you can still edit a bug title, but you are taken to a new page to do so.  The decision was fast and simple, and with a clear C-Grade verdict of “No JavaScript”, no unnecessary time was spent searching for other fun issues the browser may have presented.

Day five’s afternoon was spent with everyone in a Review Jam.  We mixed up the programming pairs, with one person from each team reviewing someone else’s code, using the guidelines we had drafted the previous day.  Martin did some final user interface design reviews, and everyone left that Friday with a list of changes before final submission to mainline.

The sprint was very successful, one of our best yet, producing four YUI 3 widgets in only three and-a-half days of coding, and we plan to have another sprint like this in the future.  Here’s some of what made this week-long coding sprint work for us:

  • You need to have a good mix of skills at the sprint, including experienced people from each discipline.  And that experience needs to be available to unblock sprinters as quickly as possible.  At this sprint, two of the ten people were dedicated to roaming the room, and at least two more had deep experience with the problems we were tackling.
  • Someone at the sprint needs to be a project stakeholder, with desicion-making authority.  They’ll be called upon as the sprint progresses.  Martin filled this role for us, having final authority on what the Launchpad user interface would look like.
  • If sprinters are building something new, but understood, come prepared with a template, plan, or example that everyone can follow.  Use that template to spend less time at the sprint understanding known problems, and more time tackling new ones.  The completely integrated and running inline editor widget had already covered all aspects of widget development for us.
  • Sprinters do very well with self-selected goals and self-made plans.  We didn’t hold any of the teams to any particular approach (aside from some good-hearted chiding about unwritten tests), which let everyone adapt their style to their own problem.  Two of the teams even picked goals that were more ambitious than the original scope.

A final word on YUI 3: the library performed very well, especially given its pre-release status.  Experienced scripters may find working with YUI 3 a little odd at first, because you are not working with the raw DOM: you have to get used to using YUI Node instances, getters, and setters, instead of DOM object properties (most of the DOM methods are passed right through).  However, once past the initial bump, you can quickly see the advantages of having features like object property getters and setters, a robust and advanced object construction system, and cross-browser APIs.  Stuart also pointed out one big, open question about YUI itself: the only major libraries to tackle JavaScript widget development “in the large” are YUI and Dojo.  We’ll see if YUI’s large approach, jQuery’s small tools approach, or YUI 3’s mix of both is best.

New privacy features for commercial subscribers

Wednesday, February 25th, 2009

Launchpad is set up to make it easy for groups to come together spontaneously around tasks big or small. Take this bug as an example: it affects three projects that share the same code. Launchpad brings them together to find a fix.

But there are times when collaboration needs to be done privately. It could be because of security concerns, or confidentiality obligations to a client, or to coordinate the timing of events for marketing purposes, etc.

Launchpad already enables individual bugs to be marked as private but, until now, that’s been all. However, we’re running an interesting beta program at the moment that allows projects to default to both private bugs and private branches. As this will be of most interest to projects who aren’t ready to share their code, we’ll offer it by commercial subscription.

We’ve already been offering basic commercial subscriptions for a few months now. While Launchpad is always free of charge for free software projects, it also hosts some projects with non-free licenses, and charges them a fee to offset the costs of running the site. Until now, these commercial subscriptions have just meant access to Launchpad with no extra features. But following our beta of private branches and private bugs, we hope to offer a wider range of features to commercial subscribers, as they are likely to have different privacy needs from free software projects. If you’re a commercial subscriber and want to take part in the beta, or if you’re considering a commercial subscription, please contact us.

Of course, commercial subscriptions don’t take anything away from the free software projects that use Launchpad. They remain the focus of our development work, particularly as we move to open sourcing Launchpad itself.

Why Launchpad for Mailman?

Friday, February 20th, 2009

Over the last 18 months, I’ve moved GNU Mailman development from SourceForge to Launchpad.  The reasons are varied.

Mailman was one of the first projects hosted on SourceForge ages ago.  I think our project id is a pride-inducing low 103, and we were even highlighted as its Project of the Month at one point.  Of course SF itself uses Mailman to serve its own mailing lists.  While the SF guys have always been great (including providing assistance during the migration to LP), I just became increasingly roadblocked by it.

The first major motivation for moving was Bazaar. This is of course the GPL’d distributed version control system developed by Canonical and used for code hosting on Launchpad.  Having come from decades of SCCS/RCS/CVS/SVN use, distributed version control systems in general and Bazaar in particular have been an enlightenment on the order of learning Python.  I mean, who’d have thunk a version control system could be fun?

After we moved code hosting to Bazaar on LP, evaluating the other benefits of Launchpad became easier.  Truth be told, there was (and still is) some resistance in the community to moving to LP because Mailman is a GNU project but LP is not open source.  That’s being fixed. The next service to migrate was the tracker, and thanks to the excellent assistance of my colleague Graham Binns, we were able to migrate the SF issues to LP.  For years, even before I joined Canonical, I let the Mailman tracker languish because I found it so difficult to use.  The simplicity and power of Launchpad’s tracker really shines for me here, especially with its ability to link across projects and artifacts (e.g. branches linked to bugs).

The next major service to consider is translations.  While being one of the first Python applicatons to be internationalized and translated, we really have a pretty crufty process for updating translations.  Some language champions have commit privileges, but others have to email or upload po-file diffs, or even entire po-files.  This really sucks for a number of reasons, probably most of which is that the code developers are too tightly coupled with the translators.  Our releases are held up for quite a while as we gather updates to more than two dozen languages.

Currently Mailman has an experimental Pootle service.  Despite the gallant efforts of its maintainer, I don’t think it’s working out too well, mostly because it’s not well known and not well integrated with the rest of Mailman’s processes.  Again, here’s where Launchpad integration would really shine for us, but until Launchpad is open sourced we won’t be moving translations here.

Broken link in translation import failure emails

Tuesday, February 17th, 2009

Due to some caching problems between our machines, some large translations have failed to import, generating strange errors such as:

  • “String not terminated” (the most common one, at a big line number)
  • “Got a truncated message!”
  • “String is not quoted”
  • “Invalid content u’msg'” (or other similar instances)

You’d see the error message in import failure emails. Retrying (sometimes a few times) would push the file through.

This morning we’ve disabled caching between the importing and file storing servers, and it seems to have fixed all the problems. However, the fix we used means that everyone will now get a broken URL in translation import failure emails. If the email mentions a URL like

http://mizuho.canonical.com:8000/SOME-NUMBER/filename

replace mizuho.canonical.com with launchpadlibrarian.net and the link should work.

If you still see one of the above errors on your imports, and you are sure it’s not a msgfmt syntax error (or bug #88831), please add a comment to the bug.

Project aliases

Thursday, February 12th, 2009

It’s not unusual for a project to be known by more than one name. Y’know, like Superman and Clark Kent. Or maybe something a bit more subtle, such as PostgreSQL/Postgres.

You can help people find your project in Launchpad, whatever name they know it by, thanks to project aliases.

Give it a whirl:

To request an alias for your project in Launchpad, ask us using Launchpad Answers. If the alias you’re requesting isn’t well-known, we’re more likely to give it to you if you let us know the background.

Launchpad in the Ubuntu Developer Week

Tuesday, January 20th, 2009

It’s Ubuntu Developer Week time again!

During this week, you can find a whole bunch of IRC sessions where members of the Ubuntu developer community introduce a topic — and take questions — that is of use to anyone wanting to get more deeply involved in Ubuntu development.

This time round, we’ve got the following Launchpad-related sessions:

Times are in UTC and you can find the sessions in #ubuntu-classroom on Freenode.

T-shirt competion results!

Tuesday, January 20th, 2009

Back in November we in the Launchpad team made our contribution to the sartorial health of the free software world: the Launchpad t-shirt!

As someone with a clue about fashion might say, the understated charcoal grey of the shirt provides the perfect background to the bright colours of the Launchpad logo. Walk into any conference wearing one of these and everyone will instantly see that you’re all about the colaboration.

But enough of the shilling: it’s time to announce the winners — that’s right, plural — of our Launchpad t-shirt competition! To stand a chance of winning one of our shirts, you had to answer this question:

What’s the average (mean) number of people per team in Launchpad?

Now, this was always going to be a moving target. Each day, people join Launchpad and people register new teams. So, the competition was more about finding a good way to work out the answer.

Before I go into who won, and why, I’d like to thank everyone who entered and also tell you what the answer is right now: 10.28. Here’s how we worked it out:

  1. add up all the active members of each Launchpad team
  2. divide the result by the number of active Launchpad teams.

An honourable mention must go to João Miguel Neves whose answer of 11 was the closest of all the guesses.

So, our winners are:

Congratulations! Launchpad t-shirts are on their way to you.

So, how did they do it?

Mark Lee

Mark was the first person to send us an answer using the Launchpad API:


from __future__ import division
from launchpadlib.launchpad import Launchpad, EDGE_SERVICE_ROOT

launchpad = Launchpad.get_token_and_login('Launchpad T-Shirt
competition', EDGE_SERVICE_ROOT, 'lp-cache')

team_len = 0
team_members = 0

for team in launchpad.people.findTeam(text=''):
   team_len += 1
   team_members += len(team.members)

print team_members / team_len

(Mark’s entry is licensed under the GPL version 3.)

Markus Korn

Markus had a similar idea:


LPMODE=STAGING ipython -p launchpad -c "x = [len(i.members) for i in launchpad.people.findTeam(text='')]; print float(sum(x))/len(x); print len(x); print sum(x)"

To run Markus’ solution, you’ll need to install ipython, bzr branch lp:~thekorn/+junk/ipython.launchpad.profile and then copy/link it to ~/.ipython

Markus noticed something odd when he was working on his entry: the number of teams the API returned was lower than the number reported on Launchpad’s people/teams page. Turns out it’s a bug in the API: launchpadlib’s findTeam only returns teams that have an email address set.

Get your Launchpad t-shirt

You can get hold of your own Launchpad t-shirt — men‘s and women‘s — over in the Canonical shop.

Congratulations to both Mark and Markus and thanks again to everyone who entered the competition 🙂

How we’re open sourcing Launchpad.

Friday, January 16th, 2009

Recently we announced that we’re open sourcing Launchpad, with a self-imposed deadline of 21 July 2009.  (22 June 2009 ed. note: we may push the date a bit, for testing purposes; it could be in August instead.)

Behind that announcement is a lot of activity. Obviously we’re not going to just throw the code over the wall and wish everyone good luck. The point of doing this is to bring the community that uses Launchpad.net into the development of Launchpad.net. There’s a philosophical reason for this: for a site whose primary purpose is to enable free software projects to collaborate, it’s natural that the users should be involved in improving the site itself.

But there’s a more practical reason, too.

One of my favorite metrics for judging a project’s success is to watch its bug tracker: the more bugs filed, and the faster the rate of new filings, the more successful the project is.

Those who don’t work in software sometimes express puzzlement at that metric: “What? How can more bugs be better?” But software developers know exactly what I mean: the rate of bug filing is proportional to the number of users, not to the number of actual bugs. (More precisely, it’s proportional to some combination of the number of users and their level of technical sophistication, since certain kinds of users are much more likely to file bugs than others.)

Well, I’ve been watching the Launchpad.net bug tracker for several weeks now, and judging by the bug-filing metric, Launchpad.net is very successful. Great. But what that means in practice is that we can’t resolve the majority of incoming bug reports and feature requests. The Launchpad development team at Canonical simply does not have enough surface area to handle it all. If we doubled — indeed, if we quadrupled — the size of the team, it still wouldn’t be enough. With a site like this, whose user base is large, and growing rapidly, there are really only two options:

  1. triage mercilessly, and leave most things undone
  2. open it up and let the users help

Clearly, answer 2 is better.

It implies some preparatory work on our part, however. We’re seeding the ground with developer documentation: this has already begun, though we’re still transferring over much of our internal documentation (remember, we have to go through everything and make sure we don’t accidentally open up any confidential business-related things — all this stuff was written by people who thought it was internal to the company at the time).

We also need to go through our dependencies and make sure we don’t have any license compatibility issues. The license on Launchpad will be the AGPLv3, but we still have to vet dependencies and work out any problems. By definition, this must happen before we release, so it’s work that Canonical must do internally.

Most importantly, we need to move our development discussion forums out into the open. Fortunately, most of Canonical’s Launchpad developers already have plenty of experence working in open source communities: they’re ready for this move, and we’re planning to do it before the code itself is opened (see the schedule), to minimize the number of simultaneous changes. We’ll also be publishing policies on how we’d like the community to work, and on how changes will be accepted back into the mainline and deployed on Launchpad.net. (The short answer is that Canonical decides, of course: we host the site, therefore we are responsible for the software that runs on it. The longer answer is that we want to make it easy for good changes to make it onto the site, and have some concrete plans for how to do that.)

In the near term, the next things you’ll see are more independent modules being opened up: code extracted from Launchpad and made into independent libraries for anyone to use (for example, Storm, LAZR.config, and LAZR.delegates).

And that’s all for now. Watch this space for more.

Notification of Launchpad Legal page changes

Tuesday, January 6th, 2009

Hi,

Today we have updated the Launchpad Legal page [1] with the following changes:

1) The Dev wiki [2] is now called out explicitly as having a CC content license.  Previously the Dev wiki proclaimed it was licensed under CC but was not listed on our Legal page.

2) The Content License section was updated for clarity. This was a housekeeping task and does not effect any Legal changes.

3) Future notifications of legal changes will be sent only to the Launchpad Announcement list [3].  Previously they were sent to the Launchpad Users list and News blog.

Joey

[1] https://help.launchpad.net/Legal
[2] https://dev.launchpad.net/
[3] https://lists.ubuntu.com/mailman/listinfo/Launchpad-announce