launchpadblog

Last year, we integrated Launchpad Translations with Launchpad’s code hosting, meaning you could import both translations and templates from a Bazaar branch and also export translations to a branch.

Even at the time, we knew that the story wasn’t complete: you still had to somehow generate your translation templates (in the form of GNU gettext’s .pot files) and get them into your Bazaar branch before people could start translating your project in Launchpad.

However, we also knew that automatically generating translation templates was a big task.

Now, though, I’m pleased to say that Launchpad can automatically generate the templates on your behalf.

How to get it all set up for your project?

Automatic translation template generation relies on something called intltool. You’ll need to be familiar with intltool before you can get started with automatic template generation.

You first need to enable your branches for intltool and then set up a translation template import fromn the Bazaar branch that is linked to your project’s release series.

This means that, provided your branch has proper structure, you don’t even have to keep the POT file committed anymore (as a matter of fact, it’s better if you don’t). If your branch is not recognized as intltool branch, everything will keep working as before.

At this time, limits to what branches we consider intltool based are pretty strict: it has to have a POTFILES.in file in each of the template subdirectories, and be able to derive the domain name from Makevars DOMAIN variable or Makefile.in.in, configure.ac or configure.in gettext_PACKAGE variable (with very limited substitution supported). This will be further improved in the future, but plan is to support much more different layouts than just the intltool one.

We’ll be writing more about how to make the most of this in the coming weeks.

The last few months we’ve been doing a lot of work to enable direct import of translations from different upstream VCS systems. For now, we’ve focused on getting one very important case right first (GNOME), and then we’ll extend it to supporting other upstreams as well.

How are we going to do it? First off, we’ve split it all into two separate stages:

• get upstream translations into Launchpad
• push upstream translations from Launchpad into Ubuntu

For some upstreams, getting them into Launchpad is trivial: they might already be hosted in Launchpad. For majority of them, however, it means pulling from different VCS systems. Thanks to Launchpad Code and Bazaar teams, getting the code in the form of bazaar branch is not that big a deal. However, when pulling translations from a VCS instead of getting them from tarballs means one slight complication. Translation templates (POT files) won’t be there, and we’ll have to regenerate them.

Regenerating templates differs from project to project. And doing it should be considered an unsafe operation. So, in the first step we are only going to support intltool-based modules, and generation of templates will happen inside a sandboxed environment. This will enable us to import upstream translations directly into read-only Launchpad projects: this is marked with green-coloured arrows on the diagram.

After that is done, we’ll start pushing all these translations directly into Ubuntu (blue-coloured arrows), minimizing the time it takes for translations to get from upstream translators to Ubuntu users.

I’ve written a more thorough explanation in my personal blog, so check it out.

Parts of this will be rolled out this cycle, but more will come in the coming months.

Launchpad 3.0 is a major milestone for Translations. Along with many small improvements across the board, here are the highlights.

Improved UI and navigation

Along with the rest of Launchpad, Translations has switched to the new and stream-lined 3.0 layout. However, that’s not all: we’ve fixed a huge number of small annoyances while doing this conversion, and we expect your experience to be much nicer.

One of bigger improvements has come through…

Personal dashboard

You are a reviewer in a translation team, and wonder if somebody has submitted any suggestions you can look at? Or, you’re stumped about what you could translate next?

Your personal dashboard now lists all the translations you’ve worked on in the past if they’ve got something you can help with: unreviewed suggestions, or strings that need translating.

Bazaar integration

You’ve got your code hosted in Launchpad, but you have to manually upload and download tarballs with translation files?

Not any more! With 3.0, Launchpad allows you to directly import all translations from your code branches, and to have them automatically committed to a separate or the very same branch.

Translations sharing

Your project has multiple series (eg. stable and trunk)? Translators have to go through both just to keep them up-to-date? Well, not with Launchpad.

If same strings appear in both series, translating it on one will have it automatically get translated in the other.

Over the past few months we’ve been slowly introducing a grand new feature in Launchpad Translations, and with 2.2.7, we are finally ready to announce it to the wide world: message sharing.

Introducing message sharing

Message sharing between different releases of a product or distribution in Launchpad means that translations done in one release (eg. trunk) would immediately apply to translations in another release (eg. stable). This should benefit everyone using Launchpad for translations, one way or another.

• Translators will not have to worry about back-porting translation fixes to older releases anymore, and they can simply translate the latest release: translations will automatically propagate to older releases. Also, this works both ways, so if you are translating a current stable release, newer development release will get those updates too!
• Project maintainers hosting their translations in Launchpad, when they upload a template to a new release series and it gets imported, will instantly get all existing translations from previous releases shared with the new series and translators won’t have to re-do their work. They won’t have to worry about uploading correct versions of translated PO files, and can just care about POT files instead.
• For Ubuntu, there’s another benefit: opening a new release for translations will take minutes instead of days. Message sharing also improves the scalability of the system, and we should soon start seeing more performance improvements as the result of migrating to this new way of managing translations.

How to benefit from message sharing?

Over the next few weeks, we’ll be migrating all existing projects to make the best use of message sharing. If there are any potential problems with existing projects, we’ll be emailing their maintainers and resolving it as we go along. Messages will be shared only between templates in one particular project, and only if they all have the same name. For Ubuntu, they will be shared only between templates in one particular source package, and again, only if they are of the same name.

For example, Ubuntu Jaunty and Ubuntu Karmic already share their messages. If you update a translation for one single string in Karmic GNOME Panel, and that same string exists in Jaunty GNOME Panel, Jaunty translation will be instantly and automatically updated to exactly the same translation.

If you want one particular release to have a different translation from all the other releases, that’s still possible. So, if you do not want a translation in Jaunty to be modified when a translation is changed in Karmic, you have that option too. Since we don’t expect this option to be used often, it’s hidden in the “zoomed in” view on each single message.

What next?

We’ll have to carry out a data migration for each of the projects using Launchpad Translations. Luckily, this migration will be completely transparent, so there’s nothing for anyone else to do. On the outside, nothing will change, until you start doing translations for more than one release series, when they’ll just be better.

New projects will get benefits of message sharing right away, and existing ones should be a bit patient because doing the migration takes some time, but allows to keep the full history of translations intact.

If you are interested to learn more, come talk to us on #launchpad on FreeNode, or join the launchpad-users mailing list.

A couple of days ago I posted that Launchpad had been assigning too much karma for certain types of translation work.

Basically, if you imported translation files — PO and POT files — into Launchpad from an Ubuntu package or a Bazaar branch then you got vastly more karma than if you simply translated directly through Launchpad’s web interface or uploaded PO files as ‘user uploads’.

Of course, that was unfair so we decided to adjust the inflated karma that Launchpad had awarded. The way Launchpad calculates karma is pretty complex, so the manual reduction we made to karma awarded for those translation imports has also affected karma given for other types of work — such as bugs, code, or blueprints — even for people who never did anything in Translations.

So, what’s going on?

Launchpad treats all its applications as equal in terms of the karma they can give out. Let’s call that the “total karma pool per application”. This means that if the total sum of all karma assigned in Blueprints to all users is X, the total karma assigned in Bugs or Translations will also be X.

However, X is determined as the largest of the actual sums of karma values all applications have. So, if Translations is handing out the most karma in Launchpad, all karma values for other applications will be scaled up to match.

There’s more in our explanation of karma calculation on the help pages.

So, what does this mean? After we fixed bug #286359 to send import notifications to Ubuntu packagers, we also started giving out translation import karma for them. An unproportionally large amount of karma was given out compared to other stuff in Translations, and that caused the total pool of Translations karma to grow, so X started growing for all applications. People noticed that their Translations karma was getting bigger and bigger, and reported bugs such as bug #337645. We also got a few comments on IRC. However, nobody noticed — or at least, they didn’t complain 🙂 — that their other karma was growing along with the growing total karma pool for Translations. Suddenly (or not that suddenly, the scaling factor increased over time), people had about 2–3 times more karma then they were supposed to.

Finally, 6 weeks after we introduced the problem, we’ve normalized translations karma so uploaders do not have an unfair advantage over actual translators. That meant that the total pool of Translations karma went down, and with it, X went down and everybody’s karma across the system went down, regardless of the application. Note that everybody’s non-Translations karma got proportionally scaled down, so relative karma scores are still the same (i.e. if you had more karma than your friend Terry a week ago, you’ve still got more karma than Terry today, unless Terry has subsequently gone on a Launchpad work rampage).

What can we do to lessen the consequences of any karma normalization in the future? We’ve got to be careful when introducing or modifying existing actions. We’ve got to normalize karma points before we work on a feature which actively gives them out. And we’ve got to better communicate changes like this.

So, what actually happened is: karma that was never meant to exist disappeared. Many parallels to the economic situation are easy to envision. Launchpad was The Karma Bank. 🙂

Translations karma we assign to Ubuntu package maintainers and project maintainers importing their translations through Bazaar is too big compared to what translators get when working through Launchpad. We’ll be scaling down the karma values for such imports, as part of resolving bug 337313 (see that for details). This means that some Ubuntu contributors will see their Translations karma drop significantly, but I am sure they won’t be surprised (or at least, honestly disappointed), since translators will get to retake their top positions as top translation contributors.

Note that this doesn’t affect uploads translators can do through Launchpad Translations web pages.

A few days ago a fix for bug #286359 landed: provide translation import notifications to Ubuntu packagers. This means that all packagers will start getting emails about translation files (basically, *.pot and *.po files) from their packages as soon as they are imported (or the import fails).

This should help packagers detect and fix l10n-related problems more easily, since they’ll have most of the debugging information they need right inside their inboxes. If you want to filter these emails, look for a sender of rosetta@launchpad.net.

The grey area of where to send notifications for automatically synced package uploads from Debian remains. I am planning on solving that with a public mailing list where anyone interested can go in and check for problems, and also fix them if they want to. Alternative is to rely on the recent improvement by Jeroen which keeps any failure messages inside the import queue itself, and simply drop all of these messages altogether.

Due to some caching problems between our machines, some large translations have failed to import, generating strange errors such as:

• “String not terminated” (the most common one, at a big line number)
• “Got a truncated message!”
• “String is not quoted”
• “Invalid content u’msg'” (or other similar instances)

You’d see the error message in import failure emails. Retrying (sometimes a few times) would push the file through.

This morning we’ve disabled caching between the importing and file storing servers, and it seems to have fixed all the problems. However, the fix we used means that everyone will now get a broken URL in translation import failure emails. If the email mentions a URL like

http://mizuho.canonical.com:8000/SOME-NUMBER/filename

replace mizuho.canonical.com with launchpadlibrarian.net and the link should work.

If you still see one of the above errors on your imports, and you are sure it’s not a msgfmt syntax error (or bug #88831), please add a comment to the bug.

The week of February 2nd – 6th has been the inaugural Launchpad Performance Week: a week where we concentrate on the most pressing performance issues and do anything we can to improve them. During that week, we also establish a hard goal that we enforce by lowering the timeout limit by 5 seconds.

The Launchpad Translations team has been concentrating on improving performance of two most problematic pages we are in charge of: POFile +translate pages (where translators do most of their work) and Person +translations pages, which display history for a person’s translation contributions inside Launchpad.

Jeroen has done a very good job improving Person:+translations pages (pages like Yannick’s translation history which used to timeout occasionally). We have since seen no timeouts on that page.

Henning has spent some time on writing a script to remove obsolete Ubuntu translations (translations for Warthy, Hoary, Breezy, Edgy and Feisty) from the database, before joining Ubuntu guys for the Jaunty sprint in Berlin. This should remove around one third of our entire data set, which should help with rendering of POFile:+translate pages. Unfortunately, he was unable to complete that — due to not being around 🙂 — so I picked it up and have completed and tested it this week. We should have this done by the end of the month, and that should bring large improvements to the rendering of +translate page.

Additionally, I’ve disabled global suggestions on edge (translations coming from different PO templates in Launchpad for the same English string) to confirm this is indeed our biggest problem. Edge has practically been flawless since, and we’ve discovered more areas for improvement. We are getting some new hardware to help with competition for resources between different jobs on the server as well (should be in by the end of the month), and as soon as we do, we’ll re-enable global suggestions (which are one of Launchpad’s essential features).

Just as the week was ending, we started getting a bunch of timeouts on DistroSeriesLanguage pages like the Brazilian Portuguese one for Jaunty: anonymous users started getting it frequently using a script, along with a large batch size of 300, and that started putting a lot of load on the server. Jeroen already has an idea how to improve the situation significantly, but that will have to wait until a later date.

I am looking forward to February release of Launchpad which should bring performance improvements for everyone to notice.