Jtv on Launchpad Blog

Translation imports: making approval conflicts visible

Jeroen Vermeulen — Mon, 19 Sep 2011 12:46:10 +0000

If you use Launchpad’s Translations feature, then as of today, you may notice a new kind of error for uploads in your translations import queue. Look for the “” icon next to your upload’s status, and click to unfold. The error message says “Can’t auto-approve upload: it’s not clear what template it belongs to.”

You may find this annoying, and as the person responsible, let me apologize. I hope when I’m done explaining, it won’t seem so bad.

To the right is an example of the error. Click it to see more.

Where does the error come from? Actually it’s a problem that’s always been there, but instead of telling you about it, we Launchpad developers tried to hide the complexity from you and take care of it all ourselves. And we weren’t keeping up. Some of your uploaded translations would just sit in the queue forever, with status Needs Review, for no clear reason. All that’s changing now is that Launchpad will tell you when this happens, so that you can deal with it without waiting for us.

Why do some translations sit in the queue forever?

Every translation you make in Launchpad has to go into a specific template and language. Usually it’s obvious where you want your translation to go: when you translate in Launchpad’s web UI, you’re already on the page for a specific template and language. If you have upload privileges for the project and language, you can follow the upload link from that same translation page and again it’s obvious which template you’re translating and to what language. If you always upload the same file with exactly the same name and path, new uploads of that file go to the same place as before. But what if you upload a new file from the release-series page, or the translation comes in from a Bazaar branch or an automated build?

Then it’s up to a script we call the Translations Import Queue Gardener. The Gardener periodically scans all waiting uploads—the ones marked Needs Review—and tries more advanced ways of matching them up with known templates and languages. When it finds a match, it approves the upload’s import to the template and language it has found.

One of the Gardener’s most important tools is a template’s ‘’translation domain.’’ This is a simple name; no slashes or weird characters allowed. Launchpad figures out a template’s domain when you first import the template, based on its directory path. In principle a template’s domain should identify the template uniquely on the end-user’s system, but Launchpad isn’t as strict about it. It just assumes that the domain name is tied to the template file’s path within a project’s source tree. You probably shouldn’t, but you can give your project two templates with the same domain if you want.

When you upload a translation, the Gardener tries to figure out its translation domain based on the file’s path, just like what happened when the template was created. The Gardener looks for a template with the right domain in the same release series. If it finds one, presto: it’s got the template that the upload is meant for. If not, then the Gardener tries a few other things and if nothing works, simply keeps the upload on the queue.

So the entry doesn’t get imported, but usually the Gardener can’t give any single reason: all it knows is that it tried many ways of matching the file to a template and none of them worked. Maybe it’s an error; maybe it’s just a matter of waiting for the right template to be created.

So what’s changed?

The new error message is about “approval conflicts.” You’ll see it when there is ‘‘more than one’’ matching template for your upload. This can happen because your project’s directory structure is unusual and Launchpad can’t extract a meaningful domain from it. Or a template’s domain may have been changed, or an old deactivated template is reactivated when a new one has already taken its place.

Whatever the cause, the new error message tells you that this has happened, and what the matching templates are. It’s up to you to decide what needs to be done about it:

update one of the templates’ domains, or
deactivate an obsolete template, or
move a file, or
re-upload your file to a specific template, or
re-upload your file with a different name, or
upload translations as a tarball so that Launchpad sees their full directory paths.

How much you can do, of course, depends on the permisisons you have. Only the project’s owners (and Launchpad’s administrators) can deactivate templates or change their domains. But you can always delete your own uploads if you want to upload your file differently: on the import-queue page, click on Needs Review and select Deleted instead.

Let us know

There’s still plenty more we’d like to do to make imports easier and more efficient, if we can find the time. But I hope this small change will make your life a little easier.

Is this error message working for you? Is it helpful? Are you seeing a lot of these errors, or none at all where you were expecting them? Is the explanation unclear? Do you see something happen for lots of people that we could fix in the same way? Please get in touch:

Contact danilos or jtv on IRC, in #launchpad on Freenode.
Find or file a bug if it’s broken.

Tips

If you want to become more familiar with the translations import queue, check out the global queue to see all uploads in Launchpad. The version you see there is just a copy on a test server, so don’t be afraid to play with it.
By the way, did you know about our test servers? We test our changes on staging and qastaging servers before they go live. You can try out most of Launchpad’s features there. Look for the grey “demo” text in the background. They get restored to a fresh copy of the real Launchpad once a week.
Tired of creating translations tarballs and uploading them to your project? Automate it all with Bazaar integration.
You want Bazaar integration but your code is hosted outside Launchpad and/or in a different revision control system? You can tell Launchpad to mirror a branch from elsewhere. Then you can import translations from the mirrored copy.
We like transparency! If you’re interested in the engineering details of this change, it’s all online.

Trying Out Launchpad Translations

Jeroen Vermeulen — Tue, 22 Dec 2009 13:31:15 +0000

You may already know this: if you want to try out Launchpad, there’s a “playground” version of the site at https://staging.launchpad.net/

This site gets refreshed daily (or almost daily) with the latest copy of the database that’s behind the real, production Launchpad site. That makes it a good place to mess around: it’s pretty realistic but you won’t damage anything, and you won’t pollute Launchpad’s real database with fake information. It also runs newer code than the production site, so the Launchpad developers themselves use it for testing their latest changes.

Until recently, you couldn’t really test Launchpad Translations on the staging site: you needed help from an admin to get your templates imported, and if you got that done before they were overwritten by the daily refresh, you couldn’t export your translations. But today things are a lot better thanks to translation imports from, and exports to, bzr branches.

You will need:

A Launchpad login. You can only get this on the real Launchpad; but it will also be valid on staging.
At least one gettext translation template (".pot file") with strings from your program.

The translation template should have a filename in all lower-case letters and end in “.pot” as a filename extension. It should contain English “msgid” strings and empty “msgstr” strings as usual in the gettext format. (The msgstr strings don’t actually need to be empty, but in a template their contents will be ignored.)

You will be setting up:

A bzr “development branch” on staging. This is how your templates get into the system.
A bzr “translations branch” on staging. The system will write your translations into this branch.

Here’s how you set up and test translations on the staging site. With each of these steps, always make sure you’re working on staging, not on the production site! The staging site has the word “demo” scrawled diagonally across the page background. You may need to log in separately on this server; your account and password are the same as on the real Launchpad site.

1. Set up a project

You can do that on the project registration page. Or if you already had your project registered on the production site, it will show up in staging as well. In that case, just work with the project you have.

On the project’s main page in staging, select “Change details.” This takes you to the project’s setting page. Enable translations: check the box labeled “Translations for this project are done in Launchpad” and submit using the Change button at the bottom.

Your project should have a “trunk” release series. This is where you will be doing most of your work. Another release series will work just as well, but trunk is the default.

Detailed documentation

2. Set up a development branch

You may already have a development branch set up on the real Launchpad. But staging is a separate environment that does not have copies of your production branches, so you’ll have to set up a branch on staging anyway. Like everything else on staging, this branch will disappear when the staging site is refreshed.

The first thing to do is to make sure your bzr program is logged into Launchpad:

bzr launchpad-login mylogin

(Where mylogin stands for your Launchpad login name). Next you’ll push a branch to the staging server. How to do this depends on whether your project already has a bzr branch on Launchpad:

a. You have a bzr branch on your local system

You’ll have to push a copy of the branch to the staging server. Open a command-line shell and go into your branch. Then, supplying your own Launchpad login and project name in the obvious places, push the branch:

bzr push –use-existing-dir lp://staging/~mylogin/acme-zyzzyx/testbranch

This may complain:

bzr: ERROR: Target directory lp://staging/~mylogin/acme-zyzzyx/testbranch
already exists, but does not have a valid .bzr directory. Supply –use-existing-dir
to push there anyway.

If it does that, just run the command again and it should work.

b. You have a bzr branch on Launchpad

Go to the regular Launchpad page for the branch. The page will show how to access the branch using the bzr command-line tool. For example,

Get this branch:

bzr branch lp:acme-zyzzyx

(That is assuming that your branch is the main development branch for a project called “acme-zyzzyx.” Yours probably has a different name; it could also be of another form such as “lp:~mylogin/acme-zyzzyx/trunk”).

On your local system, go to some empty directory where you can create temporary data. Execute the “bzr branch” command line there. This creates a directory with your branch contents. Go into that directory, and push the branch to staging. You do this using the “bzr push” command:

bzr push –use-existing-dir lp://staging/~mylogin/acme-zyzzyx/testbranch

(Of course you use your own login and project name instead of “mylogin” and “acme-zyzzyx”). The “bzr push” command creates a real branch on staging. It may complain:

bzr: ERROR: Target directory lp://staging/~mylogin/acme-zyzzyx/testbranch
already exists, but does not have a valid .bzr directory. Supply –use-existing-dir
to push there anyway.

If it does that, just run the command again and it should work.

c. You don’t have a bzr branch

It’s okay if your branch is actually in some other revision control system such as subversion or git. Create a temporary copy of your source tree, so you can play without damaging anything. From the command line, go into the copy and run:

bzr init

Now use “bzr add” to add all the files that you want to see appear in the staging branch. For testing the only part that really matters is your translation templates:

bzr add po/messages.pot

Your directory structure and filenames may be different of course, but the filename should end in .pot. Commit your changes:

bzr commit -m “Setting up test branch.”

Congratulations—you have now set up a bzr branch for your project. Push a copy to the staging server. Supplying your own Launchpad login and project names in the obvious places, type:

bzr push –use-existing-dir lp://staging/~mylogin/acme-zyzzyx/testbranch

This may complain:

bzr: ERROR: Target directory lp://staging/~mylogin/acme-zyzzyx/testbranch
already exists, but does not have a valid .bzr directory. Supply –use-existing-dir
to push there anyway.

If it does that, just run the command again and it should work.

Make a note of that “bzr push” command line. You can make changes to the template later, commit them, and push them to the same location.

Detailed documentation

3. Set up a translations export branch

Besides import your template from a bzr branch, Launchpad can also export the template’s translations to a bzr branch. The real Launchpad site does this once a day, but since staging doesn’t do any of the heavy lifting, and data on staging doesn’t have so long to live, it does this every hour.

Let’s start with an empty branch for this that will contain just the translations. There can be good reasons to do it differently in your real project, but exporting to an empty branch is always a good way to start. That way you can make sure that the exports really do what you want them to before making Launchpad write to a real branch.

In some temporary directory on your system, run this from the command line:

mkdir translations-export-test
cd translations-export-test
bzr init
bzr commit –unchanged -m “Setting up translations branch.”
bzr push lp://staging/~mylogin/acme-zyzzyx/translations-export-test

(Here you probably see another complaint about “Target directory […] already exists, but does not have a valid .bzr directory.” Just run that last command line again to get past it.)

Make a note of the exact URL in that “bzr push” command. You’ll be pulling the exported translations from there into your local branch.

Detailed documentation

4. Configuring imports and exports

We now have two branches, one to import templates from and another to export translations to. Next we configure imports from your development branch and exports to your translations branch.

For the imports, tie your staging development branch to your project’s trunk release series. Go to the project’s main page on the staging server, and from there, click on to the “trunk” release series. The page you arrive at should have a section “Code for this series.” In that section, you may see a link to your real branch on production. The production branch won’t work on staging, so replace it with your development branch on staging. Click on the little “edit” icon next to the branch URL to go to the settings form. If no branch was tied to release series yet, click “link the branch to this series” instead.

You can now ask the staging server to start importing and exporting. Back on the “trunk” page, select the Translations tab at the top of the page. Look for a link “Set up branch synchronization,” and follow it.

This takes you to the page where translations synchronization with your branches is configured. It should show your chosen development branch (the one on staging!) as the import branch. Under “Import settings” select “Import template files” for now, and click “Save settings.” From now on, every time you commit a change to your translation template and push it to your development branch on staging, the latest version of the template will be imported to the staging server automatically. It usually happens within the hour.

Important: on the real Launchpad site, the development branch can also be a mirrored version of a branch you keep elsewhere. You can register an external branch (kept in CVS, Subversion, git, or bzr) and let Launchpad mirror it to a local branch. That automates the full transport from committing your template changes to your own repository on another server, all the way to having the changes show up in Launchpad Translations.

Detailed documentation

We’ll also want to set up translation exports on this page. Under “Export translations to branch,” you’ll see a link “Choose a target branch.” Selecting a branch will enable the exports immediately. (If an export branch has already been set, you’ll see its URL with an “edit” icon next to it. Clicking the icon will take you to the page where you set the branch.) Use the link to set your translations export branch.

Important: the translations export branch must be one that you own, and it must have been pushed to the server—just registering a branch is not enough. In the future it’ll also be possible to use a branch that is owned by a team that you are a member of. That plan is tracked in Launchpad under bug 407260.

At this point you have a viable working setup. As changes to your template (or even completely new templates) appear in your development branch, they are imported automatically. Snapshots of your translations are periodically exported to your translations branch. You may want to refine this further; I’ll go into that later.

Detailed documentation

5. Check up on the imports

Go to the translations import queue for your project. You’ll see links to the queue from the Translations pages for your project and for the trunk series. Within the hour you’ll see an entry for your template appear there. Its initial status will be Approved, which means it’s sitting in the queue waiting to be processed. Pending imports are processed several times an hour. Once your template has had its turn, if you refresh the queue page, its status will show as Imported (or Failed if there were errors).

Once your template has been imported, it will show up in the Translations page for your trunk series. You will see red bars for each of the languages you have set up in your preferences, standing for untranslated messages. The number of untranslated messages is indicated next to each. Until your template is imported, the red bars will show but not the numbers.

6. Translate!

Try clicking on one of the languages in the translations list with the red bars. This will take you to the template’s translation page: the UI for entering translations. Try translating one or more messages. Remember to hit “Save & continue” at the bottom to submit your changes.

Your changes are going to show up magically in your export branch. Remember the location you pushed your translations branch to on staging? Go back into your local copy of that branch and run, with that same URL:

bzr pull lp://staging/~mylogin/acme-zyzzyx/translations-export-test

Most likely, nothing will happen and the command will just say “no revisions to pull.” But keep trying! Sometime within the next hour, your latest translations will be written into the branch. When you set things up on production, you may want to set up an automated job (e.g. using “cron”) to pull snapshots of your latest translations to wherever you like to keep your translations.

Detailed documentation

7. Update the template

As your program changes, so will your template. You’ll probably run the xgettext command to extract the latest message strings. But today, since we’re only trying things out, you can edit the .pot file in a regular text editor.

So edit the template in your staging development branch. Add a new message:

msgid “Launchpad translations test message.”
msgstr ""

Now commit the change, and push it back onto staging using the same URL where you pushed your development branch before:

bzr commit -m “Template update.”
bzr push lp://staging/~mylogin/acme-zyzzyx/testbranch

Your template’s entry on the import queue will go back to being Approved, and soon after that, will be imported again. Now go back to the translation page. The new message, “Launchpad translations test message” will appear in the same relative position where you inserted it into the template.

This is how your project’s translations in Launchpad will keep up with development. From time to time you’ll want to start work on a big new release while translators continue to work on the existing release. That’s where you create a new release series. Each series has its own branches and its own translations. If the same English string occurs unchanged between two release series, the two will share their translations by default; but the old release series will be less of a “moving target” for the translators as the English strings probably won’t change so much between updates as you focus on the new release series.

8. Import translations

Remember how we set up imports of templates only? You can auto-import translations (".po" files) as well. Go back to the branch synchronization options (on the trunk page, under the Translations tab) and set the import option to “Import template and translation files.”

Now, go back to your development branch. Did you have any PO files in there? If not, copy the one that was created in your translations export branch to the same directory where your template is. Translate a previously untranslated message so that you can see the changes happening. Add the file to the branch with “bzr add,” commit the change with “bzr commit,” and finally re-run the “bzr push” command.

If you already had PO files in the branch, all you need to do is use the one-time import option on the branch synchronization page. Each PO file should be in the same directory as its template, and be named after its language code with the “.po” extension: de.po for German, pt.po for Portuguese, pt_BR.po for Brazilian Portuguese, el.po for Greek, zh_CN.po for Simplified Chinese and zh_TW.po for Traditional Chinese, es.po for Spanish, and so on.

Now wait for up to one hour (but probably less) for your PO file to appear on the import queue for your project. This time it will appear as “Needs review”; a script that runs about once or twice an hour will approve it automatically. (If it doesn’t, check that you followed the naming rules). It will be imported soon after that.

If you go back to the translation pages, you’ll now see that the changes you pushed into the development branch have become visible in the translation UI. In the translations overview for trunk you’ll see translated messages show up as green (translated) bars instead of red (untranslated).

Note: if you made a translations change in the branch for a message that you had already translated in the Launchpad UI, you may not see the change appear. Translation changes made locally in Launchpad override translations that are imported from elsewhere; you’ll see such changes show up as light blue in the status bars. This allows you to keep track of changes made in Launchpad, and if the project’s translations are centralized elsewhere, feed the changes back “upstream.” You should try to keep the differences minimal by making sure that improvements are fed back to the origin, and that changes made in Launchpad are real improvements.

Detailed documentation

9. Going full-circle

By now you’re probably thinking that it’s not very useful to import translations from one branch and export them to another. Things get much more interesting when you export the Launchpad translations back to the branch they came from!

This is actually possible if your development branch is hosted on Launchpad. There’s nothing stopping you from using the development branch as the translations export branch as well. If you do that, you’ve got full two-way synchronization between your branch and the Launchpad user interface. Your translators can work in the Launchpad UI, or they can edit PO files from the branch and push them right back using bzr.

We normally recommend that you import only your templates from a branch, and export your translations to a branch (which may be the same one). But depending on how you choose to use Launchpad, two-way synchronization of translation files can be useful as well. There are a few important things to keep in mind though.

First, the exported translations may not always be exactly what you expect. If you name your PO files after invalid language codes, like “de_DE.po” (should be “de.po”) or “zh.po” (should be either “zh_CN.po” or “zh_TW.po” depending on which language it really is), they will normally not be imported. The same can happen if you include other things than the language code in the filenames, such as the template name. But if these files ever do get imported, they’ll be exported with their normalized, proper names. So it’s possible to end up with both a de.po and a de_DE.po in your branch. Of these, only the correctly-named one will be updated and re-imported—de.po in this case—even though your translators may still try to work on the other one.

Second, the translations export will simply overwrite any files that were already there. There is no intelligent merging, and “merge conflicts” are ignored. If you set up translations export to your branch before the first import has happened, the export can overwrite translations that were in the branch. Make sure this is really okay before enabling the exports!

There’s also the risk of “crossing” translations: translator A makes a change in the UI, around the same time translator B pushes an updated PO file to the branch. If there are pending changes in the branch when the export is scheduled to run, the export will not happen so as to avoid overwriting the changed files. Of course this does mean that frequent changes to a large branch can stop the exports from happening.

Third, an exported translation file will not be completely identical to the original, even if there were no changes in the Launchpad UI. Messages can appear in a different order, and the file’s header is updated. You may want to try merging Launchpad’s exported files into the original branch manually to get past the initial changes.

So, before you set up two-way synchronization, always test the exports on an empty branch. The staging server is the ideal environment for this. Or if you do it on the real Launchpad site, remember to save space by deleting the experimental branch once you’re done with it.

10. Help! Staging refreshed and now everything’s gone!

This happens once a day, or depending on the circumstances, once every few days. If you still have your branches on your local system, and your project exists on the real Launchpad site, it’s not hard to set things up on staging again for another experiment. Just re-run the “bzr push” command lines for the branches and set up the imports and exports again.

Once you’re satisfied, we hope you’ll decide to set up translations permanently on the real Launchpad.

Exporting translations to a Bazaar branch

Jeroen Vermeulen — Thu, 30 Jul 2009 10:06:35 +0000

There used to be only one way of exporting translations from Launchpad—by requesting your files in the Launchpad UI and waiting for an email with the download URL. It works, but it’s not very convenient if you’re trying to automate things. It’s not that easy to make the request automatically, and then right in the middle of your script you also have to wait for the email, catch it, and parse it to get the file.

Now there’s another option: automatic exports to a Bazaar branch. If you set up this option, Launchpad will regularly produce a snapshot of your translations and commit it to a Launchpad-hosted branch of your choice. Now you can always find a reasonably fresh export of your translations in the same place, and download it automatically, without any asynchronous requests.

It also means that large numbers of developers or translators can get regular, fresh updates of the translation files.

How to use it

Let’s say you have a project called bzamqi. It’s set up for translation in Launchpad, and you want regular translation exports of the 1.0 release series.

The first thing you need is a branch. It doesn’t have to be a branch for the bzamqi project, but you do have to be the owner. (The branch can belong to a team, but then you do need to be a direct or indirect member of that team.) The Launchpad help wiki describes various ways to create a branch:

https://help.launchpad.net/Code/UploadingABranch
https://help.launchpad.net/Code/TeamBranches
https://help.launchpad.net/Code/PersonalBranches

Just remember that the branch must be Hosted, i.e. the master branch must be stored on Launchpad where we can commit to it.

Update: at the moment, team-owned branches don’t work. To work around this, make yourself the owner of the branch; set it as the translations branch; and then make the team the owner of the branch again.

Now, go to the page for bzamqi’s 1.0 release series. You’ll be wanting the Translations tab. Once there, pick the Settings option.

On the page that takes you to, near the bottom, there’s a section “Export translations to branch.” You can set a branch there, and translations for the series will be exported to that branch.

If you want to disable the exports, go back (use the “pencil” icon next to the branch name to select a different branch) and select no branch at all. Just clear the input field. The settings page will go back to how it was before you selected a branch.

How it works

The main thing to grok is that this is not Launchpad “editing” text files for you. The export brutally overwrites any previous versions of the files in your branch. In the file that Launchpad writes, translations may have changed—but messages may also appear in a different order, or with different comments, and so on.

So do not expect files that you can merge back into the originals without any further work! The Launchpad database and bzr have completely different views of this data. There is a trick though; see the “Advanced” bit below. Also, it will help to keep the messages in translation files in the same order in which they appear in the template.

Directory layout

In the branch, each translation file will have a normalized name (de.po for a German translation, zh_CN.po for Simplified Chinese, eo.po for Esperanto, and so on). Each translation sits in the same directory where its template was on import.

If you have multiple templates, make sure that they have different directory paths, or the translation files for one will overwrite those for the other! It was already recommended practice to give each template its own directory.

Templates are not exported.

Advanced: going two-way

It is possible to import translations from and export translations to the same branch. If you push changes to a translation file into the branch, the export will not overwrite the changed file unless your updated file has already been copied onto the translations import queue.

Of course it’s still possible for the import to fail for whatever reason, and in that case your file will be overwritten. This hopefully won’t hurt; the file that’s being exported is “better” in the sense that it shouldn’t cause any import errors. But it may not always be what you want, so be careful if you do choose this setup.

(Your templates will not be touched, since the export doesn’t write them.)

Don’t use this “two-way” setup lightly, since it complicates things. Complications are where accidents happen. Either wait until there’s more practical experience with this, or make sure in some way that you won’t lose data if something goes horribly wrong.

A bit less adventurous but still useful is to have only your templates imported from a branch, and have your translations committed to the same branch.

Questions

How often do the exports happen?

For now, once a day. We may tweak that later depending on how much this feature is used and how heavily it loads the servers.

If the export runs at a moment when you’ve pushed changes into the branch that Launchpad hasn’t processed yet, the export will not happen. This is a deliberate protection against conflicting updates. Unless you update your branch all the time or just at the wrong time of day, the next day’s export should catch up with the latest changes.

Why do I need to own the branch?

Otherwise you would be making Launchpad write data into someone else’s branch. Not a nice thing to do to someone who isn’t expecting it!

If you want someone else to own the branch, you can change who owns it later, using the normal branch settings UI. You only need to be the owner when you set it as the translations export branch.

What happens if I leave the project?

The exports keep happening to the same branch. If that is not what you want, it’s up to the remaining project owners to choose a different branch or disable the exports.