launchpadblog

Jonathan Lange writes on his blog:

Launchpad has a pretty awesome public API, implemented using lazr.restful. I’ve written a few small scripts for it, and the Launchpad team has a few scripts that they use internally for doing admin tasks.

The Ubuntu Platform team does a heap of stuff with the Launchpad API. James Westby has been using it to make sure that there’s a branch on Launchpad for every single package in Ubuntu.

There’s all this great work, but there’s been nothing to tie the room together. I’ve seen hardly any discussion about how to write Launchpad API applications, or how to test them, or how to get launchpadlib working in GTK+. I haven’t even seen much code sharing.

So, borrowing a trick from Twisted’s tx super-project, I’ve created an ‘lpx‘ project group on Launchpad. Bring it your scripts, your applications, your huddled masses. If you want to know more about the API, look at the API help page.

Also, if you’re using the Launchpad API — directly or through the launchpadlib Python library — add some info about your app to the API Uses wiki page.

People are starting to integrate Launchpad into applications, talking to it directly through its API — either by using the launchpadlib Python library or by doing RESTful HTTP requests.

Below are some of the uses we’ve found — if you know of some not on this list, please tell us (in the comments here is fine). This is partly to help us see what areas of the API are getting the most mileage, and partly because this collection might be a good resource for people who want example code for using the Launchpad API themselves.

(Note: we’ve now collected this information into a Launchpad API Usage directory, and will maintain the list there from now on.)

• bzr-eclipse, a Bazaar plugin for Eclipse, that talks to the Launchpad/Bazaar integration using launchpadlib. The developer says: “The API is straightforward to learn, also if you can use launchpadlib it’s far easier, just start the python interactive interpreter, import launchpadlib and start prototyping your app :)”

• Tarmac, “The Launchpad Landing Strip”: an automatic branch lander for the Bazaar branches hosted on Launchpad. Tarmac uses the Launchpad API to manage a development focus branch’s proposed merges. It will automatically merge approved branch merge proposals and push them back up to Launchpad.

• apport, a crash detection system that automatically generates reports with debugging information from crashed programs and provides UI frontends for handling these reports. Apport uses launchpadlib in a script (apport-collect) that adds apport data to existing bug reports.

• ubuntu-dev-tools, a collection of useful tools that Ubuntu developers use to make their packaging work a lot easier (bug filing, packaging preparation, package analysis, etc). Ubuntu Dev Tools uses launchpadlib in the manage-credentials, requestsync, massfile, lp-set-dup, grab-attachments, and hugdaylist scripts. They’re all sharing the ubuntutools/lp/libsupport.py library, which might be a good place to start.

• ubuntu-qa-tools, a collection of tools used by the Ubuntu QA Team (sort of parallel to ubuntu-dev-tools, but for QA). See the ml-fixes-report.py, ml-team-fixes-report.py, b-tool, package-bugs-gravity.py, team-reported-bug-tasks.py, and team-assigned-bug-tasks.py scripts.

• ilaunchpad-shell, an interactive shell to Launchpad.net.

See also the API category on the Launchpad Blog, which has posts with code examples, news, etc. And this thread on the Launchpad Users mailing list may accumulate more examples, even after this blog post goes up.

I’m in Montreal this week, with the rest of the Launchpad Foundations team and a few other Canonical engineers. We’re preparing for the open sourcing of Launchpad by doing a lot of refactoring, so that when the code is released in July it will be more comprehensible, and reusable in projects other than Launchpad.

In addition to our main body of Launchpad code we’ll be releasing a framework project called LAZR. Code that’s not Launchpad-related will go into LAZR or one of its subprojects: lazr.config, lazr.uri, lazr.batchnavigator, and so on. These will be little libraries that can be reused in other projects.

For instance: underneath the Launchpad web service is a generic library that lets you create a web service from your existing data objects by annotating the Zope interfaces. All this code lives in LAZR: the only web service code in Launchpad has to do with Launchpad’s web service specifically.  We’d like you to reuse the LAZR webservice code in any project that uses Zope interfaces to describe its data objects.

Unfortunately, there’s a lot of code in Launchpad that really should be in LAZR. If that wasn’t bad enough, the web service code in LAZR is full of dependencies on that Launchpad code. At the moment, you can’t run the LAZR web service code without bringing in all of Launchpad.

The object of this sprint is to refactor this code and move it all into LAZR packages. Our medium-term goal is a working example web service (code name “Sharks”) that uses no Launchpad code, but it won’t stop there. The more utilities we can move out of Launchpad’s monolithic code base, the more likely it is you’ll be able to find a bit of code you want to reuse.

The frequency of these posts has fallen off recently because I’ve been working on a Javascript client for the Launchpad web service, which we’ll be using to add some Ajax functionality to Launchpad. But it’s time for another one, featuring progress that’s visible to you.

First, an announcement: on Tuesday, the 20th of January, I’ll be doing an IRC chat on the Launchpad web service as part of Ubuntu Developer Week. See the schedule for details.

Newly published objects

The Bugs team has published CVEs through the web service. There’s a cves collection associated with every bug, and you can link and unlink CVEs from bugs.

The Soyuz team has published archives. You can use the web service to inspect a distribution archive or PPA, copy packages from one archive to another, and manage the permissions of who’s allowed to upload to an archive.

Merge proposals have also been published, but they’re read-only for now, so not very useful yet. But pretty soon you should be able to integrate them into an external code review tool.

Modify fields directly instead of through named operations

Consider the bug task object. Up to this point it’s had a number of read-only fields like ‘status’, ‘importance’, and ‘assignee’. These fields aren’t really read-only: you can modify them, but you have to know the secret. Each has a corresponding named operation: transitionToStatus, transitionToImportance, and transitionToAssignee. To modify ‘status’ you need to invoke transitionToStatus.

So you can’t modify a bug task’s status the way you can modify a bug’s description. This launchpadlib code works:

>>> bug.description = "A new description"
>>> bug.lp_save()

But this code doesn’t (except now, on staging):

>>> task.status = "New"
>>> task.lp_save()

You have to do this instead:

>>> task.transitionToStatus("New")

I’ve long felt that this transitionTo stuff is an internal detail you shouldn’t have to worry about, and judging from the bugs you’re filing, some of you agree. So I’ve made ‘status’, ‘importance’, and ‘assignee’ look like regular read-write fields.

The transitionTo methods are still available, so your old code will still work. In fact, the new code just calls the server-side transitionTo methods behind the scenes. The validation errors you used to get from passing a bad value into transitionToStatus will happen the same way if you set ‘status’ to a bad value.

There are other named operations that could be replaced by making the read-only field they modify into a read-write field, but I only changed those three bug task fields. I’ve talked to other Launchpad programmers about this new tool, and they’ll start using it according to their own schedules.

http_etag

Finally, there’s one part of the Javascript work that’s generally useful if you’re writing code against the web service on the level of HTTP requests rather than using launchpadlib. That’s the http_etag field now associated with entry objects such as people and bugs. It’s the same information provided in the HTTP header “ETag” when you get the entry object. So why send it again?

Well, imagine that you get a whole collection of bugtasks, and then decide you want to edit one of them. Ideally you’d use the value of “ETag” to make a conditional PUT or PATCH request, so that you wouldn’t accidentally overwrite someone else’s changes.

But you never got the ETag for that bugtask, because you got it as part of a collection. The http_etag field comes to the rescue here, allowing you to make a conditional PUT or PATCH without having to send a separate GET just to get the bugtask’s ETag.

Guillermo Gonzalez – the man behind the bzr-eclipse plugin – has recently been working with the Launchpad API to produce an Eclipse plugin that integrates with Launchpad.

That seemed like a pretty cool use of the API so I emailed him to find out more.

Matthew: What does your plugin do?

Guillermo: The user visible plugin allow the user to search the branches of a project. Basically it’s a view with a search field. This is going to become part of bzr-eclipse, as an extension, to allow searching for branches and branching into a new project directly from one of the results.

But actually it’s a set of plugins 😉

The heart of it is the launchpadlib plugin, it abstracts common features needed by others plugins that need to interact with launchpad, at this moment it provides authentication and access to projects and bugs.

Matthew: How are you accessing the API? Directly or using the Python library?

Guillermo: The first approach was to use the API via Java (and a java implementation of wadl), but as I was reinventing the wheel, I started to look on how to use launchpadlib from java. I’m currently using launchpadlib with an alpha version of Jython-2.5 and some extra patches/libraries missing in Jython and required by launchpadlib.

Matthew: Could you have done this without the Launchpad API?

Guillermo: Doing something like this never crossed my mind before knowning of the API and launchpadlib. Such a task would require screen scraping and all sorts of hacks to get things working … until something in the UI changes, and makes it useless. Also that would increase the load on Launchpad itself and maybe affect other users.

Matthew: How did you find both learning and using the API?

Guillermo: The API is straightforward to learn, also if you can use launchpadlib it’s far easier, just start the python interactive interpreter, import launchpadlib and start prototyping your app 🙂

Matthew: What do you like best about the API?

Guillermo: From my point of view the most important aspects are:

• it’s based on standards (wadl, http)
• there is a reference implementation in python (python is awesome to learn how things works)
• it’s well documented, and if you can’t find what you want it’s easy to check if it’s possible using launchpadlib and python interactive interpreter.

Matthew: What’s the worst thing about the API?

Guillermo: I don’t think there is a “worst thing”, the API is evolving quite well.

I’m pretty sure (and know that by experience), that if there is a missing feature that it’s required by a user, the ~launchpadlib-developers team would try to fix it. Obviously, it will require the proper bug report/feature request 😉

Matthew: What would you like to see changed or improved?

Guillermo: Back when I started to work on these plugins, there was no support for the code hosting bit, nor the possibility to search the bugs assigned on a specific project, but after reporting the issues the ~launchpadlib-developers team fixed them.

For more about Guillermo’s Launchpad plugin for Eclipse, see the bzr-eclipse project in Launchpad.

As part of the API beta, we’ve recently exposed a number of new features that work with the Launchpad registry. One outcome is the ability to simplify part of the release management that has been cumbersome.

You can now have your build system generate all of the files you wish to distribute, such as installers, change logs, README files, and upload them automatically using the API. Below is a recipe to perform the task. You can take this idea and customize it as you need.

import launchpadlib

FILE_TYPES = dict(tarball='Code Release Tarball',
                  readme='README File',
                  release_notes='Release Notes',
                  changelog='ChangeLog File',
                  installer='Installer file')

def upload_file(lp, project_name, release_version,
                filename, description,
                content_type, file_type,
                signature_filename=None):
    """Upload a file, and optionally its signature, to Launchpad.

    :param lp: Launchpad instance
    :param project_name: the project that owns the uploaded file
    :param release_version: the release the file will be associated, e.g. 1.0
    :param filename: the name of the file to be uploaded.  For this demo it is
                     the name that will be associated with the upload and the
                     file where the contents reside, in the current directory
    :param description: a short sentence describing the file like 'minimal
                        installer'
    :param content_type: the content MIME type, e.g. 'text/plain'
    :param file_type: one of the acceptable file types for upload as shown in
                      the FILE_TYPES dictionary.
    :param signature_filename: the optional name of the GPG signature file.
    """
    try:
        # Look up the project using the Launchpad instance.
        proj = lp.projects[project_name]
        # Find the release in the project's releases collection.
        release = None
        for rel in proj.releases:
            if rel.version == release_version:
                release = rel
                break
        assert release is not None, (
            "Release %s could not be found for project %s." % (release_version,
                                                               project_name))

        # Get the file contents.
        file_content = open(filename, 'r').read()
        # Get the signature, if available.
        if signature_filename is not None:
            signature_content = open(signature_filename, 'r').read()
        else:
            signature_content=None
        # Create a new product release file.
        product_release_file = release.add_file(filename=filename,
                                                description=description,
                                                file_content=file_content,
                                                content_type=content_type,
                                                file_type=file_type,
                                                signature_filename=signature_filename,
                                                signature_content=signature_content)
    except launchpadlib.errors.HTTPError, e:
        # Handle error here.
        print "An error happened in the upload."
        print e.content
        product_release_file = None

    return product_release_file is not None

Now that we’ve defined the routine upload_file you can call it as shown here. The handle lp is an instance returned from passing your credentials to launchpadlib.launchpad.Launchpad. This snippet of code shows how you would use the previous routine to upload your files and signatures.

    # We assume 'lp' is a valid Launchpad instance.
    # Information about the project and release.
    proj_name = 'acme-widgets'
    release='1.0'
    # Information about the file and signature.
    filename = 'README.txt'
    description = 'python installer script'
    content_type = 'text/plain'
    file_type = FILE_TYPES['readme']
    # The signature file is created by executing:
    # gpg --armor --sign --detach-sig <file>
    signature_filename = 'README.txt.asc'

    upload_file(lp, proj_name, release, filename, description, content_type,
                file_type, signature_filename)

Let me know if this recipe works out for you by commenting on this post. Also, we’d love to see snippets of code that you’ve found useful in working with the API. Post them here or email to feedback@launchpad.net

This status report was put off until today because the edge site was frozen. This update is huge. We’ve made one change to launchpadlib, exposed a whole lot of Launchpad’s project registry, and published branches for the first time.

Since the development cycle is over, the next update will probably be in two weeks.

(more…)

Barry Warsaw, Francis Lacoste, and I gave an IRC chat about the Launchpad web service as part of Ubuntu Developer Week. There’s a transcript available in case you missed it.

The big news from last week is that we finally have the API documentation being generated whenever a new version of Launchpad is deployed. Whatever server your script is running against, the /+apidoc URL will contain the reference documentation for that version of Launchpad. launchpad.net’s documentation will change when we do an official release of Launchpad, and edge.launchpad.net’s and staging.launchpad.net’s will be updated much more often, as our changes get pushed out to those servers.

But there’s more! I also added support to launchpadlib for access to the binary files hosted by Launchpad: mugshots, bug attachments, and so on. This is everything described in the reference doc as “a file resource.” If you upgrade launchpadlib, you can now access these files with a Pythonic file object interface.

The server side of this has been in place since we launched the web service, so in theory you could have been messing with uploaded files this whole time by writing a custom-built client. But only recently did I update the hacking document that makes it easy to see how to do this.

Finally, Edwin, Brad, and I have continued working on exporting the Launchpad registry. Of note are three new hosted-file resources: a project’s brand, logo, and icon. Project groups also have more of their fields published.

This week: more things published through the registry, hopefully some progress on bugs as well.

Last week the Launchpad web service didn’t change, because the source tree was frozen pending a Launchpad release. But this week we landed the branches we wrote last week, as well as some new ones.

The biggest change this week is client-side caching. Update your copy of launchpadlib to revision 17 and change your scripts to pass a directory name into the Launchpad constructor. That directory will be used as a cache.


>>> cachedir = "/home/me/.launchpadlib/cache/"
>>> launchpad = Launchpad(credentials, STAGING_SERVICE_ROOT, cachedir)


Without the cache, it takes several seconds just to start up launchpadlib. Every time you create a Launchpad object, the library has to download information about how Launchpad
works. With the cache in place, launchpadlib will download that information once and reuse it for every Launchpad object you create. The cache will also store the bugs, people, and other objects you retrieve as you use launchpadlib. The more often you use the same Launchpad objects, the more time the cache will save you. When we change Launchpad, launchpadlib will notice and will download the documents that have changed.

Second, Edwin Grubbs continued his work on publishing the Launchpad registry. launchpad.projects is now open for business. You can get basic information about Launchpad projects, their milestones and branches.

>>> bzr = launchpad.projects['bzr']
>>> bzr.title
u'Bazaar Version Control System'
>>> bzr.programming_language
u'Python'
>>> [milestone.name for milestone in bzr.all_milestones]
[u'0.9a', u'1.3.1rc1', u'1.4rc2', u'1.5', u'1.6', ...]


Finally, I’ve given launchpadlib some code that uses the client-side cache to avoid the “lost update problem.” This happens when two people modify the same object without coordinating, and the second one unknowingly overwrites the first person’s changes. If you use launchpadlib to make a change and it turns out someone else has changed the same object, you’ll get a chance to look at the new version of the object and see if your changes are still relevant.

One thing we haven’t done yet is gotten the API HTML reference to be regenerated whenever we change the web service. I planned to work on this yesterday, but instead spent the day fixing bugs. I’m working on it now and we should have it up early next week.

Next week we’ll see also more of the Launchpad registry exposed, and some long-awaited aspects of Launchpad’s bug tracker. And on Tuesday, Barry Warsaw and I will be talking about Launchpad’s web service API as part of Ubuntu Developer Week. That’s at 1900 UTC, in #ubuntu-classroom on irc.freenode.net.

Now that the Launchpad web service API has entered beta, I’ll be posting an update every week about the improvements we make to it. This way you’ll always know what’s going on, and you’ll see when something you want to do becomes possible.

Web service improvements

This week I made one major change to the web service itself. There’s now a special top-level resource that’s an alias to “you” on Launchpad. This is a nice convenience when you’re writing scripts for yourself, but it’s practically essential if other people are going to be running your program.

Here’s how to get a reference to the person who’s running the script, using launchpadlib, our Python interface to the web service.

>>> me = launchpad.me
>>> print me.name
leonardr


If you’re not using launchpadlib, GET the service root at https://api.staging.launchpad.net/beta/, and you’ll see the URL to this resource as ‘me_link’. When you GET that URL you’ll be redirected to your own user account.

launchpadlib improvements

I spent most of my time working on launchpadlib. Apart from some bugfixes and performance improvements that you won’t even notice, I made three big improvements.

Introspection

Previously, to see what capabilities a launchpadlib object had, you had to check the reference documentation. (That documentation is two weeks out of date, by the way; we’ll be fixing that next week.) Now you can just call dir() on an object, and all of its Launchpad-derived attributes and methods will show up in the list. If you only want to see the Launchpad attributes or methods and not all the internal launchpadlib stuff, you can check lp_attributes or lp_operations. This code shows what you can do with a launchpadlib person object:

>>> me.lp_attributes
['self_link', 'resource_type_link', 'longitude', ... 'homepage_content']

>>> me.lp_operations
['addMember', ... 'setLocation']


Slices

The second new feature is the ability to slice Launchpad collections as though they were Python lists. Here’s some code that gets the 10 most recently filed bugs in Launchpad.

>>> recent_bugs = launchpad.bugs[:10]
>>> [bug.id for bug in recent_bugs]
[258042, 258041, 258040, 258039, 258038, 258037, 258036, 258033, 258032, 258030]


Previously, the only way to do this was to iterate over launchpad.bugs and insert a break statement when you’d had enough, which was very awkward.

Loading from bookmarks

The third new feature is the ability to bookmark launchpadlib objects and go back to them later, the way you can bookmark web pages in your browser. Here’s launchpadlib code to acquire a bug.

>>> a_bug = launchpad.bugs[251497]
>>> print a_bug.title
Make it possible to instantiate a resource from a URL


I can play around with that bug all I want within a Python session, but if I exit the Python session the bug will disappear. If I want to get the bug back later, I’ll need to find it again from scratch. Unless I store the bug’s unique ID (also known as its URL).

>>> print str(a_bug)
https://api.staging.launchpad.net/beta/bugs/251497


At this point I can write that string to a file or database. Later on, a different process might come online and load the string back into memory. That process can get the bug object back by passing the bug’s URL into launchpad.load().

>>> a_bug = launchpad.load("https://api.staging.launchpad.net/beta/bugs/251497")
>>> print a_bug.title
Make it possible to instantiate a resource from a URL


Pretty simple stuff–people have been saving URLs and passing them around to each other for over fifteen years. The advantage of our web service’s design is that it gives you the same power in a scripted environment.

Upcoming work

Next week I plan to spend most of my time on behind-the-scenes performance improvements. You won’t notice anything if you use launchpadlib. If you’re writing your own client, you’ll know what I’m talking about when I say “ETag support.”

Meanwhile, Edwin Grubbs will be working to expose Launchpad’s project registry through the web service.

See you next week!