launchpadblog

The last time I posted to this blog I was describing a sprint in Montreal, where we were working on refactoring the web service code and moving it from Launchpad to a standalone library. Several weeks of work later, we’re done. You can now create your own web service using the same techniques we use in Launchpad.

The library is lazr.restful, and there’s a companion client library called lazr.restfulclient. They’re also on PyPI (restful, restfulclient). The place to discuss these libraries is the lazr-users mailing list.

I’ll show you lazr.restful first. If you just want to use the package, and you have easy_install, try “easy_install lazr.restful“. You can also download the tarball and use “python setup.py install“ on it. Or you can get check out the source code:

$ bzr branch lp:lazr.restful mybranch
$ cd mybranch
$ python bootstrap
$ python bin/buildout
$ python bin/test

The test command starts up the web service defined in src/lazr/restful/example and puts it through its paces with simulated HTTP requests. The example web service shows how to use Python decorators to turn Zope interfaces into a web service with the same features as Launchpad’s web service. Just as an example, here’s what a normal Zope interface might look like:

class IPerson(Interface):
    first_name = TextLine(title=u"First name", required=True)
    last_name = TextLine(title=u"Last name", required=True)

Here’s what it would look like with lazr.restful annotations:

class IPerson(Interface):
    export_as_webservice_entry(plural_name='people')
    first_name = exported(TextLine(title=u"First name", required=True))
    last_name = exported(TextLine(title=u"Last name", required=True))

lazr.restful will inspect that interface and know that you’re defining an ‘entry’ type resource, with each entry containing two bits of data. It’ll publish a WADL file describing that interface. Clients will be able to make GET, PUT, and PATCH requests to the ‘person’ resources to manipulate your application data.

lazr.restful provides annotations for every web service feature you see in Launchpad: entries, collections, hosted files, named operations, and so on. It even has some features that aren’t used by Launchpad yet, like the ability to make entries respond to DELETE requests.

lazr.restfulclient is the client side. We took almost all the code out of launchpadlib and made a generic client that will run against any lazr.restful service.

The same bootstrap–buildout–test cycle that worked for lazr.restful also works for lazr.restfulclient. The test command will start up the web service from lazr.restful, and manipulate it with Pythonic launchpadlib-like commands instead of fake HTTP requests. Given a ‘person’ resource like the one defined in the example above, you could run this code in lazr.restfulclient.

person_resource.first_name = "new first name"
person_resource.lp_save()

Before long we’ll be removing almost all the code from launchpadlib leaving only the OAuth handling and some Launchpad-specific information. We’re also working to make it easier to use lazr.restful with standard WSGI applications, so you don’t have to know as much Zope.

I’m in Montreal this week, with the rest of the Launchpad Foundations team and a few other Canonical engineers. We’re preparing for the open sourcing of Launchpad by doing a lot of refactoring, so that when the code is released in July it will be more comprehensible, and reusable in projects other than Launchpad.

In addition to our main body of Launchpad code we’ll be releasing a framework project called LAZR. Code that’s not Launchpad-related will go into LAZR or one of its subprojects: lazr.config, lazr.uri, lazr.batchnavigator, and so on. These will be little libraries that can be reused in other projects.

For instance: underneath the Launchpad web service is a generic library that lets you create a web service from your existing data objects by annotating the Zope interfaces. All this code lives in LAZR: the only web service code in Launchpad has to do with Launchpad’s web service specifically.  We’d like you to reuse the LAZR webservice code in any project that uses Zope interfaces to describe its data objects.

Unfortunately, there’s a lot of code in Launchpad that really should be in LAZR. If that wasn’t bad enough, the web service code in LAZR is full of dependencies on that Launchpad code. At the moment, you can’t run the LAZR web service code without bringing in all of Launchpad.

The object of this sprint is to refactor this code and move it all into LAZR packages. Our medium-term goal is a working example web service (code name “Sharks”) that uses no Launchpad code, but it won’t stop there. The more utilities we can move out of Launchpad’s monolithic code base, the more likely it is you’ll be able to find a bit of code you want to reuse.

The frequency of these posts has fallen off recently because I’ve been working on a Javascript client for the Launchpad web service, which we’ll be using to add some Ajax functionality to Launchpad. But it’s time for another one, featuring progress that’s visible to you.

First, an announcement: on Tuesday, the 20th of January, I’ll be doing an IRC chat on the Launchpad web service as part of Ubuntu Developer Week. See the schedule for details.

Newly published objects

The Bugs team has published CVEs through the web service. There’s a cves collection associated with every bug, and you can link and unlink CVEs from bugs.

The Soyuz team has published archives. You can use the web service to inspect a distribution archive or PPA, copy packages from one archive to another, and manage the permissions of who’s allowed to upload to an archive.

Merge proposals have also been published, but they’re read-only for now, so not very useful yet. But pretty soon you should be able to integrate them into an external code review tool.

Modify fields directly instead of through named operations

Consider the bug task object. Up to this point it’s had a number of read-only fields like ‘status’, ‘importance’, and ‘assignee’. These fields aren’t really read-only: you can modify them, but you have to know the secret. Each has a corresponding named operation: transitionToStatus, transitionToImportance, and transitionToAssignee. To modify ‘status’ you need to invoke transitionToStatus.

So you can’t modify a bug task’s status the way you can modify a bug’s description. This launchpadlib code works:

>>> bug.description = "A new description"
>>> bug.lp_save()

But this code doesn’t (except now, on staging):

>>> task.status = "New"
>>> task.lp_save()

You have to do this instead:

>>> task.transitionToStatus("New")

I’ve long felt that this transitionTo stuff is an internal detail you shouldn’t have to worry about, and judging from the bugs you’re filing, some of you agree. So I’ve made ‘status’, ‘importance’, and ‘assignee’ look like regular read-write fields.

The transitionTo methods are still available, so your old code will still work. In fact, the new code just calls the server-side transitionTo methods behind the scenes. The validation errors you used to get from passing a bad value into transitionToStatus will happen the same way if you set ‘status’ to a bad value.

There are other named operations that could be replaced by making the read-only field they modify into a read-write field, but I only changed those three bug task fields. I’ve talked to other Launchpad programmers about this new tool, and they’ll start using it according to their own schedules.

http_etag

Finally, there’s one part of the Javascript work that’s generally useful if you’re writing code against the web service on the level of HTTP requests rather than using launchpadlib. That’s the http_etag field now associated with entry objects such as people and bugs. It’s the same information provided in the HTTP header “ETag” when you get the entry object. So why send it again?

Well, imagine that you get a whole collection of bugtasks, and then decide you want to edit one of them. Ideally you’d use the value of “ETag” to make a conditional PUT or PATCH request, so that you wouldn’t accidentally overwrite someone else’s changes.

But you never got the ETag for that bugtask, because you got it as part of a collection. The http_etag field comes to the rescue here, allowing you to make a conditional PUT or PATCH without having to send a separate GET just to get the bugtask’s ETag.

You didn’t expect it, but here it is. “This Week in Launchpad’s Web API” returns for one more week! This week we’re pleased to announce that you can now search the bug tasks associated with a project, project group, distribution, or milestone. We know a lot of people have been waiting for this feature.

>>> launchpadlib_project = launchpad.projects['launchpadlib']
>>> me = launchpad.me
>>> my_launchpadlib_bugs = launchpadlib_project.searchTasks(assignee=me)


The interface to searchTasks is very similar to the interface to Launchpad’s advanced search form. That fact should get you started searching easily.

The other bug news is that you can now post a comment to a bug by invoking newMessage on the bug.

Release files are now exposed through the web service. This means you can now integrate release management tools into Launchpad. When you do a release of your program, the same tool that packages the release can upload the release file to Launchpad and make a Launchpad release for it.

>>> series = launchpad.projects['myproject'].series[0]
>>> release = series.addRelease(
...     version=new_item_name, code_name='sumo',
...     summary='super new beta',
...     description='The be-all end-all version for the next century.',
...     changelog='Fixes security bug. Adds external support.')
>>> release.add_file(filename="release.tgz",
...     file_content="[binary data]",
...     content_type="application/x-gzip")


Finally, we’ve changed launchpadlib so that when you upload a file, you need to set the filename you want the file to have on the server side. Previously, there was no way to set the filename.

>>> release_file = release.files[0]
>>> filehandle = release_file.open("application/x-gzip", "new-filename.tgz", "w")
>>> ...


This status report was put off until today because the edge site was frozen. This update is huge. We’ve made one change to launchpadlib, exposed a whole lot of Launchpad’s project registry, and published branches for the first time.

Since the development cycle is over, the next update will probably be in two weeks.

(more…)

Barry Warsaw, Francis Lacoste, and I gave an IRC chat about the Launchpad web service as part of Ubuntu Developer Week. There’s a transcript available in case you missed it.

The big news from last week is that we finally have the API documentation being generated whenever a new version of Launchpad is deployed. Whatever server your script is running against, the /+apidoc URL will contain the reference documentation for that version of Launchpad. launchpad.net’s documentation will change when we do an official release of Launchpad, and edge.launchpad.net’s and staging.launchpad.net’s will be updated much more often, as our changes get pushed out to those servers.

But there’s more! I also added support to launchpadlib for access to the binary files hosted by Launchpad: mugshots, bug attachments, and so on. This is everything described in the reference doc as “a file resource.” If you upgrade launchpadlib, you can now access these files with a Pythonic file object interface.

The server side of this has been in place since we launched the web service, so in theory you could have been messing with uploaded files this whole time by writing a custom-built client. But only recently did I update the hacking document that makes it easy to see how to do this.

Finally, Edwin, Brad, and I have continued working on exporting the Launchpad registry. Of note are three new hosted-file resources: a project’s brand, logo, and icon. Project groups also have more of their fields published.

This week: more things published through the registry, hopefully some progress on bugs as well.

Last week the Launchpad web service didn’t change, because the source tree was frozen pending a Launchpad release. But this week we landed the branches we wrote last week, as well as some new ones.

The biggest change this week is client-side caching. Update your copy of launchpadlib to revision 17 and change your scripts to pass a directory name into the Launchpad constructor. That directory will be used as a cache.


>>> cachedir = "/home/me/.launchpadlib/cache/"
>>> launchpad = Launchpad(credentials, STAGING_SERVICE_ROOT, cachedir)


Without the cache, it takes several seconds just to start up launchpadlib. Every time you create a Launchpad object, the library has to download information about how Launchpad
works. With the cache in place, launchpadlib will download that information once and reuse it for every Launchpad object you create. The cache will also store the bugs, people, and other objects you retrieve as you use launchpadlib. The more often you use the same Launchpad objects, the more time the cache will save you. When we change Launchpad, launchpadlib will notice and will download the documents that have changed.

Second, Edwin Grubbs continued his work on publishing the Launchpad registry. launchpad.projects is now open for business. You can get basic information about Launchpad projects, their milestones and branches.

>>> bzr = launchpad.projects['bzr']
>>> bzr.title
u'Bazaar Version Control System'
>>> bzr.programming_language
u'Python'
>>> [milestone.name for milestone in bzr.all_milestones]
[u'0.9a', u'1.3.1rc1', u'1.4rc2', u'1.5', u'1.6', ...]


Finally, I’ve given launchpadlib some code that uses the client-side cache to avoid the “lost update problem.” This happens when two people modify the same object without coordinating, and the second one unknowingly overwrites the first person’s changes. If you use launchpadlib to make a change and it turns out someone else has changed the same object, you’ll get a chance to look at the new version of the object and see if your changes are still relevant.

One thing we haven’t done yet is gotten the API HTML reference to be regenerated whenever we change the web service. I planned to work on this yesterday, but instead spent the day fixing bugs. I’m working on it now and we should have it up early next week.

Next week we’ll see also more of the Launchpad registry exposed, and some long-awaited aspects of Launchpad’s bug tracker. And on Tuesday, Barry Warsaw and I will be talking about Launchpad’s web service API as part of Ubuntu Developer Week. That’s at 1900 UTC, in #ubuntu-classroom on irc.freenode.net.

Now that the Launchpad web service API has entered beta, I’ll be posting an update every week about the improvements we make to it. This way you’ll always know what’s going on, and you’ll see when something you want to do becomes possible.

Web service improvements

This week I made one major change to the web service itself. There’s now a special top-level resource that’s an alias to “you” on Launchpad. This is a nice convenience when you’re writing scripts for yourself, but it’s practically essential if other people are going to be running your program.

Here’s how to get a reference to the person who’s running the script, using launchpadlib, our Python interface to the web service.

>>> me = launchpad.me
>>> print me.name
leonardr


If you’re not using launchpadlib, GET the service root at https://api.staging.launchpad.net/beta/, and you’ll see the URL to this resource as ‘me_link’. When you GET that URL you’ll be redirected to your own user account.

launchpadlib improvements

I spent most of my time working on launchpadlib. Apart from some bugfixes and performance improvements that you won’t even notice, I made three big improvements.

Introspection

Previously, to see what capabilities a launchpadlib object had, you had to check the reference documentation. (That documentation is two weeks out of date, by the way; we’ll be fixing that next week.) Now you can just call dir() on an object, and all of its Launchpad-derived attributes and methods will show up in the list. If you only want to see the Launchpad attributes or methods and not all the internal launchpadlib stuff, you can check lp_attributes or lp_operations. This code shows what you can do with a launchpadlib person object:

>>> me.lp_attributes
['self_link', 'resource_type_link', 'longitude', ... 'homepage_content']

>>> me.lp_operations
['addMember', ... 'setLocation']


Slices

The second new feature is the ability to slice Launchpad collections as though they were Python lists. Here’s some code that gets the 10 most recently filed bugs in Launchpad.

>>> recent_bugs = launchpad.bugs[:10]
>>> [bug.id for bug in recent_bugs]
[258042, 258041, 258040, 258039, 258038, 258037, 258036, 258033, 258032, 258030]


Previously, the only way to do this was to iterate over launchpad.bugs and insert a break statement when you’d had enough, which was very awkward.

Loading from bookmarks

The third new feature is the ability to bookmark launchpadlib objects and go back to them later, the way you can bookmark web pages in your browser. Here’s launchpadlib code to acquire a bug.

>>> a_bug = launchpad.bugs[251497]
>>> print a_bug.title
Make it possible to instantiate a resource from a URL


I can play around with that bug all I want within a Python session, but if I exit the Python session the bug will disappear. If I want to get the bug back later, I’ll need to find it again from scratch. Unless I store the bug’s unique ID (also known as its URL).

>>> print str(a_bug)
https://api.staging.launchpad.net/beta/bugs/251497


At this point I can write that string to a file or database. Later on, a different process might come online and load the string back into memory. That process can get the bug object back by passing the bug’s URL into launchpad.load().

>>> a_bug = launchpad.load("https://api.staging.launchpad.net/beta/bugs/251497")
>>> print a_bug.title
Make it possible to instantiate a resource from a URL


Pretty simple stuff–people have been saving URLs and passing them around to each other for over fifteen years. The advantage of our web service’s design is that it gives you the same power in a scripted environment.

Upcoming work

Next week I plan to spend most of my time on behind-the-scenes performance improvements. You won’t notice anything if you use launchpadlib. If you’re writing your own client, you’ll know what I’m talking about when I say “ETag support.”

Meanwhile, Edwin Grubbs will be working to expose Launchpad’s project registry through the web service.

See you next week!

If you’ve been wanting to integrate Launchpad into your development tools, or create scripts that read and write Launchpad’s dataset, your wait is over. We’ve released the initial version of our RESTful web service API to the beta testing team. Now you can integrate with Launchpad using our Python library or by making simple HTTP requests.

No longer will you need to screen-scrape or write scripts that pretend to be a web browser; your programs will be able to communicate directly with Launchpad. Our support for OAuth means your users can delegate a subset of their Launchpad privileges to your program or website, without handing over their Launchpad pasword.

Right now, the web service provides basic access to Launchpad’s people and bugs. We’re working now to expose more of Launchpad’s data–projects, milestones, and so on–and to improve the usability of the Python client. Once we’ve stabilized the framework, we’ll open up the web service API to everyone. In the meantime, you can try it out by joining the Launchpad Beta Testers team. Once you’re on the team, you can get started by visiting the help homepage.

Our Python library, launchpadlib, works as a “web service browser”. You can use it to navigate the web service from a Python script, the way you surf the Launchpad website from your web browser. You don’t need any special knowledge of web services to use launchpadlib; you can write natural-looking code like this:


>>> me = launchpad.people['my-username']
>>> me.display_name = 'My new display name'
>>> me.lp_save()


We’ll be improving launchpadlib and the web service simultaneously; this is just the beginning.