Three tips for faster launchpadlib api clients

Three tips from Leonard’s lightning talk in Prague about writing faster Launchpadlib API clients:

1. Use the latest launchpadlib. It gets faster from one release to the next. (The versions in the current Ubuntu release should be fine; otherwise run from the branch or the latest tarball.)

2. Profile:

    import httplib2
    httplib2.debuglevel = 1

will show each http request and response, so that you can see what’s taking time.

3. Fetch objects only once:

Don’t do this:

    if bug.person is not None:
        print bug.person.name

instead

    p = bug.person
    if p is not None:
        print p.name

In the first case, the client may fetch the Person object twice. (We may fix this in future.)

Tags: , , , ,

7 Responses to “Three tips for faster launchpadlib api clients”

  1. Markus Korn Says:

    Another trick is to use Uris instead of objects in method arguments, for example

    In [6]: timeit zeitgeist.searchTasks(assignee=str(launchpad._root_uri) + “~thekorn”)
    1 loops, best of 3: 337 ms per loop

    In [7]: timeit zeitgeist.searchTasks(assignee=launchpad.people["thekorn"])
    1 loops, best of 3: 537 ms per loop

    The api doc at [0] tells you how the uri of a certain object has to llok like.

    [0] https://launchpad.net/+apidoc/

  2. james Says:

    For a read-only script of mine trawling bugs, I saw a roughly 5-times speed increase by wrapping my bug-task objects in this below – a very blunt caching mechanism:

    class LPWrap:
        "Simple wrapper Cache-Proxy for LP objects"
        def __init__(self, lpObj):
            self.lpObj = lpObj
        def __getattr__(self, attr):
            result = getattr(self.lpObj, attr)
            if result.__class__.__name__ == "Entry":
                result = LPWrap(result)
            setattr(self, attr, result)
            return result
    

    [edited to fix formatting -- mbp]

  3. james Says:

    oh dear – indentation not preserved :(. the if … “Entry” only acts on the single succeeding line. I think everything else is relatively unambiguous

  4. drubin Says:

    “May fix”

    It should be the API’s responsibility to handle caching. Else you are coding around the core problem instead of fixing it. Passing references all around the place to try limit your http calls is silly. The API is supposed to be the interface to LP not just a dumb layer on top of http.

    It should at the very least be an option that is enabled by default. Since most people that use the API will be small time fish and the bigger fish would simple know the API well enough to turn off the caching and handle it internally.

    Just my 2cents.

  5. Martin Pool Says:

    @drubin I agree we should have at least a sensible minimal level of caching, and we need the API to at least take the concept of caching into account. I can imagine over-aggressive caching also being confusing to clients.

  6. Brian Murray Says:

    Latest version of launchpadlib is rather vague – talking with Leonard about this he really meant the version of launchpadlib in Maverick which is still under active development.

  7. Brian Murray Says:

    Working with Leonard I’ve included the performance improvement patches into a Lucid version of the package which can be found in my PPA – https://edge.launchpad.net/~brian-murray/+archive/ppa. This should also make its way into Lucid updates and you can follow bug 607947 regarding that.

Leave a Reply