launchpadblog

Launchpad’s bug and branch privacy features are being replaced by information sharing that permits project maintainers to share kinds of information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

Maintainers can share and unshare their project with people

Project maintainers and drivers can see the “Sharing” link on their project’s front page. The page lists every user and team that the project shares with. During the transition period of the beta, you might see many users with “Some” access to “Private Security” or “Private” user information. They have this access because they are subscribed to bugs and branches. Maintainers can unshare with users who do not need access to any confidential information, or just unshare a bug or branch with a user. Maintainers can share share with a team to give them full access to one or more kinds of confidential information.

I have prepared a video that demonstrates the features (my apologies for the flickering)

Commercial projects can set bug and branch policies

Projects with commercial subscriptions can also change bug and branch sharing policies to set the default information type of a bug or branch, and control what types they may be changed to. Maintainers can set policies that ensure that bugs and branches are proprietary, and only proprietary, to ensure confidential information is never disclosed.

Sharing can be managed using API scripts

I maintain many project which have a lot of private bugs and branches. The sharing page lists a lot of people, too many to read quickly. I know most work for my organisation, but I don’t even know everyone in my organisation. So I wrote a Launchpad API script that can be run by any project maintainer to share the project with a team, then unshare with the team members. The members still have access to the bugs and branches and their subscriptions still work, but they will lose access to my project when they leave the team. This arrangement makes it very easy to manage who has access to my projects. share-projects-with-team.py is run with the name of the team and a list of projects to share with it.

./share-projects-with-team.py my-team project1 project2

The Ubuntu Foundations team has sponsored work on various improvements to Launchpad’s archive handling lately, mainly to expose various new facilities on the API where we were previously using privileged scripts.  This has involved cleaning up a substantial amount of old code along the way, and it has become possible to fix some other old bugs as spin-offs.

One of these old bugs is “Archive:+copy-packages nearly unusable due to timeouts”.  The +copy-packages page allows anyone who can upload to a PPA to instead copy packages from another PPA.  This saves effort, and in the “Copy existing binaries” mode it can save a substantial amount of build time as well.  For example, the LibreOffice packaging team uses this to deliver packages to different sets of users after they have passed various levels of testing.

Unfortunately, the very cases where this is most useful, namely large and complex packages, are also the cases where it is most likely to break.  Copying large numbers of binary packages involves large numbers of database queries and can quite easily overrun the timeout for a single request to the Launchpad web application.  Doing this for several series at once, a common case which seems reasonable, is proportionally less likely to work.  Various attempts have been made to optimise the database interactions here, but ultimately doing lots of complex synchronous work in time for a single web request is doomed to failure.

The solution to all this is to copy packages asynchronously.  For some time Launchpad has had the ability to schedule “package copy jobs” which run very shortly after the request (typically within a minute) but not immediately.  For example, the Ubuntu team uses these when copying new versions of packages from Debian unstable in cases where there are no Ubuntu-specific modifications, and when releasing proposed updates to stable releases for general use after verification.  A similar facility has been present in the code for the +copy-packages page for some time, but not exposed due to various bugs.  We believe that these bugs have been fixed now, and so we would like to start copying packages asynchronously when requested via the web UI.

We have exposed this to beta testers first.  The effect is that, if you are a beta tester when you ask for packages to be copied, you will be told something like “Requested sync of 2 packages.  Please allow some time for these to be processed.”  The processing should normally happen within a minute or two, and you will be able to see it in progress on the +packages page for the target archive.  If it succeeds, the in-progress notification will be removed and you will be able to see the changes in the target archive.  Otherwise, you will see a failure notification along these lines:

If beta-testing goes well, then we will enable this for all users, and remove the old synchronous copying code shortly afterwards; so please do report any problems you see.

If you are relying on package copies in the web UI happening immediately rather than within a few minutes, firstly, please contact us (e.g. #launchpad-dev on freenode IRC, or launchpad-users@lists.launchpad.net) as we would like to understand your requirements in more detail; secondly, you may be able to use the Archive.syncSource API method instead, which also has timeout constraints but is at least guaranteed to remain synchronous.  However, we hope that most people will not have such a requirement.

Launchpad has a web UI, an email interface, and a ReST API that exposes every object in the database.

There are also a bunch of client programs, command line and graphical, that talk to Launchpad to do various things.

What we don’t yet have, and what I think would be great, is a systematic client that lets you manipulate
everything from the command line. There’s some code that starts towards this in Hydrazine, lptools and others, but I think having just a single tool that eventually does everything would be more discoverable and avoid unnecessary fragmentation or duplication.

(That’s not to say there’s not room for others that are guis, that are specialized to particular projects or that encapsulate a lot of policy or opinion about what they’re doing.)

So dobey and I have agreed to gradually merge hydrazine into lptools, and with other people to work towards making lptools cover everything you can do through the web UI or the API. If you have scripts you’ve written yourself, perhaps you’d like to merge them in.

Launchpad has a REST API that exposes almost every object within Launchpad. Most of them have API URLs that closely match the human-readable URL: for instance, https://bugs.launchpad.net/bugs/316694 can also be obtained in machine-oriented JSON or XML form from http://api.launchpad.net/1.0/bugs/316694. (The “1.0” in that URL means this is in the 1.0 API namespace.)

Previously, many Launchpadlib clients used string transformation to get from the API URL to something they could show a human in a web browser.

We’ve just added a web_link property on all Launchpad objects, so client applications can (and should) now just use that instead. Because this is just sent in the object representation you don’t need to upgrade launchpadlib to see it.

Three tips from Leonard’s lightning talk in Prague about writing faster Launchpadlib API clients:

1. Use the latest launchpadlib. It gets faster from one release to the next. (The versions in the current Ubuntu release should be fine; otherwise run from the branch or the latest tarball.)

2. Profile:

    import httplib2
    httplib2.debuglevel = 1

will show each http request and response, so that you can see what’s taking time.

3. Fetch objects only once:

Don’t do this:

    if bug.person is not None:
        print bug.person.name

instead

    p = bug.person
    if p is not None:
        print p.name

In the first case, the client may fetch the Person object twice. (We may fix this in future.)

You asked for it and we’ve delivered. Your launchpadlib scripts can now get read-only access to the Launchpad web service API without going through any authorization process. Previously, to authorize your script, you had to open a web page in the end-user’s web browser and get the user to click a button. Now, you only have to go through that process if you want to access someone’s private data or modify the Launchpad dataset on their behalf.

If you’re using the latest version of launchpadlib (1.5.4), you can log in anonymously with the login_anonymously helper method. You just have to pass in a string identifying your client, and the name of the Launchpad server you want to access.

from launchpadlib.launchpad import Launchpad
launchpad = Launchpad.login_anonymously("my client", "edge")
print launchpad.bugs[1].title
# Microsoft has a majority market share'

(That code also illustrates how the latest version of Launchpadlib lets you specify a short string like “edge” instead of having to importing a constant like EDGE_SERVICE_ROOT.)

If you’re using an older version of launchpadlib, you can log in anonymously using the login helper method. You can get anonymous access by passing in empty strings for the token string and access secret:

from launchpadlib.launchpad import Launchpad, EDGE_SERVICE_ROOT
launchpad = Launchpad.login("my client", "", "", EDGE_SERVICE_ROOT)
print launchpad.bugs[1].title
# Microsoft has a majority market share'

Right now, anonymous access only works on Launchpad’s ‘edge’ and ‘staging’ instances. It’ll be installed on production at the end of the month, with the release of Launchpad 10.01.