launchpadblog

After a few weeks of development and testing, we are proud to finally announce that Git protocol v2 is available at Launchpad! But what are the improvements in the protocol itself, and how can you benefit from that?

The git v2 protocol was released a while ago, in May 2018, with the intent of simplifying git over HTTP transfer protocol, allowing extensibility of git capabilities, and reducing the network usage in some operations.

For the end user, the main clear benefit is the bandwidth reduction: in the previous version of the protocol, when one does a “git pull origin master”, for example, even if you have no new commits to fetch from the remote origin, git server would first “advertise” to the client all refs (branches and tags) available. In big repositories with hundreds or thousands of refs, this simple handshake operation could consume a lot of bandwidth and time to communicate a bunch of data that would potentially be discarded by the client after.

In the v2 protocol, this waste is no longer present: the client now has the ability to filter which refs it wants to know about before the server starts advertising it.

The v2 protocol is not the default on git clients yet, but if you are using a git version higher than 2.19, you can use v2: simply run git config --global protocol.version 2, and you will be using the most recent protocol version when communicating with servers that support this version. Including Launchpad, of course.

And even if you have repositories hosted in a server that is not yet compatible with v2, don’t worry: the git client is backward compatible. If the server does not support v2, git client should fall back gracefully to the previous version and everything should continue to work as expected. We hope you enjoy the new feature. And let us know if you have any feedback!

We’ve had Git hosting support in Launchpad for a few years now. One thing that some users asked for, particularly larger users such as the Ubuntu kernel team, was the ability to set up per-branch push permissions for their repositories. Today we rolled out the last piece of this work.

Launchpad’s default behaviour is that repository owners may push anything to their own repositories, including creating new branches, force-pushing (rewriting history), and deleting branches, while nobody else may push anything. Repository owners can now also choose to protect branches or tags, either individually or using wildcard rules. If a branch is protected, then by default repository owners can only create or push it but cannot force-push or delete; if a tag is protected, then by default repository owners can create it but cannot move or delete it.

You can also allow selected contributors to push to protected branches or tags, so if you’re collaborating with somebody on a branch and just want to be able to quickly pair-program via git push, or you want a merge robot to be able to land merge proposals in your repository without having to add it to the team that owns the repository and thus give it privileges it doesn’t need, then this feature may be for you.

There’s some initial documentation on our help site, and here’s a screenshot of a repository that’s been set up to give a contributor push access to a single branch:

Launchpad has had Git-to-Bazaar code imports since 2009, along with imports from a few other systems.  These form part of Launchpad’s original mission to keep track of free software, regardless of where it’s hosted.  They’re also very useful for automatically building other artifacts, such as source package recipes or snap packages, from code hosted elsewhere.  Unfortunately they’re quite complicated: they need to be able to do a full round-trip conversion of every revision from the other version control system, which has made it difficult to add support for Git features such as signed commits or submodules.  Once one of these features is present anywhere in the history of a branch, importing it to Bazaar becomes impossible.  This has been a headache for many users.

We can do better nowadays.  As of last year, we have direct Git hosting support in Launchpad, and we can already build snaps and recipes straight from Git, so we can fulfil our basic goal more robustly now with a lot less code.  So, Launchpad now supports Git-to-Git code imports, also known as Git mirroring.  You can use this to replace many uses of Git-to-Bazaar imports (note that there’s no translations integration yet, and of course you won’t be able to branch the resulting import using bzr).

See our Git documentation for more details.

We just rolled out a new feature for Launchpad’s Git repository hosting: Git-based merge proposals can now be linked to Launchpad bugs.  This can be done manually from the web UI for the merge proposal, but normally you should just mention the Launchpad bug in the commit message of one of the commits you want to merge.  The required commit message text to link to bugs #XXX and #YYY looks like this:

LP: #XXX, #YYY

This is the same form used for Launchpad bug references in debian/changelog files in source packages, and the general approach of mentioning bugs in commit messages is similar to that of various other hosting sites.

Bugs are not automatically closed when merge proposals land, because the policy for when that should happen varies from project to project: for example, projects often only close bugs when they make releases, or when their code is deployed to production sites.

Users familiar with Bazaar on Launchpad should note that the model for Git bug linking is slightly different: bugs are linked to merge proposals rather than to individual branches.  This difference is mainly because individual branches within a Git repository are often much more ephemeral than Bazaar branches.

Documentation is here, along with other details of Launchpad’s Git hosting.

If you are a member of Launchpad’s beta testers team, you can now try out webhooks for Bazaar branches and Git repositories. These can be used to set up integration with external sites for various purposes, such as running CI jobs or publishing documentation. We expect to open this up to all Launchpad users soon, but in the meantime please do file a bug against Launchpad itself if you encounter any problems.

See our webhooks documentation for more details.

Update: as of 2015-11-20, this feature is enabled for all Launchpad users.

Launchpad’s bug and branch privacy features are being replaced by information sharing that permits project maintainers to share kinds of information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

Maintainers can share and unshare their project with people

Project maintainers and drivers can see the “Sharing” link on their project’s front page. The page lists every user and team that the project shares with. During the transition period of the beta, you might see many users with “Some” access to “Private Security” or “Private” user information. They have this access because they are subscribed to bugs and branches. Maintainers can unshare with users who do not need access to any confidential information, or just unshare a bug or branch with a user. Maintainers can share share with a team to give them full access to one or more kinds of confidential information.

I have prepared a video that demonstrates the features (my apologies for the flickering)

Commercial projects can set bug and branch policies

Projects with commercial subscriptions can also change bug and branch sharing policies to set the default information type of a bug or branch, and control what types they may be changed to. Maintainers can set policies that ensure that bugs and branches are proprietary, and only proprietary, to ensure confidential information is never disclosed.

Sharing can be managed using API scripts

I maintain many project which have a lot of private bugs and branches. The sharing page lists a lot of people, too many to read quickly. I know most work for my organisation, but I don’t even know everyone in my organisation. So I wrote a Launchpad API script that can be run by any project maintainer to share the project with a team, then unshare with the team members. The members still have access to the bugs and branches and their subscriptions still work, but they will lose access to my project when they leave the team. This arrangement makes it very easy to manage who has access to my projects. share-projects-with-team.py is run with the name of the team and a list of projects to share with it.

./share-projects-with-team.py my-team project1 project2

Project maintainers can now see all the private bugs in their project. While Launchpad tried to ensure the proper people could see private bugs in the past, the old subscription mechanism was brittle. Users could unsubscribe themselves and lose access, or retarget a bug to another projects which does not update bug subscriptions. The Purple squad migrated project configurations to project sharing so that all private information was shared with project maintainers. Project sharing ensures that confidential information is disclosed to the proper people.

If you are a project maintainer, you might be surprised to find old private bugs that you have never seen before. This happened to me. Some ancient private bugs were in the “New” listing of bugs, other were buried in search results. You can search for just private bugs to review all private bugs.

Privacy terminology is restored

We reverted the information type terminology changes introduced a few months ago.

• User data ➙ Private
• Embargoed Security ➙ Private Security
• Unembargoed Security ➙ Public Security

While the jargon-laden terms helped the small number of people who work with confidential information, the people who report bugs were confused. The most common reason for unwanted disclosure is that people enter confidential information, and cannot see how to make it private. Sometimes a user may not notice the mistake until a few minutes later. We also revised the descriptions of the information types to help new users quickly select the correct information type.

Nothing breaks my heart quite like a request to make a project private–make it invisible to everyone except to the people the project trusts. I am utterly crushed when someone who works for Canonical or on Launchpad asks for one. I have been planning this feature for more than two years, and the Purple squad has been working on it for 13 months. I blog about this, I send emails about this, I present reports on this, but the people who most need private projects don’t know what the Purple squad is doing. I think the problem here is that Launchpad squads no longer use Launchpad to plan and execute work. There is no place for any interested party to see what the goals of Disclosure is and gauge how we are progressing.

I present my first draft of a report that states the simple goals of that the Disclosure feature wants to achieve . The report provides some summaries of the work that allows anyone to see what the Purple squad is doing, recently done, and will do next. There is also some analysis that provides insight into the amount of work remaining. This report complements the Purple squad’s kanban board. While kanban is excellent for tracking branches of code and technical tasks, the level of detail is unsuitable for non-Launchpad developers. The kanban board is also only accessible to a small number of people. I want a report that anyone interested in private projects or managing the disclosure of private information can see and understand. Mostly, I want everyone to see that the Purple squad is delivering valuable features and know when we will be done.

I based the report on the intended reporting UI for Launchpad series and milestones. I really miss using series and milestones to plan releases. For every milestone, I wrote our goals in the summary, and targeted bugs to the milestone. Though we abandoned the analytics because of performance concerns, I could reliably judge  the contributors’ velocity, and see when I needed to retarget work to another milestone because the remaining effort exceeded the milestone’s work capacity. Though I didn’t provide a burn down chart of the work, I could sort the milestone to see the colour change. I could confidently see and predict 3 months of work.

This report replaces the canvas-based chart I planned for series and milestones with a YUI 3 chart. The listing of bugs are split into categories so that I can focus on scheduling or provide Diogo with a list of bugs that need exploratory testing. Though this report thinks it is talking to Launchpad, it is actually using JSON for the 500+ bugs that I pulled using a trivial Launchpad API script. Since the data is cheap to retrieve, I can load the chart multiple times, each looking a different set of bug tags so that I can see specific themes of work.

The report shows that there is more than 60 days of work to complete the features needed by private projects. The Orange squad will work on private projects while the Purple squad finishes the prerequisites.

We want the project maintainer to be the default party that the project shares private information with. The problem at the moment is that Launchpad does not know how to set a team as the project maintainer during setup. Improper project setup is the root cause of most cases where information is disclosed to the wrong people. We need to improve project registration and setup to ensure users can ensure private information is managed properly. This issue is complicated by a very old issue, it is not possible to register a team at the moment you discover you need one. Launchpad must let me register a new team that will maintain my project when I first setup my project.

The Purple Squad discussed what we can do to simplify team registration and perform the registration in any page that allows you to set a team. We discovered several areas where we can make improvements.

• Do not ask for non-essential information like contact address.
• We can simplify the team membership policy language.
• We can fix the confusion about membership renewal.
• Launchpad can pre-fill the form with sensible defaults when the team will be used in a role.

Ian put together a demonstration to prove we could extend the person picker to also permit you to register a team.

When you want to set the project maintainer to a new team, Launchpad will ask you to confirm its suggestions for the Launchpad Id, display name, and membership policy. You can change the values, but most of the time you will choose to continue, and Launchpad will register the team and place it in the role.

All users can now see the information type section that replaces the privacy and security section shown on bug and branch pages. This change allows users to clearly state the kind of information a bug or branch contains. Launchpad will soon permit project maintainers to share information types instead of managing individual bug and branch subscriptions. Project maintainers can see a link on their project’s front page to the  “Sharing” page. Sharing lists all the users and teams their project shares some private bugs and branches with. This list might be surprising. Launchpad Beta testers will soon be able share and unshare kinds of information to simplify management of whom the project discloses private information to.

See Reimaging the nature of privacy in Launchpad for more details.