How we’re open sourcing Launchpad.
Recently we announced that we’re open sourcing Launchpad, with a self-imposed deadline of 21 July 2009. (22 June 2009 ed. note: we may push the date a bit, for testing purposes; it could be in August instead.)
Behind that announcement is a lot of activity. Obviously we’re not going to just throw the code over the wall and wish everyone good luck. The point of doing this is to bring the community that uses Launchpad.net into the development of Launchpad.net. There’s a philosophical reason for this: for a site whose primary purpose is to enable free software projects to collaborate, it’s natural that the users should be involved in improving the site itself.
But there’s a more practical reason, too.
One of my favorite metrics for judging a project’s success is to watch its bug tracker: the more bugs filed, and the faster the rate of new filings, the more successful the project is.
Those who don’t work in software sometimes express puzzlement at that metric: “What? How can more bugs be better?” But software developers know exactly what I mean: the rate of bug filing is proportional to the number of users, not to the number of actual bugs. (More precisely, it’s proportional to some combination of the number of users and their level of technical sophistication, since certain kinds of users are much more likely to file bugs than others.)
Well, I’ve been watching the Launchpad.net bug tracker for several weeks now, and judging by the bug-filing metric, Launchpad.net is very successful. Great. But what that means in practice is that we can’t resolve the majority of incoming bug reports and feature requests. The Launchpad development team at Canonical simply does not have enough surface area to handle it all. If we doubled — indeed, if we quadrupled — the size of the team, it still wouldn’t be enough. With a site like this, whose user base is large, and growing rapidly, there are really only two options:
- triage mercilessly, and leave most things undone
- open it up and let the users help
Clearly, answer 2 is better.
It implies some preparatory work on our part, however. We’re seeding the ground with developer documentation: this has already begun, though we’re still transferring over much of our internal documentation (remember, we have to go through everything and make sure we don’t accidentally open up any confidential business-related things — all this stuff was written by people who thought it was internal to the company at the time).
We also need to go through our dependencies and make sure we don’t have any license compatibility issues. The license on Launchpad will be the AGPLv3, but we still have to vet dependencies and work out any problems. By definition, this must happen before we release, so it’s work that Canonical must do internally.
Most importantly, we need to move our development discussion forums out into the open. Fortunately, most of Canonical’s Launchpad developers already have plenty of experence working in open source communities: they’re ready for this move, and we’re planning to do it before the code itself is opened (see the schedule), to minimize the number of simultaneous changes. We’ll also be publishing policies on how we’d like the community to work, and on how changes will be accepted back into the mainline and deployed on Launchpad.net. (The short answer is that Canonical decides, of course: we host the site, therefore we are responsible for the software that runs on it. The longer answer is that we want to make it easy for good changes to make it onto the site, and have some concrete plans for how to do that.)
In the near term, the next things you’ll see are more independent modules being opened up: code extracted from Launchpad and made into independent libraries for anyone to use (for example, Storm, LAZR.config, and LAZR.delegates).
And that’s all for now. Watch this space for more.
January 16th, 2009 at 6:06 pm
“There’s a philosophical reason for this: for a site whose primary purpose is to enable free software projects to collaborate, it’s natural that the users should be involved in improving the site itself.”
And yet, the Soyuz component of Launchpad..the component does deals with everything associated with building and releasing Ubuntu packages.. is not going to be opened for the benefit of the community of Ubuntu contributors who make use of Soyuz.
https://launchpad.net/~ubuntumembers
450 active team members 904 proposed members Don’t all of those people make use of the Suyuz component…the component of Launchpad that isn’t being opened up.
What is the largest team in Launchpad at the moment? Does that team’s work flow require the use of Suyuz?
How many people in Launchpad have signed the Ubuntu Code of Conduct and are now listed as Ubunteros and can thus get access to the Ubuntu specific PPA feature of Launchpad? Isn’t the soyuz component important for those Launchpad users who are making use of the Ubuntu PPA feature? Don’t the same philosophical and practical reasons which apply to opening up Rosetta apply to opening up Soyuz?
-jef
January 16th, 2009 at 6:48 pm
I think we already explained that here. (I understand it won’t make you happy, but it is still the explanation.)
Best,
-Karl
January 16th, 2009 at 7:12 pm
Are you going on record as saying that Soyuz is not a critical component in Ubuntu collaboration?
Are you going on record as saying that Ubunteros do not find the Ubuntu PPA feature an important collaboration tool?
https://launchpad.net/ubuntu/+ppas
6258 registered PPAs 38009 published binaries
Isn’t soyuz involved in building and publishing all of those 38000 binaries? And you consider that sort of functionality as non critical to collaboration?
Is Canonical committed to putting an alternative replacement in place for Soyuz at launchpad.net that is open and does not have Canonical’s deep customizations that users of the PPA service can choice to use and to help openly develop?
-jef
January 18th, 2009 at 3:18 am
I find the rate of bug ticketing to be an interesting metric indeed.
Open Bugs Total Reported Bugs
Soyuz : 512 1395
Rosetta: 293 1369
blueprint: 156 299
answers: 119 315
By Launchpad’s own bug reporting statistics, soyuz has seen more bug reports than rosetta or blueprint or answers..and has more bug reports open as well. By the reasoning stated above, the Lauchpad users are more interested in soyuz than these other Launchpad components and thus by the same reasoning Soyuz is more in need of being opened up to contribution and should be opened first before these other components.
Its unfortunate that Lauchpad was designed in such a way that Canonical’s own business processes are getting in the way of Soyuz being opened up. By all the given rationale for opening Lauchpad piecewise, Soyuz has earned to be one of the first pieces opened.
-jef
January 18th, 2009 at 6:01 am
For the reasons I stated earlier, opening Soyuz would be quite a bit more work for Canonical than would the other components; we just don’t think that investment would pay off for us. It is a business, remember. Disentangling a component takes time — time we spend doing that is time we don’t spend doing something else.
Best,
-Karl
January 18th, 2009 at 12:14 pm
Karl,
“There’s a philosophical reason for this: for a site whose primary purpose is to enable free software projects to collaborate, it’s natural that the users should be involved in improving the site itself.”
The open source model is only used when it suits you for business reasons:
“…we just don’t think that investment would pay off for us. It is a business, remember.”
So this whole “philosophy” thing is not true.
Hmmm.
“Put your knives on for the right reasons” -Jophery Brown, Cyborg
January 19th, 2009 at 12:59 am
I see speculation, but no hard commentary on which parts *aren’t* being open sourced. Could you give us a list, and some more information on the open-source alternatives you allude to?
January 19th, 2009 at 9:30 pm
Karl,
Yes.. it does take time to disentangle corporate interest from code when the code was originally designed, written and implemented without planning for it to be opened. Are you prepared to go on record with a target date as to when Soyuz will be opened?
Soyuz is simultaneously critical to both Canonical’s corporate needs and the Ubuntu community colloration needs only because Canonical chose to design Soyuz’s functionality in a way that entangles the two and makes it harder to now give equitable code access to the Ubuntu community.
This seems at odds with the idea that Canonical has always planned to open up Launchpad. Are you going on record as a Canonical employee saying that Canonical didn’t plan ahead for Soyuz and its functionality to be opened?
-jef
January 20th, 2009 at 9:41 am
@Jef
False dichotomies may be thrilling oratory where you come from, but around here they are rather boring.
In my judgment, Canonical’s decision to publish the source code to Launchpad now is appropriate, and consistent with our commitment to do so from the very inception of the project. With around 10,000 projects using Launchpad.net as of January 2009, we have achieved the thresholds needed for this step. It is equally appropriate for Canonical to retain privileged access to the capabilities in Soyuz, which we wrote. As with any contributor to Ubuntu, Canonical gets to decide how it can best participate, and what it wishes to contribute.
On balance, in my judgment, the Ubuntu community will benefit most from Canonical’s continued investment in key technologies, even if we do so internally and without community contribution, and that benefit outweighs other goods from other approaches.
While I appreciate the energy you are willing to devote to the Ubuntu community, I don’t think it helps your work with Fedora to do so under false pretenses. It would be far preferable to me, and to the broader free software community, if you would channel that energy into collaborative ventures between Ubuntu and Fedora, as other Red Hat and Ubuntu developers do. Sniping from a distance, albeit under the cover of a supposed moral high ground, achieves little. Especially when you have yet to bring the same guns to bear on your own project and it’s parent company, Red Hat.
I’m sorry that you have taken the opportunity to pour negativity on what is really a very exciting milestone – the release of the code that drives Launchpad.net.
While you haven’t stepped outside the bounds of the Ubuntu code of conduct, there is little point in us continuing in a conversation where the other party is not acting in good faith. For that reason, I’ll invite you to have the last word now, after which we will consider the conversation exhausted.
Mark
January 20th, 2009 at 5:49 pm
Mark,
Correction, you are publishing some of sourcecode for Launchpad…not all of it launchpad. Is it so very difficult to use qualifiers when you say something? Soyuz is a significant component of the collaborative experience. Most likely the most significant in terms of impacting the Ubuntu community collaboration. Every single time you fail to mention Soyuz I’m going to bring it up.
That 10,000 you quote is a very intriguing number. How many of those 10,000 project registrations are just registered in Launchpad as part of Ubuntu workflow but upstream development is done somewhere else?
quote: https://launchpad.net/projects
“There are 9884 projects registered in Launchpad, of which
2811 have bugs reported, 605 have translations, 6215 have Bazaar branches,
1749 have blueprints, and 1572 have questions & answers.”
Out of that, how many projects “use” Launchpad as part of upstream development? Launchpad’s summary page doesn’t say. Just because a project is registered and has a bzr branch doesn’t mean the upstream project is “using” launchpad… it could just mean that an Ubuntu member is using Launchpad to maintain Ubuntu packages of that project and are thus required to register a project in Launchpad.
Case in point NetworkManager.
https://launchpad.net/network-manager
“Does not use Launchpad for development.”
But its a registered project in Launchpad, and it has blueprints and it has answers and it has bugs and it has bzr branches and it has ppas (thanks to Soyuz). But its not developed in Launchpad…its developed somewhere else.
Does of begs the question… what does it really mean to have a project registered in Launchpad?
Sure some projects are listed as “using” launchpad for development, such as Mysql and inkscape. But how many out of that 10,000 does that represent? And how many are just registered in Launchpad as part pf Ubuntu workflow, a workflow that requires Soyuz? If the vast majority of that 10,000+ project listings are like NetworkManager or like Blender or like OpenOffice or like FUSE… aren’t you quoting that 10,000 number under false pretenses? Counting every single project registered in Launchpad as part of Ubuntu workflow, doesn’t really give an accurate picture of Launchpad as a project hosting service. So out of that 10,000 how many are really hosted at Launchpad as the primary upstream development? Half? Are Half of the registered projects in Launchpad only there for Ubuntu distribution work flow?
You need to be more careful with how you throw numbers around. Since Ubuntu’s packaging and release process is intimately tied with Launchpad its difficult to separate out which projects are actually hosted there, and which projects are merely registered there as a necessary requirement for Ubuntu’s packaging process. I know you would certainly hate to think that people were mislead into thinking that 10,000 individual projects were actually using Launchpad for upstream development processes. If you were just a little more careful with how you use the numbers, I wouldn’t need to spend time explaining what you really mean. Unless of course you are quoting numbers that I don’t have access to. let me re-iterate, here is what I see: https://launchpad.net/projects
Out of all those 10,000 projects you say use Launchpad (really 9884, but whose counting)
Only 605 have translations
Only 2811 have bugs reported
Only 1749 have blueprints
6215 have Bazaar branches, but bzr is required for PPA so that maybe indicative of how important Soyuz is.
Did someone mention PPA’s? https://launchpad.net/ubuntu/+ppas
* 6324 registered PPAs
* 1504 active PPAs
* 8667 published sources
* 38258 published binaries
38000+ published binaries! 6324 registered PPAs! There are more PPAs than bzr
This is clearly Launchpad’s most used feature, a feature which Soyuz is a critical part of. It’s more used than blueprints or answers. Fascinating, maybe that’s because the Ubuntu contributors as users of Launchpad are more interested in the PPA feature than blueprints or answers. By every metric so far used to justify opening up Launchpad to community contribution, Soyuz has earned the right to be open as well from a community interest perspective. What is holding it back is Canonical’s decision to deeply wed community facing processes with internal business processes in such a way that it makes it more difficult to now give the contributors equitable access to the codebase of the tools they are using.
You are free to see negativity in my advocacy for having Soyuz opened. You are free to claim bad faith, an accusation which presupposes my intentions (and let me remind everyone that you already misidentified me as a Red Hat employee so you have already have shown a willingness to presuppose my intentions). I certainly cannot make a better defence other than to say I am a genuine believer in the idea that open development is the better process, and that the Ubuntu community would be more empowered to innovate if they had complete access to the toolset used to build the distribution.
Even if you aren’t engaged directly with me, I am going to continue to talk about how important Soyuz is to the Ubuntu process when the discussion is relevant. Every single time Launchpad is mentioned in the laypress where I am free to comment, I am going to be refining my arguments about the importance of opening Soyuz. I’m quite sure the laypress will welcome my feedback if its sensational enough to drag more eyeballs to their advertising sponsored websites. But you know this already, you already know how to get press attention. I understand that you have come to a decision, for Canonical business reasons, to keep Soyuz closed. But just because I understand that, doesn’t mean I’m going to let it be. It’s the wrong decision for the community contributors, and I am nothing if not an advocate for equitable treatment of community contributors.
-jef
January 20th, 2009 at 9:50 pm
“False dichotomies may be thrilling oratory where you come from, but around here they are rather boring.”
Isn’t that side stepping the question though?
“It is equally appropriate for Canonical to retain privileged access to the capabilities in Soyuz, which we wrote. ”
Privileged? You mean proprietary access to only a commercial company that excludes the rest of the community which I don’t think is appropriate or in line with the commitment you speak of
“As with any contributor to Ubuntu, Canonical gets to decide how it can best participate, and what it wishes to contribute.”
And to hell with the community. nice.
“Especially when you have yet to bring the same guns to bear on your own project and it’s parent company, Red Hat.”
Maybe. Just maybe because Red Hat actually does deliver what it promises and is a much more active contributor and far more of a true free software citizen than Canonical ever has been
http://fedoraproject.org/wiki/RedHatContributions
You were also exposed for your false claims about having a better security record at
http://lwn.net/Articles/290156/
“While you haven’t stepped outside the bounds of the Ubuntu code of conduct”
When did your blog become restricted by some “code of conduct”? Step off your high horse and answer the valid question. Why is soyuz being kept proprietary? what sort of commitment is that?
January 21st, 2009 at 2:01 am
@Jef
I read your blog through Fedora Planet and am not a contributor or part of the Fedora communinty. The posts there and here are really turning me away from that group. Is there a reason you are so anti-Canonical in everything you say?
Jonathan
January 21st, 2009 at 2:16 am
Mark,
While you may disagree with Jef he did post recently on Fedora Planet that he doesn’t think the 10,000 project number is accurate (http://jspaleta.livejournal.com/33352.html). I’d agree, there are a ton of things in launchpad that are upstream elsewhere and not in your distribution that appear to be ‘in launchpad’.
I work on a couple of these and they are /using/ fedorahosted.org, they just have a /record/ in launchpad. It would be like saying projects with records on freshmeat were hosted on freshmeat — it doesn’t mean much.
This doesn’t make launchpad less valuable to your users, it just means that the projects aren’t really USING launchpad and what you say appears to overstate the size of things).
I’ll avoid weighing in on proprietary software at this point; my point was pointing out that the launchpad number is inaccurate.
January 21st, 2009 at 4:18 am
@Johnathan
“Is there a reason you are so anti-Canonical in everything you say?” It’s called jealousy.
January 21st, 2009 at 10:05 am
@jef
please stop the negative attacks and vigilantism.Redhat had satellite closed for a long time, SUSE had yast. Those companies are allowed to add value thru some closed components. it’s not all or nothing. Don’t use it if you dislike it.
January 21st, 2009 at 1:53 pm
To be fair, Satellite is no longer closed. The value in it being open as tremendously more important, for without openness, you cannot build a community and users cannot share in development and adapt it to meet their needs.
January 21st, 2009 at 6:04 pm
@sam
The problem is not proprietary software as a product. I’m not going after Canonical for its proprietary Landscape service. Well at least not yet. I will though. Landscape and Satellite are very close cousins in terms of market positioning as products. Satellite is open in the form of Project Spacewalk…open for community contribution…with all that implies. I think that’s an important story to be told at some point, thank you for reminding me to get around to telling it.
The problem isn’t the proprietariness of Soyuz in and of itself. The problem is its proprietariness in the context of community contribution. The problem is when a community is asked to participate in the development process directly and the tools are kept closed to them for the sake of preferred business interests against the best interests of the community effort.
If Ubuntu were developed like Xandros (the leading pre-installed consumer linux distribution) as a proprietary process, with very little community input into the packaging work, Soyuz would be a non-issue. But Canonical has from day one positioned Ubuntu as a community effort, to choose to downplay the importance of Soyuz to the community, is to choice to devalue that community effort. Soyuz is very important to the work the Ubuntu community does under Canonical’s leadership. Far more than other launchpad components like answers and blueprints. Launchpad’s own publicly available numbers support that. If that leadership is going to expound a rationale for opening up other parts of Launchpad for contribution, that rationale should be self-consistently applied to Soyuz as well.
The fact that business interests put Canonical and community interest into conflict over Soyuz is commentary on how deeply Canonical has planned for this moment…if they planned for it at all.
Canonical claims to understand that open development is the better process. Canonical claims to understand the value of a contributing community and an open codebase. None of the reasons for preferring open code has been a subject of debate. The only issue is whether Soyuz is “different” than the other components of Launchpad..so different that it doesn’t need the benefits of open development. And what is that difference? Soyuz unlike the other components of Launchpad is somehow more important to Canonical than it is to the community. Canonical has tried to downplay how important Soyuz is relative to its importance to Canonical’s business interest. Without Soyuz there would be no Ubuntu packages. I think that makes it very important to the contributing community and worthy of discussion.
Canonical built the Ubuntu process, and built the Ubuntu tools, they invited community to participate and use the tools. Canonical had every chance to build a process which firewalled business and community interests to keep them from directly conflicting. But they chose not to do that. They chose to build a development process that entangled business and community interest together. And as we see, when such entanglement exists, when push comes to shove, privileged business interests win the day. I’m not satisfied with that outcome.
I for one think the leadership of Canonical is extremely smart and make decisions in a deliberate manner. And insofar as the decision to keep Soyuz closed is a mistake which over emphasizes business interests over community interests, the mistake is deliberate. But here’s the great thing about business interests…they are exquisitely malleable and responsive to all sorts of different pressures. If Canonical is holding up business interests as a reason to keeping Soyuz closed and out of the hands of the community, then for the sake of the community I can work to see those business interests realign by putting pressure on Canonical publicly so that an open Soyuz becomes something that is in Canonical’s best interest.
-jef
January 22nd, 2009 at 4:04 am
@Nathaniel: We’re keeping Soyuz and Codehosting internal. We are planning to open source a branch scanner that pulls bzr branch metadata into Launchpad. That doesn’t replace Soyuz, though, and doesn’t replace all of Codehosting — it just makes it possible to test changes to other parts of Launchpad that touch Codehosting. I’ve updated the wording on dev.launchpad.net to reflect the above, to be more explicit about the reasons, and to not make claims about anything’s integrality to collaboration (since that really depends on what a given project is trying to do).
-Karl
January 22nd, 2009 at 10:01 pm
Karl:
Thank you. It would help to have those two components described in further detail as the open-source launch approaches, so those considering using Launchpad code can better evaluate the choice.
You’ve just been given a laundry list of arguments for open-souring those components, so I won’t reiterate them. I will say that keeping the subject of doing so on the table internally will only be to your benefit. Qt and Java are now both open source, despite years of refusal (and angry response); while I’m not happy with some parts not being so currently, I have plenty of hope in the longer term.
Jef:
I understand some of your irritation, but either you aren’t here to actually change any minds within Canonical, or you have no concept of how to achieve it. You would do well to remember how ineffective angry insult was in changing your mind, when you were last confronted with it. Please examine your motives or your methods, as appropriate.
February 10th, 2009 at 11:22 pm
what are the main functionalities of codehosting?
February 11th, 2009 at 5:47 pm
Jef:
I await with baited breath the start of your development on a free Soyuz replacement once the rest of Launchpad has been freed. Alternatively, perhaps you would be interested in talking to Mark Shuttleworth about buying Canonical from him, so you can free Soyuz yourself?
February 11th, 2009 at 5:48 pm
Wow, opening up Launchpad. That’s a big step for this project. I don’t want to imagine the server load it takes, but I’m impressed with this whole thing.
I would have been OK with LP being closed for years longer. Keeping those two pieces closed makes a lot of sense too from what I’ve seen.
It’s impressive that you’re opening this in hard financial times too.
February 11th, 2009 at 5:54 pm
Anonymous,
Codehosting provides the integration between Bazaar and Launchpad. For example, when you push a bzr branch to lp:foo, that’s going through codehosting. The back end of the vcs-imports service (continuous import from external Subversion repositories) is also part of codehosting.
Note that the front end of that (CSCVS — CVS/Subversion to Bazaar mirroring tool) is separate from codehosting, as is most of the glue that connects codehosting to other parts of Launchpad (e.g., automatic linkage between a bzr branch and a bug report).
February 26th, 2009 at 1:13 am
Jef:
Get a grip.
What is ubuntu? It is the essence of community togetherness. It is vaguely communistic, except that it’s more than that. It is the sense of working together, the sense of building to help everyone instead of building to help only oneself.
This does not mean that one is not allowed to be selfish, it does not mean that one is not allowed to have privacy, and does not mean that one is required to share more than one wishes to. Mazlow’s pyramid suggests that the first and foremost concept in any entity’s mind is the ability to continue one’s own existence. Canonical has the right to do so, and in fact the responsibility to do so, by keeping their business-secret stuff secret, so that they can continue to operate — so that they can continue to contribute to the whole.
By demanding that they open everything, you’re essentially demanding that the things that allow them to continue their own existence, the things that allow them to contribute, be removed — so that they can no longer contribute. (It’s a bit like trying to cut something to make it bleed — sure, you get the raw material in the blood, but you lose the mind and ability to direct the resources where they’re useful. Put another way, it’s like killing the goose which laid the golden eggs.)
Every system has detractors, every system has entities which work within the system for malicious, malignant purposes. Your demands suggest that your entire desire is to destroy one of the participants in this community, one of the greatest contributors to the community. This suggests that you are acting cancerously, malignantly.
“alright, you’re right, it can’t happen, your character is DEAD. Happy?”
There’s being “true to ideals”… and then there’s being “realistic to ideals”. Canonical is willing to bend, and is taking some of the steps that they need to take to make their things more open — but you want them to do it all instantly?
God, you’re even more idealistic than Stallman.
June 7th, 2009 at 11:07 am
Richard Stallman uses Ubuntu!
June 8th, 2009 at 7:29 pm
Kyle H:
It’s more like allowing the goose, which lays the golden eggs, to be free in an environment where there is lots of pampering but no food, thus the goose starves to death. A cruel, sad truth.
That’s not to say it will never be released to freedom, but the environment must first be made livable, or else another form of income be made to fund the feeding of the goose in it’s free environment.
Note: in the above analogy, “feeding” refers to whatever is required to keep the community together and functioning as a whole; this can be money, direction, goals, contributers, or whatever else is required for a project to survive. The environment refers to the community and all contributers (including Canonical), and what they do to provide for the project. The golden egg would be any benefit that Soyuz (or any other project/product) brings to Canonical, which allows the company to justify spending resources on it.