Ibid chat bot
Published by Matthew Revell November 9, 2009 in Projects
Chat bots can be controversial. Apart from the fun to be had watching a friend carrying out an inadvertent Turing test, finding the right balance between useful information and annoyance seems to be one of the harder aspects of chatbotology.
I spoke to Jonathan Hitchcock, Michael Gorven and Stefano Rivera about their work on the Ibid chat bot.
Matthew: What are you doing with Ibid that’s different to other chat bots?
Ibid team: Like all bots, Ibid has two parts: the front end, and the back end, and we like to think that there is something to distinguish Ibid in both.
On the front end, we attempt to make the bot as intuitive as possible to interact with. All commands are natural language, and our philosophy is that the plugins should try to work out what the user actually wants to know and answer that, rather than forcing him/her to learn an esoteric set of syntax before he can use the bot. Queries are bubbled through plugins in order of priority so that the most relevant reply can be given.
Secondly, on the back end, Ibid is also divided into two parts: “sources” (i.e. transports, communication mediums – IRC, jabber, email, etc) and “plugins” (modules that tell the bot how to respond to various queries). The design aims to be clean and modular: all sources and plugins are discrete units that can be enabled or disabled without breaking any assumptions elsewhere in the code. Unlike other bots where non-IRC protocol support seems to be an afterthought, Ibid plugins deal with all protocols equally (hopefully without making them all equally bad).
This modularity helps both new and existing developers. It is easy to jump straight into a small part of the code without having to understand the entire system. It is also very easy to add new sources: we had a DC chat source up and running in under a day, and a prototype GSM SMS gateway in a weekend. In addition, hot-pluggability is an important goal of the project: you can add and remove plugins and sources on the fly, without having to restart the bot.
Matthew: Who do you see using Ibid?
Ibid team: As we’ve been developing the bot, we’ve started thinking in terms of three different personas: users, owners and developers.
Users are the end-users who happen to be in IM channels where an Ibid is running — they should just see the bot as a fun and useful tool. They’ll notice people using it to google things, store factoids, check the weather, do currency conversions, and so on, and start following suit. We’ve noticed that our bots very quickly become integral to channel communities, and we try to make sure the bot’s “personality” facilitates this: the natural language interface, and the characteristic responses that the bot gives aid this. Michael and Jonathan both have Ibid instances running internally at their companies, and the other employees find them very useful.
Owners are the people that run instances of the bot — they probably run the channels too, and they need a way to integrate other things with IM. Ibid aims to make that easy. We’ve already got plugins for checking RSS feeds, working out distances, fetching tweets, and so on — the plugin nature of the bot’s features mean that as soon as a new need arises, we can churn out some code to fulfill it.
This is where the third persona comes in: the developer. Ibid is designed to have a very low barrier to entry, and to make it very easy to write plugins, so it can be used as a framework for interfacing with almost anything. It already has plugins for interfacing with Bazaar, Trac and Buildbot, and a Launchpad plugin is in development. Because Ibid is so modular, stripping out all the funky stuff to build an IM interface to a single service is pretty simple, and Ibid hopes to be a good building block for such systems. Which brings us to your next question…
Matthew: Are you hoping to attract developers to Ibid?
Ibid team: See here.
That would be a yes. In the channels where we have Ibids running, users have contributed plugins for features they want, and in one case used it as a quick way to get an IRC game up and running. We’d like more of all of those.
We’ve done our best to make it as simple as possible to write Ibid plugins. There’s still some work to go (mostly documentation), but anyone who knows some basic Python should be able to hack a quick plugin together in a very short time.
We haven’t made an initial release yet, as some of the Internal APIs have some biggish problems that need to be resolved, but the bot is packaged and already works out of the box, and we’d love for people to start using it and contributing plugins.
Matthew: The project’s a fairly heavy user of Bazaar. What made you choose Bazaar over other VCSs?
Ibid team: It was almost accidental — we didn’t have much experience with DVCS, but we definitely wanted to use one rather than, say, subversion. We saw that the Ubuntu community was punting Bazaar, which is relatively sane (compared to, say Git), and ran with it. It turned out to be a good choice: it’s simple to learn and pretty powerful. It also has a very low barrier to entry, coming from more traditional VCSs, but you can ramp up to using the advanced features in fairly small, but logical steps (which makes it very similar to python, as a learning language, I suppose).
Matthew: What aspect of Launchpad has most helped your development of Ibid?
Ibid team: We originally started with Launchpad just to get quick bug-tracking integration, but soon discovered the other features on offer. After moving the source over to be wholly hosted on Launchpad, we started to use branches and merge-requests, and these have now become integral to our development methodology. Every feature and fix that we work on is developed in a separate branch and the merge reviewed by all three of us. Waiting for reviews can stagnate development a bit when all the developers are busy with their dayjobs, but we think the results are worth it — apart from the obvious benefits of catching bugs before they’re merged into trunk, the request-and-review process keeps us all involved and aware of what is happening in the project.
I imagine that as we grow, this benefit will lessen (since not everybody will review each merge), but it is still a good way of signalling to the rest of the team that changes are happening in an area, so that if they are interested, they can make their opinions heard before the changes get finalised.
As a whole, the tight Lauchpad-Bazaar integration has been a huge bonus, allowing us to develop from a few guys committing bits of code to trunk into a community with workflows, milestones, goals and a clear vision of where we’re headed. Having the VCS and the bug-tracker and the discussion boards and everything all in one place really builds cohesion in the project.
Matthew: What would you most like to see improved or otherwise changed in Launchpad?
Ibid team: We have been moving more and more of our project management into Launchpad (throwing out Trac, and our own Bazaar repository, and our own mailing lists, and so on), but there are still a few things that we have to host ourselves: repositories for non-Ubuntu packages (just Debian debs right now, but we’d like to include RPMs, etc, in the future), the build environments to create them (we have our own pbuilders presently), documentation, and our wiki.
There are free services that provide each of the above, such as the OpenSUSE build service, and the many “create your own wiki” services, but filling those gaps would mean our project could live entirely on Launchpad, and would bring along greater integration and easier flows.
Other neat things we could use would be support for Bazaar hooks, and permanently linking merge commits with the merge request — currently we include the merge request URL in the commit log.
Basically, tighter integration of everything would make project management much easier, and let us focus more on coding, which is always a win.
Matthew: Thanks Jonathan, Michael and Stefano!
Launchpad offline 09.00 – 10.30 UTC 5th November
Published by Matthew Revell November 4, 2009 in Notifications
Launchpad will be offline for a scheduled code release for around 90 minutes from 09.00 UTC on the 5th November.
Going offline: 09.00 UTC 5th November 2009
Expected back: 10.30 UTC 5th November 2009
This is to release version 3.1.10 of Launchpad. We’re sorry for the short notice of this down-time.
Commenting on questions
Published by Gavin Panella October 30, 2009 in Cool new stuff
If you’re using edge, you can now just comment on a question in Launchpad. For all questions on Answers, the “Just Add a Comment” button is now always visible.
Previously, you might have only seen “Add Answer” and “Add Information Request” (or others; the exact buttons vary), both of which add a comment and cause the question status to change. But often, for example, all you want to do is clarify an earlier comment, add some detail, or give a progress update. For that, “Just Add a Comment”.
It’s been put at the rightmost position of all the buttons because we think it should be the least used option. Normally it’s appropriate to use one of the other buttons to move the workflow forward.
The button will land in production with the 3.1.10 release next week.
Accessing Git, Subversion and Mercurial from Bazaar
Published by Jelmer Vernooij October 29, 2009 in Bazaar
bzr-svn, bzr-git and bzr-hg are plugins for bzr that make Subversion, Git and Mercurial branches first class citizens in the Bazaar world by allowing you to access them in the same way that you would access native Bazaar branches.
Bazaar has supported multiple file formats from its early days. Both its model and its implementation allow this:
- Revisions are not identified by the checksum of their layout on disk (as they are in systems like Git or Mercurial) but by a (pseudo-)random string. This means that copying data to a different file format does not affect the revision id.
- Repositories are accessed through a well defined interface. Other parts of the codebase are ignorant about the structure of the files on disk.
This has made it easy to introduce better and experimental repository formats without having to break old repositories or render them unusable for previous versions of Bazaar by forcing upgrades. Initially new formats were introduced at a very high pace (perhaps even a too high pace?), but fortunately this has slowed down nowadays: the last default format change before the 2.0 release was in 2007.
Having grown interested in Bazaar through Martin‘s talk at Linux.Conf.Au 2005 and his blog posts I started looking into Bazaar in 2005. Since Samba (the main FOSS project I work on) had just switched to Subversion, I was interested in ways to interact with Subversion using Bazaar, in particular so I could do off-line commits. On the Bazaar wiki Aaron had suggested implementing the well defined interface for repository formats for other version control systems (such as Subversion) as well. This sounded very neat, so I decided to see how far I could get and looked into learning Python and becoming more familiar with the Bazaar API.
Now, four years later, 700 bug reports and about 4400 revisions later, we have released bzr-svn 1.0. The models of Subversion and Bazaar have significant differences, and bzr-svn has to take care of mapping between the semantics of both. Perhaps the best example of this is the fact that a Subversion repository is basically a versioned filesystem; there may be some directories that are commonly used as containers for branches or tags, but there are a lot of exceptions to this convention. In Bazaar on the other hand, a branch is a primary object.
In 2006 Rob and Aaron created a simple plugin for accessing local Git repositories in 2006 called bzr-git. Originally it was based on “stgit”, a tool which (among other things) exposed a Python wrapper around the git executables. Following a switch from Samba to Git I took over in 2008 and changed bzr-git to use a new native implementation of Git in Python, based on a project by James. bzr-git now supports accessing remote repositories, working trees and merging changes back into Git.
At the moment I am working on the bzr-hg plugin, again based on an initial proof of concept by Rob. Last month ago the first version (0.1) was released, providing sufficient support for cloning local and remote Mercurial repositories and accessing working trees. There are still some problems to work out — memory usage is excessive, commit and push do not yet work — but there should be a stable plugin in a few months.
Meet Francis Lacoste
Published by Matthew Revell October 28, 2009 in Meet the devs
Francis Lacoste recently started on a six-month stint of running the Canonical Launchpad team. It seemed like a good time to find out a little more about him.
Matthew: How did you get into free software?
Francis: It was 1996, with a few friends at university, we started an online cinema magazine and) I developed the first generation of the content management system for the site. I was looking to develop this as a “free for non-commercial use” software. Since my Mac at the time kept crashing (Internet and Apple didn’t worked well), I looked into mklinux which was a Linux variant for Powermac. And then I stumbled upon the GNU Manifesto. This made so much sense to me, that I ditched the “free for non-commercial use” and became a GNU head.
Matthew: What’s more important? Principle or pragmatism?
Francis: What is more important, water or air? This is a false dichotomy. You need principles because otherwise your actions are meaningless, at the same time if your principles cannot be applied, cannot drive to action, then your principles are just wishful thinking. The beauty of this interrelation is that you can start anywhere and find the whole. Start at action, ask yourself what are the principles behind it and then you discover more actions you could make to be more effective. Or you can start at principles and then ask yourself how to apply them which will bring more insights as to some way to improve your principles.
Matthew: Do you/have you contribute(d) to any free software projects?
Francis: Not much since joining Canonical (and having a kid). I still receive emails for perl (sic) modules I developed eons ago. I also worked for a few years on the LogReport project. I also met future-Canonical-colleague Martin Pool on the first implementation of what was to become the Apache Java project. Specifically, I updated the mod_jserv implementation to support the Java 1.1 Servlet specification.
Of course, now that Launchpad is open-source, I do daily contributions to open-source :-p (actually, that’s not even true as my responsibilities don’t leave me a lot of time to code — but contributions is more than just code).
Matthew: Tell us something really cool about Launchpad that not enough people know about.
Francis: Launchpad Answers is a really easy way to build community support around your project. (Full disclosure, that was the app I was responsible for when I joined the Launchpad project.)
Matthew: In the Principia Discordia, Malaclypse the Younger states that all things happen in fives. What five things are coming soon in Launchpad that you’re most excited about?
Francis: 1. In September, we defined a strategic vision for Launchpad around “bridging the gap between upstreams and the Ubuntu distribution”. This will help us to develop a much more focused application.
The current set of features related to the strategic focus you can expect are:
2. Forwarding of triaged bugs from Ubuntu to the upstream bug tracker.
3. Daily Ubuntu packages of upstream tip.
4. Sharing of translations between upstreams tip and the Ubuntu source packages.
5. Zero OOPS Policy. This means rooting out all time outs and errors.
Matthew: Okay, Kiko’s special question! You’re at your computer, you reach for your wallet: what are you most likely to be doing?
Francis: Ordering books on amazon.ca. Although by now, I don’t need to reach for my wallet for this.
Matthew: Thanks Francis!
Photo of Francis Lacoste by Simon Law, CC-BY-SA, published on Flickr.
Launchpad offline 04.00 – 04.30 UTC 26th October
Published by Matthew Revell October 23, 2009 in Notifications
Launchpad will be offline for roughly 30 minutes from 04.00 UTC on Monday the 26th October 2009 for database maintenance.
Going offline: 04.00 UTC 26th October
Expected back: 04.30 UTC 26th October
Meet Matthew Revell
Published by Julian Edwards October 21, 2009 in Meet the devs
Our intrepid communications expert, the venerable Matthew Revell, has been quite busy interviewing the Launchpad developers of late. I thought I’d turn the tables on him and ask a few questions!
So, here goes:
Julian: Mr Revell, I’ve heard your name pronounced in different ways. Can you please slay this dragon once and for all, is it RE-vell or re-VELL?
Matt: I pronounce it with the emphasis on the second syllable, to rhyme with
“bell”. But I’m not precious about it 🙂
Julian: Ok excellent, I’m glad we got that sorted out. So, tell us about your role in Launchpad. What are your day-to-day tasks, and do you have any special duties on top of that?
Matt: I work in the new Launchpad Strategy team, with Karl Fogel and Jonathan Lange. My main responsibility is communication with the people who use Launchpad: documentation, blogging, in-line help, the Launchpad tour, announcements, that sort of thing. So, you should speak to me (mrevell on Freenode) if you have ideas or suggestions for that sort of thing.
With Karl and Jonathan I’m also talking to lots of people who use Launchpad to find out how we can improve what we offer. So, at November’s Ubuntu Developer Summit in Dallas we’ll be organising lots of meetings to talk to people who work on Ubuntu to find out how Launchpad can better connect them with upstream projects.
Something else I’m interested in is usability and I’m now one of Launchpad’s trainee UI reviewers, which can be fun and daunting.
As for day-to-day tasks, most days are pretty different for me but I do have the fun job of weeding out the spam sent to our feedback@launchpad.net address each day. With such a huge volume of spam, even those few messages that slip through the filter still add up to a hundred or more a day.
Julian: Where do you work?
Matt: Right now, I’ve converted a corner of our dining room into an office and work there mostly. I live in the city of Wolverhampton, which is about 40 miles east of the England/Wales border and around 130 miles north-west of London. From time to time I pop up to an arts centre in the city, as they have free wifi in their cafe, for a change of scene.
We’re moving to a village a few miles from here soon, though.
Julian: What can you see from your office window? Having been to your office, I can appreciate you might want to make up an answer here.
Matt: Heh, I can see other houses in my street and that’s about it 🙂
Julian: What did you do before working at Canonical?
Matt: I did a similar job at an ISP/web host. I spent most of my time writing articles about how to set up a dedicated server’s DNS and that sort of thing.
Julian: Apparently you used to do stuff with LUG radio! Tell us more about that.
Matt: Yeah, I was a presenter on the LugRadio podcast from the first show until May 2007, when I left to do a politics show on a local station in Wolverhampton. Although the LugRadio podcast is now over, the LugRadio Live event continues and is taking place this Saturday in Wolverhampton.
Julian: How did you get into Free software?
Matt: I got my first computer when I was six. It was a Sinclair ZX81 and was a great introduction to computing. A couple of Sinclair Spectrums, a Commodore Amiga (many hours spent in writing bad code in Amos) and several Amstrad PCWs later, my dad got a PC. I had always enjoyed trying to get my machines to do something different so we ended up putting IBM’s OS/2 on it.
From there I ran a Fidonet BBS and then got into the internet, where free software was pretty hard to avoid 🙂 It wasn’t until 2001, though, that I first installed SuSe Linux on my PC, and from there went through various versions of Red Hat, with a Gentoo pit-stop, until I settled on Debian. When the Ubuntu Warty beta became available, I tried that out and have run Ubuntu since then.
Julian: What’s more important? Principle or pragmatism?
Matt: I think the pragmatic answer is that it’s important to find a route between the two 🙂 I believe in Free software both as a matter of principle and because of the practical benefits it offers me.
I think the only non-Free software I use right now is Skype but that’s because it’s the only way to speak to certain people free of charge and I think that benefit outweighs the relatively minor risks of a VOIP app being proprietary. I also use SIP quite a bit but even then it’s through a hardware adapter so, while it’s an open standard, it’s actually using proprietary firmware.
More generally, I think it’s important to have principles that guide you but none of us is ever right all the time and so it’s important to re-think from time to time.
Julian: Do you/have you contribute(d) to any free software projects?
Matt: Yes, Launchpad 🙂 I used to do a lot for Ubuntu’s Fridge news site and the Ubuntu marketing team. I’ve also worked on some documentation for Bazaar.
My wife and I have two small children so any time I’m not doing paid work is rarely mine 🙂 I keep hoping I’ll find some time to get stuck into the Gnome Documentation Project and to take my Python skills beyond less-than-rudimentary; I’m not holding my breath, though.
Julian: There is a vicious rumour circulating that you like to talk like a Leprechaun when drunk. Can you confirm or deny this rumour?
Matt: I don’t have to be drunk.
Julian: Okay, Kiko’s special question! You’re at your computer, you reach for your wallet: what are you most likely to be doing?
Matt: Usually buying something I don’t have time to use. I bought a Bluetooth remote control a few weeks ago with intention of turning an old laptop into a Boxee machine. The remote is still in its packaging.
Thanks very much Matt! And now, I return you to your regularly scheduled programming.
Launchpad: The next six months
Published by Jonathan Lange in General
A couple of weeks ago, the Launchpad team leads at Canonical gathered in Millbank Tower to talk about what we’ll be doing over the next six months. We talked with each other, we talked with Martin Pool from Bazaar, we talked with people on the Ubuntu Platform team, we talked with Mark Shuttleworth, we talked a lot.
Over the week, two very important things slowly began to dawn on us. I’ll talk about one of them now, and leave the other one to hang tantalizingly in the air like some forbidden fruit that’s learned how to hover.
The first important thing we realized is that Launchpad was originally conceived as a way of helping better connect the Ubuntu operating system to the upstream projects on which it depends. We further realized that could do that much better than we are right now.
A flood of bugs
Zillions of bugs get filed against Ubuntu every day. While some of them are introduced when the Ubuntu community packages software, many are really bugs in the underlying upstream code.[citation needed] And quite often they’re already fixed in the latest upstream version — it’s just that the Ubuntu package doesn’t have the fix yet.
Yet even though Ubuntu is drowning in this sea of bugs, it can’t simply forward them upstream indiscriminately. Upstreams shouldn’t be bothered with old bugs; they only want to hear about bugs that are still in their code. And Ubuntu needs to know when such a bug has been found, both to tell users that a fix is coming and to help plan packaging updates.
Package of the day
Launchpad should be doing much more to help rescue Ubuntu from this deluge. With PPAs and source package branches, Launchpad ought to be able to make it really easy to create a packaged version of the tip of any upstream, to test against, and to file bugs and provide patches directly to that upstream. That is,
Launchpad needs to make Ubuntu Daily Builds rock.
That’s going to be our overall focus now. At the same time, we’re also aware that we need to spend time polishing what we already have. So, for this month and for UDS, we’re going to be focusing only on reducing technical debt, fixing OOPSes and cleaning up the UI.
Where to now?
The Canonical Launchpad team are going to be focused on “bridging the gap” between Ubuntu and its upstreams. We’ll focus on better, faster bug triage, on making it really easy to get upstream tip on the Ubuntu desktop and really tight translations integration between Ubuntu & its upstreams. Early next week, we’ll email out a high-level roadmap of where we want to go.
We are interested in getting real-user feedback about our solution to better integrating upstreams and Ubuntu developers. If you are an upstream or Ubuntu developer interested by that problem, please contact us.
PS. If you’ve read this far, you are probably wondering what the second Very Important Thing was. I’m afraid you’ll just have to wait.
New lpx project group for Launchpad extensions
Published by Matthew Revell October 16, 2009 in API
Jonathan Lange writes on his blog:
Launchpad has a pretty awesome public API, implemented using lazr.restful. I’ve written a few small scripts for it, and the Launchpad team has a few scripts that they use internally for doing admin tasks.
The Ubuntu Platform team does a heap of stuff with the Launchpad API. James Westby has been using it to make sure that there’s a branch on Launchpad for every single package in Ubuntu.
There’s all this great work, but there’s been nothing to tie the room together. I’ve seen hardly any discussion about how to write Launchpad API applications, or how to test them, or how to get launchpadlib working in GTK+. I haven’t even seen much code sharing.
So, borrowing a trick from Twisted’s tx super-project, I’ve created an ‘lpx‘ project group on Launchpad. Bring it your scripts, your applications, your huddled masses. If you want to know more about the API, look at the API help page.
Also, if you’re using the Launchpad API — directly or through the launchpadlib Python library — add some info about your app to the API Uses wiki page.
Launchpad’s status page
Published by Matthew Revell October 13, 2009 in Notifications
When writing about the hardware running Launchpad, or even the complexity of the codebase, I’m always tempted to start off by borrowing from Douglas Adams’ introduction to The Hitchiker’s Guide to the Galaxy: “Launchpad is big. Really big.”
With such a big system, it’s inevitable that from time to time we have to rearrange the furniture a little. Aside from our monthly code roll-out, where Launchpad goes read-only for an hour or so, we occasionally have to swap out or reconfigure hardware, as you’d expect. Up until now, we’ve used a combination of this blog and the launchpad-announce mailing list to keep you up to date on any Launchpad service-affecting issues but we haven’t had a canonical (heh) status page to which you can refer and know you’re going to get a definitive answer.
So, now we have the Launchpad status page!
It’s hosted by the excellent identi.ca so you can subscribe using your identi.ca account or to the Atom or RSS feed. We’re also automatically copying everything over to the launchpadstatus Twitter account.
What does it cover?
We’re using the identi.ca-hosted status page to announce service-affecting issues that are happening right now or that we know will happen in the following 48 hours.
Also, we’re only going to post when the main production Launchpad site or the edge environment are affected, as they’re the two environments in which you can do real work.
Longer-term planning
As the status page only covers what’s happening in the next 48 hours, it’s not so useful for longer-term planning. So, as soon as we know about any pending service-interruption, we’ll add it to the maintenance page of the Launchpad dev wiki.
Blog and email updates
We’ll still continue to post notification of down-time here on the Launchpad blog and to the launchpad-announce mailing list. However, we’ll limit those posts to planned maintenance that’s likely to affect a sizeable number of people.
For things happening right now, or that are likely to be no more than the equivalent of a minor bump in the road, we’ll post them to the identi.ca-hosted status page only.
Elsewhere on identi.ca
Don’t forget, we also have a news account on identi.ca and also Twitter.
