Imagine the scene, dear reader. You’re running the latest Ubuntu pre-release, dilligently testing that everything works the way it ought. An application crashes horribly; Apport pops up to tell you that it spotted the crash – would you like to report a bug?
“Of course I would,” you cry, “I am a desktop testing hero!” And so Apport does its thing and takes you to Launchpad to file the bug report. And then Launchpad times out.
At this point, if you’re like me, you might shout a bit. You refresh and refresh with all your might but still Launchpad will only give you an error page. Finally, defeated, broken, you close your browser and shuffle off into the corner of your room, where you bury yourself under the mountain of discarded CD-Rs that contain daily ISOs of Ubuntus past and sob into your coffee.
Okay, so maybe I’ve exaggerated it a bit, but that doesn’t change the basic fact that timeouts on the +filebug page when you’re filing a bug via Apport are intensely, soul-destroyingly annoying. They’re annoying to us in the Launchpad Bugs team too, because we see them in our error reports. They’re so annoying, dear reader, that we’re moved to hyperbole when writing about them for the official Launchpad blog.
The problem with processing data supplied by Apport has always been one of scale. Originally the extra data that Apport would upload to Launchpad wouldn’t be massive, maybe hitting 10MB if the application was particularly busy. Because of this, Launchpad simply processed those data synchronously whilst loading the bug-filing form for you, so that the data you’d uploaded would be included with the bug.
Of course, that approach doesn’t scale very well, and recently we’ve been seeing the data blobs that Apport uploads hit ~100MB in size. That’s far too big for Launchpad to handle in a timely manner whilst doing all the additional work of rendering the bug filing form for you, so in those circumstances it invariably times out and you get that ever-annoying error page.
This was an interesting problem to solve, and we came up with a number of different possible solutions. One viable solution for this, which we eventually decided not to implement, involved having a separate request for processing the extra bug data and then loading the data into the filebug form with AJAX. We discarded that idea because there was always the chance that that request would time out, leaving us in an only slightly better position than we were already.
There was some discussion on the Launchpad developers mailing list about whether we could just defer loading the extra data into the bug until after it had been filed, but we quickly realised that not only could the extra data carry an initial subject for the bug, but it could also indicate that the bug should be filed as private, something which currently can’t be done via the Launchpad web UI.
The solution we chose to implement, which we’ve now rolled out to the Launchpad edge and production servers, is to have a queue of blobs waiting to be processed. We already have quite a robust job-processing system built into the Launchpad codebase, which we use for creating the diffs in merge proposals and calculating bug heat, amongst other things. Adding support for processing uploaded blobs was quite simple, since the existing blob parsing code was well documented. The blob processing jobs are picked up and run by a cron script that runs every minute or so, and the data retrieved from the blobs are stored with the original processing job as a JSON-formatted string. When you hit the +filebug page it checks to see if the relevant blob has been processed. If not, it waits until processing is complete. Once processing is complete the serialized data are loaded into the +filebug page for use later on.
The advantage to you as a user, then, is that you should never again see a timeout on the +filebug page due to the size of the blob that Apport has uploaded. With the upcoming release of Ubuntu Lucid, we’re hopeful that this will make a real difference to those people testing the pre-release versions of the distro.