launchpadblog

Back at the beginning of February, we ran our first Launchpad Performance Week.

Our aim was to find pages that, although they weren’t slow enough to trigger our internal warning, were nonetheless taking too long load. The first thing we did was to get our internal warning system to generate a report when a page takes 25 seconds to load, rather than 30 seconds.

So, armed with a list of sloth-like pages, each part of the Launchpad team went away with one aim: to fix whatever was preventing those pages from loading in a reasonable time.

Those fixed pages have been around on Launchpad’s edge environment for some time now and hit production with our 2.2.2 release.

So, let’s see what the team have been working on.

Faster bug pages

As a bug in Launchpad gets more attention — whether you measure that by the number of comments or subscribers — it takes more work to display it. That has led to particularly popular bugs, including the oft-linked-to bug #1, taking too long to load.

Two of the big pain points, on bug pages, were comments and the box listing the bug’s subscribers. For most bugs, they present no problem. However, bugs with many comments/subscribers were nudging Launchpad’s hard timeout limit.

Gavin tackled the slow loading of multiple comments by changing the bug page to display only 80 comments. Where the 81st comment would be there’s now a link to view the rest of the comments or post a new comment.

Tom‘s approach to the subscriber list was to load it asynchronously, preventing it from acting as a bottle-neck to loading the rest of the page. So far, there’s very little wait between the rest of the page loading and the subscriber list appearing.

Personal Package Archive overview pages

The Soyuz team worked on PPA overview pages. Take a look at the Exaile development team’s PPA as an example.

Two hold-ups were caused on this page, particularly for PPAs with many packages: the disk usage information and the detail for each package. Both are now loaded asynchronously, the package detail only being loaded when you click its header.

There’s more detail from Julian in his blog post.

Translation pages

The translations pages are some of the most data-intesnive in all of Launchpad. Danilo talks about the tranlsations team’s Launchpad Performance Week work in his blog post.

Code project cloud

Launchpad Code developer, Jonathan, wrote on his own blog about his work on the project cloud page:

The Launchpad Code home page is now a little bit nicer and a little bit faster.

For a while now, the page has had a tag cloud showing some projects that host branches on Launchpad. The tag cloud is drawn based on these two rules:

• the more branches, the bigger the tag
• the more recent the last commit, the less faded the tag.

Up until recently, the home page only showed some projects starting with ‘A’ — which is a bit silly. My recent patch changed the query to show the most branch-y projects on Launchpad. It looks a lot nicer, IMO.

We also got rid of the green / blue distinction. No one knew what it meant and it made the page more visually muddled. There are still some tweaks we can make, but I think this is definitely an improvement.

And elsewhere

We made a number of smaller changes right across Launchpad, as part of the Performance Week. We’re also working on introducing much more AJAX into Launchpad’s web interface, which should make the experience of using Launchpad feel much quicker.

We’ll be holding another Launchpad Performance Week in April, this time cutting our definition of slow page by a further five seconds. Again, we’ll be posting details of what we did right here.

The week of February 2nd – 6th has been the inaugural Launchpad Performance Week: a week where we concentrate on the most pressing performance issues and do anything we can to improve them. During that week, we also establish a hard goal that we enforce by lowering the timeout limit by 5 seconds.

The Launchpad Translations team has been concentrating on improving performance of two most problematic pages we are in charge of: POFile +translate pages (where translators do most of their work) and Person +translations pages, which display history for a person’s translation contributions inside Launchpad.

Jeroen has done a very good job improving Person:+translations pages (pages like Yannick’s translation history which used to timeout occasionally). We have since seen no timeouts on that page.

Henning has spent some time on writing a script to remove obsolete Ubuntu translations (translations for Warthy, Hoary, Breezy, Edgy and Feisty) from the database, before joining Ubuntu guys for the Jaunty sprint in Berlin. This should remove around one third of our entire data set, which should help with rendering of POFile:+translate pages. Unfortunately, he was unable to complete that — due to not being around 🙂 — so I picked it up and have completed and tested it this week. We should have this done by the end of the month, and that should bring large improvements to the rendering of +translate page.

Additionally, I’ve disabled global suggestions on edge (translations coming from different PO templates in Launchpad for the same English string) to confirm this is indeed our biggest problem. Edge has practically been flawless since, and we’ve discovered more areas for improvement. We are getting some new hardware to help with competition for resources between different jobs on the server as well (should be in by the end of the month), and as soon as we do, we’ll re-enable global suggestions (which are one of Launchpad’s essential features).

Just as the week was ending, we started getting a bunch of timeouts on DistroSeriesLanguage pages like the Brazilian Portuguese one for Jaunty: anonymous users started getting it frequently using a script, along with a large batch size of 300, and that started putting a lot of load on the server. Jeroen already has an idea how to improve the situation significantly, but that will have to wait until a later date.

I am looking forward to February release of Launchpad which should bring performance improvements for everyone to notice.

Last week was the first of some Launchpad Performance Weeks that we’re embarking upon. The Soyuz team worked hard on improving the performance of the PPA page, which had a tendency to be very slow or even time out on PPAs that have a lot of binaries published.

The more observant of you will have noticed that the PPA page on edge is now using asynchronous requests to fetch both the “Repository disk usage” section, and the package information section that is expanded when you click on the triangle at the left of each source package row. This means that the initial page load time is a lot quicker! There’s still some more speed we can get out of the page by reducing the query load that it places on the database server, and we’ll deliver that change soon.

We also fixed a smaller snafu — as part of the webservice work that we’re doing, we managed to accidentally enable a URL that let you view the main Ubuntu repository as if it were a PPA page. These links nearly always timed out because of the size of the Ubuntu archive, so that URL will now redirect to the /ubuntu page on Launchpad instead.

Also in the pipeline is a major overhaul of the PPA page. Watch out for that soon!

Over the next few months we’re turning our attention to making Launchpad sing; not so much Edith Piaf, more the note struck by a perfectly tuned Ferrari.

This month — actually, this week — and again in April and June, we’re holding Launchpad Performance Week (LPW)!

LPW is all about tackling those pages in Launchpad that keep us waiting.

As you’d expect, performance is something we’re working on all the time but we’re approaching Launchpad Performance Week slightly differently. You know how the longer you play Tetris the faster the blocks fall? Well, we’re kinda doing the same thing with LPW. Each month we’re going to tighten our definition of “slow” so that we bring more and more pages onto our radar and then fix them!

Soft and hard timeouts

First off, we need to look at how Launchpad handles timeouts. Let me turn to Ursula in the Launchpad QA team.

“In Launchpad we have two types of timeout: soft and hard. Both help us to measure Launchpad page performance, and also to set limits of how long a page could take to load. Hard timeouts occur when a page takes more than this pre-defined limit to load. When it happens, Launchpad aborts the load and generates an error report, which we call an OOPS, showing the user an error page with the OOPS ID on it.

“On the other side, a soft timeout is more a warning for the Launchpad team rather than an error itself. When Launchpad successfully loads a page, it measures how long it took to load and checks if the soft timeout limit was exceeded. If so, Launchpad generates an OOPS report and lets the Launchpad team know that that page may be a potential problem.

“This month we’ve reduced both timeout limits on edge by five seconds. That means we’ll see more timeouts on edge, so we’ll become aware of more slow pages and be able to optimize them.

“Next time we do a Launchpad Performance Week, we’ll decrease the timeout limits even more, meaning more slow pages are exposed and we can keep working to make them faster, aiming to improve the whole Launchpad experience.”

What you’ll notice

So, each time we reduce the timeout limits more pages will hit the timeout limits on edge. We’ll fix those during the Launchpad Performance Week so that they come within the new soft timeout limit.

However, there is a chance that one or two pages may hit our hard timeout limit on edge and so fail to load. If you’re in the Launchpad beta testers team — or like to use Launchpad’s edge environment for any other reason — you may want to switch to using production Launchpad during that time.

If any pages do hit the hard timeout limit — and that would only happen on edge — then we’ll aim to get them fixed within 24 hours.

Either way, the main changes you’re likely to see are speed improvements and, possibly, small changes in the way that those pages work. We’ll always aim to maintain the same level of functionality in a page but we may find creative ways of making the page more responsive, such as loading some of the data asynchronously.

When we’re doing it

The first Launchpad Performance Week is now. You can expect to see updates on this blog and also a round-up of what we achieved sometime next week.

We’re planning the next LPW for the week starting 6th April, with another starting June 1st. Although we may move that depending on how this week affects people.