More Build Farm Improvements
Continuing with the recent improvements to the build farm – Jelmer has made another massive one.
The last major scalability problem that we had was one where the whole farm was blocked when a single builder was ready to upload a build. In the case of large packages, like the kernel, the manager process could block for over a minute while it waited for the upload processor to unpack the package and verify its contents.
Jelmer’s work has decoupled the upload processing from the build farm manager process. What happens now is that the files collected from the builder are thrown into a staging queue area and then the manager process immediately continues with polling the rest of the builders, unblocked. A cron job will then process the builder upload queue at 1 minute intervals.
You can see the dramatic effect this has had on the overall queue for the PPA builders here:
This is quite an incredible improvement as you can see! But we’re not stopping there, we’re currently doing a massive refactoring of the builder dispatching code so it’s all fully asynchronous. When this is all done we’re going to be in superb shape to support an increase in load that’s anticipated from the increasing number of people using the package recipes.
Tags: front-page
October 4th, 2010 at 11:48 am
All I can say is yaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaay I remember when I had something in the build queue and it didnt get finished for like 8 hours. It was bad timing on my part it was when maverick or lucid (I cant remember) was being built but still it would have taken 20 seconds on the build farm to do it but the queue was way too long.
October 14th, 2010 at 10:33 pm
I used to remember when builds took 2 days to get processed 🙂
This is great news. I had noticed an improvement in the last few days, so it’s great to hear the official word on what’s changed.