Every release day is exciting in one way or another. Lucid’s was no disappointment. April 29th, 2010 was my 8th Ubuntu release as the ubuntu.com webmaster. Counting testing releases, betas and RCs I’ve participated in about 50 releases.

There are many aspects related to a release. I can only talk about my own perspective, as it pertains to managing the website. Usually, a week or so before release we’ve got a pretty good idea of what the website will look like and people are viewing it on a testing server. Invariably there are last minute changes, and I do mean up to the last minute.

Wednesday evening the release manager starts the process of dispersing CD images to the corners of the globe so that when its time to go live they show up on the mirrors pretty quickly. I talk to him and plan a time for me to come online to prepare the website for release. This time we agreed I’d be online at 5:00 am (10:00 UTC), aiming for a 12:00 UTC release time.

There’s an element of risk in pre-seeding the CD images because the image testing happens in parallel. Normally its not a problem but this release a serious bug was found. I’m not sure of the details but it was bad enough that there was a “day-of” re-spin. When I came online at 5:00 the disks were being remastered.

This went OK apparently but a new problem popped up. We have a mirror prober application that checks our mirrors to see if they have the correct images. We do this by making HTTP HEAD requests to ensure the file name exists and the file size matches what we have. Unfortunately, the remastered disk images were exactly the same size as the defective ones. Exact to the byte.

We couldn’t tell which mirrors had the new version and which had the bad version. We use a protocol to communicate with our mirrors telling them which files to mirror. Therefore we updated the list to include a dummy file. That allowed us to identify which mirrors were up to date and which were not.

Finally, about 12:21 my time (17:21 UTC) we got the word that it was about time to “push the button.” I had prepared by opening up a bunch of tabs with the pages that were changing, updating the pages so that all I had to do was submit each form. The release team gave me some last minute changes and in the process of applying those I accidentally closed my browser window. If you were in central Iowa you may have heard me scream.

I’m an optimist, so I hoped that firefox would remember the tabs and values of the forms, but unfortunately our authentication system redirects you to an openid auth page when you first visit it. All was lost.

So I copy and pasted again while people constantly instant messaged me asking if it was ready yet. Finally, at about 12:46 (17:46 UTC) the website launched. I kid you not, multitudes of people open their browser to the homepage and refresh again and again waiting to see the site change. I hear someone in the Millbank office with a bottle of champagne waits until the website updates so that they can celebrate. Talk about pressure.

We’re fortunate that our intense loads are so predictable. It gives us plenty of time to beef up our infrastructure and plan appropriately. When we get slashdotted we usually don’t even notice (unless they link to our wiki). A release day is at least 12 times more traffic than when we hit the homepage of digg and slashdot. For a reference, check out this image. The big bump is a release day, the small bump to the left of it is when we hit some big news site.

After the site launches I get a rush of bug reports. There’s nothing like having a million people proof reading your site all at once. The next couple hours are spent doing low-priority updates and fixing little issues that show up. About 3:00 my time (20:00 UTC) I stepped away from my computer and took a nap.

And that is a release day.