I created a light-weight HTML page, a “hello world” php app (no db or sessions) and a light weight “hello world” django app using mostly default values (sessions are enabled). I then ran the apache benchmark (ab) on the three using a small VPS w/ 512 MB of RAM from localhost (so no latency is involved giving ideal conditions).

Apache is configured with a max clients of 40 so I tested with a concurrency of 40.

• Static HTML:
  Requests per second:    3249.36 [#/sec] (mean)
  (no measurable impact on server load)
• PHP:
  Requests per second:    2786.73 [#/sec] (mean)
  (no measurable impact on server load)
• Django:
  Requests per second:    157.97 [#/sec] (mean)
  (no measurable impact on server load)
• Static HTML using SSL:
  Requests per second:    10.30 [#/sec] (mean)
  (during this, server load rose uncomfortably high)

What do we understand from this? First, considering the added functionality of the Django framework, I’m quite comfortable with the performance here. If I need as lean of an app as possible, I may want to consider not using a framework. Carefully crafted PHP is probably going to be faster.

However, with a concurrency of 40 the Django app was able to support 157 connections per second. The server load and memory utilization indicates that this VPS can probably handle a higher value for max-clients.

I’m quite shocked at the poor SSL performance. I have no idea why the VPS would perform so poorly. I wonder if the VPS technology forces the server to offload SSL decryption to a virtual network adapter. The CPU of the VPS is an Intel Core 2 Quad @ 2.4GHz.

This is an OpenVZ VPS running Ubuntu 10.04 LTS, Apache 2.2.14, PHP 5.3.2, Django 1.2.3 on Python 2.6.5 using mod_wsgi 2.8.

You can get a good deal on a VPS at burst.net for $5.99/mo (this link uses my “refer a friend” feature so I get credit). I’m extremely happy with the quality of their service and have had 0 downtime in the few months I’ve been a customer.

There are many aspects related to a release. I can only talk about my own perspective, as it pertains to managing the website. Usually, a week or so before release we’ve got a pretty good idea of what the website will look like and people are viewing it on a testing server. Invariably there are last minute changes, and I do mean up to the last minute.

Wednesday evening the release manager starts the process of dispersing CD images to the corners of the globe so that when its time to go live they show up on the mirrors pretty quickly. I talk to him and plan a time for me to come online to prepare the website for release. This time we agreed I’d be online at 5:00 am (10:00 UTC), aiming for a 12:00 UTC release time.

There’s an element of risk in pre-seeding the CD images because the image testing happens in parallel. Normally its not a problem but this release a serious bug was found. I’m not sure of the details but it was bad enough that there was a “day-of” re-spin. When I came online at 5:00 the disks were being remastered.

This went OK apparently but a new problem popped up. We have a mirror prober application that checks our mirrors to see if they have the correct images. We do this by making HTTP HEAD requests to ensure the file name exists and the file size matches what we have. Unfortunately, the remastered disk images were exactly the same size as the defective ones. Exact to the byte.

We couldn’t tell which mirrors had the new version and which had the bad version. We use a protocol to communicate with our mirrors telling them which files to mirror. Therefore we updated the list to include a dummy file. That allowed us to identify which mirrors were up to date and which were not.

Finally, about 12:21 my time (17:21 UTC) we got the word that it was about time to “push the button.” I had prepared by opening up a bunch of tabs with the pages that were changing, updating the pages so that all I had to do was submit each form. The release team gave me some last minute changes and in the process of applying those I accidentally closed my browser window. If you were in central Iowa you may have heard me scream.

I’m an optimist, so I hoped that firefox would remember the tabs and values of the forms, but unfortunately our authentication system redirects you to an openid auth page when you first visit it. All was lost.

So I copy and pasted again while people constantly instant messaged me asking if it was ready yet. Finally, at about 12:46 (17:46 UTC) the website launched. I kid you not, multitudes of people open their browser to the homepage and refresh again and again waiting to see the site change. I hear someone in the Millbank office with a bottle of champagne waits until the website updates so that they can celebrate. Talk about pressure.

We’re fortunate that our intense loads are so predictable. It gives us plenty of time to beef up our infrastructure and plan appropriately. When we get slashdotted we usually don’t even notice (unless they link to our wiki). A release day is at least 12 times more traffic than when we hit the homepage of digg and slashdot. For a reference, check out this image. The big bump is a release day, the small bump to the left of it is when we hit some big news site.

After the site launches I get a rush of bug reports. There’s nothing like having a million people proof reading your site all at once. The next couple hours are spent doing low-priority updates and fixing little issues that show up. About 3:00 my time (20:00 UTC) I stepped away from my computer and took a nap.

And that is a release day.