The server meltdown

By 11:30am we were averaging 400+ concurrent people on our Halloween page, shocks were delivered every 20 seconds and the stream of tweets was constant.

We proactively monitor our servers which we host with Rackspace, by midday the web services were using up all of their allocated RAM trying to deal with the traffic. Snapshot images of us being shocked were slow to come through and for some visitors, our site didn’t load at all in time. We had to distribute the load and fast.

Oli, who handles our servers and networks, decided that rolling out a load balancer would be our best option as we could add more servers where needed. To do this, we needed to move people from our main site (sidigital.co/halloween) to a new address. The new address would be serviced by a load balancer, which would then send that load to a minimum of 2 new servers. As the team were coming up to lunch, it was decided to take it early so this new solution could be rolled out as quick as possible.

Oli (from Si digital) and Oli (from Rackspace) discussed the load balancing option and how we would go about doing it, including any issues with deploying our saved server images. It’s comforting to know that we have these resources at hand. By hosting with Rackspace, it provides an extra level of confidence.

Within 80 minutes we had 2 servers and a load balancer servicing a new domain to send people to (halloween.sidigital.co). It was by far the quickest production server deployment we had ever done.

After switching visitors to the new domain, we noticed a significant speedup on our primary server, which was still dealing with image uploads from our workstation webcams. Everything had smoothed out and the 400 concurrent visitors were able to see the site in all its glory. Rackspace then tweeted about what we were up to, driving a load more traffic to the servers – just to make sure!

The Twitter ban

One great feature that only 11 people got to see was a tweet from our halloween bot with a direct link to the shock photo. We setup an account https://twitter.com/sidgtl_shock that would tweet the user the photo after the shock occurred.

Unfortunately, despite testing this feature without issues, Twitter suspended the account only 5 minutes after launch for sending “unsolicited tweets”. We can understand the reasoning, Twitter had no way of telling that this wasn’t spam and hopefully for our next experiment we can find a way around this!