Why we moved away from “the cloud” to a “real” server

By David Mytton,

CEO & Founder of Server Density.

Published on the 8th September, 2009.

Up until the end of August, our server monitoring service, Server Density, was hosted in “the cloud”. We had no physical hardware and were using virtualised instances provided by Slicehost and The Rackspace Cloud. This allowed us to start very small and cheap and expand as the service grew – we were paying for what we used, something which is ideal for the first stages of a startup. Everything worked very well and we had few problems.

Progression

Back in Feb 2009 we started out with a 256MB Slicehost instance for only $20 per month. By the end of August we had a 2GB Slicehost instance and an 8GB Rackspace Cloud instance, costing $130 and $345 respectively. We also had a 256MB Rackspace Cloud instance for a VPN we used to access the servers through at $10 per month. The total expenditure was therefore $485 per month.

The Rackspace Cloud options only became available after we had been using Slicehost for several months and given they were half the price, we decided to expand our database server using them rather than Slicehost. The reason for the price difference is Slicehost include a data transfer allowance whereas you pay per GB with The Rackspace Cloud; since most of our transfer is incoming, that allowed us to save substantially.

Slicehost is also owned by Rackspace and they run in the same data centre which meant we were able to use the internal network to communicate between our Slicehost and Rackspace Cloud instances. There was no disruption to the service as we kept our application serving with Slicehost and the database with The Rackspace Cloud.

The disk space problem

The problem we were facing was disk space. Since we store a lot of historical monitoring data, we needed a large amount of storage. The only way we could increase our available storage was to increase the instance size. This was not scalable and so we set about looking at the alternatives.

From a technical interest point of view, as well as scalability, we wanted to move to Amazon EC2. It would have been very cool to deploy on top of a virtualised environment that we could load balance, implement a failover system and make use of the Elastic Block Storage for our data storage needs.

Although we needed little instance storage, our database needs to run 64bit and so we would have had to go with the large EC2 instance. Our pricing calculations looked like this:

Large EC2 Instance: $292

Transfer In: $15 (150GB)

Transfer Out: $4 (20GB)

Elastic Block Storage: $90 (360GB inc backup)

Elastic Block Storage Requests: $30 (300 million, based on tests)

The total cost of which is $431, but we also wanted their support services which are priced at $100, so a total of $531 USD per month. The Elastic Block Storage requests per second is a big variable that is hard to predict. We ran our database on an instance for 24 hours and estimated that we would be using at least 10 million i/o requests per day.

Hardware is cheap

There are also technical complexities in that you have to provision EBS storage before you use it, and if you need to resize then you have to take a snapshot and rebuild from that.

We would have had to build our own infrastructure management system. This would have to handle the starting and failover of EC2 instances, backups and provisioning additional storage. In order to have no downtime when storage was being re-provisioned, we would need a second instance to replicate the database on. Aside from the extra server costs, all of this would have taken development time away from our efforts of improving the product itself.

We had the choice of working on infrastructure that makes no difference to the customer experience (but which would have been technically interesting and fun to develop) verses tangible progress with our product:

With six years of experience running my own software company I can tell you that nothing we have ever done at Fog Creek has increased our revenue more than releasing a new version with more features. Nothing. The flow to our bottom line from new versions with new features is absolutely undeniable.

– Joel Spolsky

And that’s the hidden cost – development time – something that is particularly important for a startup. As Jeff Atwood says, “Hardware is Cheap, Programmers are Expensive“.

Given the situation, the move we made was to get a physical server with Rackspace. Although there was a cost increase compared to the existing solution, it was not much more expensive than the move to Amazon EC2 would have been, and there were no development costs. Since Slicehost and The Rackspace Cloud are in the same data centre as our current servers, the move was very easy.

Uptime and security

As a server monitoring product, the service needs to be available. There is no such thing as 100% uptime but Rackspace is known for its reliable network and systems.

Their support was another factor in the decision making process. Rackspace know how our systems are set up and, like us, continuously monitor them so that if there is an issue, we can work together to fix the problem as quickly as possible. Their support is indeed “Fanatical” – they even swapped out a lower Cisco firewall model for a newer one for free so we could access our servers through a secure VPN on our iPhones.

You can also get an amazing deal by negotiating with the sales team. Cloud pricing is fixed but moving to Rackspace got us amazing value for money for the hardware we have. And now we have an existing relationship, future upgrades can be negotiated.

Further, since we process customer card transactions through our servers (we collect details on our site but do not store them ourselves), we have to be PCI compliant, something that Amazon EC2 is not.

Not entirely clear skies

That said, we are making use of some “cloud” services. Our server has large internal storage but we also make use of the Rackspace Utility Network Attached Storage product. This allows us to scale disk space indefinitely and pay per GB we use. Unlike Amazon EBS, it really is per GB, not per provisioned storage. This has saved a lot of development time working out how to deploy our database across multiple disks or handle resizing existing volumes.

Frequent review

We are likely going to be a Rackspace customer for life. Their support is amazing and we know we can rely on them to deliver the service we would expect as customers. However as we grow we will frequently review the situation to see if savings can be made without sacrificing unnecessary development time and quality of service. Even if not for the primary hosting infrastructure, for other aspects of our systems – for example, we are currently looking into how we can use The Rackspace Cloud to run off-site database replication for further redundancy in addition to our current backups.

“The cloud” has its uses, especially when starting up, but it is not always the best option.