Our customers should clearly know where and how their data are kept and what kind of protection guards their precious documents

This is the last part from the security talk series where I’m going to explain you the methods and technologies we use to host and test the web servers of Invoicebus.

When I talk about hosting, the very first things that pop-up in my mind are reliability, speed and redundancy. Three reasons which were plenty enough to choose Rackspace as our back keeper, one of the biggest cloud hosting providers in the world with cutting edge technology and state of the art data centers across the globe. According to Wikipedia its clients include 40% of the members of the Fortune 100 Magazine and its infrastructure platform is 60% bigger than Google’s. But let’s see what actually we are into.

Reliability

The primary data center of Invoicebus is located in Chicago, Illinois, USA and it’s called “ORD1”. To familiarize better, you can visit the gallery and get a quick view of the facility environment. They claim that can provide 100% uptime, so one month before the official launch of Invoicebus, we decided to conduct a test and make sure if the statement is true. We set up the Rackspace server, transfer the domain from the old hosting, re-hosted our early marketing site, and started experimenting.

One of the tools we’ve been using is called Pingdom, an online service that provides analytics for the server up-time, response time and a couple of other stats and tests. We set up to inspect the server repeatedly on one minute intervals by pinging it from 41 different locations around the world. We were checking the stats almost every day for 30 days, but the dashboard constantly was displaying the same boring message: “Last error seen: NEVER”.

The one minute pings are still running, so if something goes wrong, couple of alerts are triggered and sent instantly to our mobile phones via SMS and email. However, during the 94 days period of Invoicebus, we haven’t got any critical alert, so far so good. We decided to share these results with you:

The graphic shows 99.993% of uptime and 10 minutes of downtime due to scheduled regular maintenance of Invoicebus (applying upgrades and new features). Rackspace promise fulfilled.

Speed

To test the speed and response time we used couple of other services. The simplest was Speedtest.net which yielded

350+ Mbit/s Download ( Rackspace claims it’s unlimited, or as much as the test server supports )

( ) 30Mbit/s Upload

constant rates initiated from the ORD1 datacenter, which means from the user’s perspective download and upload would be reversed. However, this was just out of curiosity, because it doesn’t mean anything especially when you deal with concurrent connections from different parts of the world. That’s why, we took in consideration a few other parameters, and most of them depend on the server hardware.

Currently, our server runs on:

Quad-core AMD Opteron 2375HE @ 2.33 GHz

1GB of RAM Memory

Windows Server 2008 R2 64 Bit

To stress test the web server we’ve used a service called Blitz that initiates so called “rushes” – parallel connections generated from multiple points of presence around the world, so you can calculate how much load your web server can handle in a given period.

The Blitz analysis showed the following:

“This rush generated 3,981 successful hits in 1.0 min and we transferred 125.89MB of data in and out of your app. The average hit rate of 63/second translates to about 5,501,902 hits/day.”

This went awesome, but what about response times and error rates?

The error rate was 0.

The response time peak was 1.6 sec at 240users/second, which is pretty high.

The average response time was 733ms which also is considerably higher than most other.

Our goal is to keep the response time below 250ms, which means our upper limit would be 120users per second or 7200 in a minute (if we assume uniform distribution on interval between 0 and 60 seconds). If the number of users / visitors increases more than 120/second, we should upgrade our server probably with additional RAM memory (1GB+), so we can maintain the response time within the desired range and keep our customers bright and happy.

Upgrading the Rackspace server is very straightforward and fast process, but for now, this hardware configuration is plenty enough.



What’s on tap?

Reduce the load

Currently our marketing site and web-app are on the same physical machine, however, as the number of visitors/customers increases, we plan to separate them on different machines and reduce the load.

Cloud files via CDN

Cloud files is a service of Rackspace that provides an online storage for files which can be delivered across the globe at blazing speeds over Akamai’s content delivery network (CDN). This CDN caches content at global edge locations saving users time as the requested content is received from within the region instead of coming from the origin server. Let me explain what it means.

The cloud files keep caches of the web-app files (Images, CSS, JavaScript etc.) at different locations around the world and are served depending on the user’s current location. For example, if the user/visitor is located in India, than the content will be requested from the nearest data server in Asia; or if the user is in Germany, the files will be requested from the data server in Europe, regardless if the origin server is located in Chicago, Illinois, US.

Redundancy

Invoicebus creates database backups on a daily basis and here’s how: The backup procedure is scheduled automatically at a specific time of the day when the traffic is least congested, and the server is minimally engaged. The whole database is archived, encrypted (additionally), and uploaded to cloud flies, thereafter replicated to 8 datacenters across the globe on 3 different continents:

North America: Grapevine, TX;

Richardson, TX; Chicago, IL; Herndon, VA;

Ashburn, VA; Europe: London, UK;

Slough, UK; Asia:

Hong Kong

The data in each of these 8 nodes is mirrored to three storage disks instantly with dual power supplies and round the clock surveillance.

We’ve been keeping backups from the very first day, which means today we have exactly 94 copies of the database for all of our users, regardless if they run a premium or free plan. If the database drastically increases in future, we plan to implement additional logic for creating backups only from the new or modified data, and not the whole data set.





I believe I’ve covered the most interesting aspects and techniques we’ve been using for our hosting and I hope you’ll find some of them useful.

Now it’s your turn.

Have any suggestions or personal experiences with cloud serves?

Share with us your story; we’re eager to hear how you manage your hosting and backups.