Accelerating Web Page Analyzer

The free Web Page Analyzer tool we provide has become popular (see Figure 1). In fact, it has become so popular that the traffic and load caused by the analyzer have started to effect the response times of the rest of the site. As the average web page has grown over the years, the analyzer finds itself working harder, dealing with larger and more complex pages. The size of the average web page has more than tripled in the last five years, and the number of objects has nearly doubled. The tool is used hundreds of thousands of times each month, grabbing and analyzing web pages for users testing sites. All of these analyses of growing web pages add up to many HTTP requests. The tool provides some side benefits for WSO, giving us leads and links, but it comes at a price since it is currently hosted on the same server as the site proper.

Figure 1: Web Page Analyzer 0.98 Screenshot

While we've updated the software of the analyzer over the years to improve performance (moving from Perl to PHP for example), this hardware upgrade focused on improving the performance of our dedicated server. We'll address the software updates, such as CAPTCHA and throttling, that we added in a future tweak.

Server Upgrade Performance Goals

The goal with this upgrade was to improve the response times of both the site and the web page analyzer tool. In our previous tests we found that the large number of requests required to grab and analyze web pages was memory and processor intensive. Before upgrading, our server had the following specifications:

WebsiteOptimization.com Dedicated Server Specifications Circa September 2008:

Intel P4 3.0 Ghz Dual

6 GB RAM

2x160 GB SATAII Harddrives in Raid 0+1

100 Mbps NIC.

Enterprise Linux 4

Some users were reporting slow response times when using the tool during peak hours. We benchmarked the tool using Pylot, which gave the response time results shown below (see Figure 2).

Figure 2: Response Time Graph - Before Server Upgrade

Figure 2 shows the results from Pylot load testing the site by ramping up responses per second and agents. The response time averaged 17.97 seconds, and once ramped up to 2 or more requests per second the response time averaged about 47 seconds! This jibed with our other tests of the Web Page Analyzer during higher loads, the response time would average 30-40 seconds to return results. Figure 3 shows the corresponding throughput for the initial benchmark test of the old server.

Figure 3: Throughput Graph - Before Server Upgrade

Upgraded WebsiteOptimization Server

In an effort to improve response times we upgraded to a new server with faster processors (and more cores, from 2 to 8), faster hard drives (from ATA to solid-state drives in a RAID 1), and twice the bandwidth. The specifications of the new server were as follows:

WebsiteOptimization.com Upgraded Dedicated Server Specifications Circa October 2008

Processor: Dual Xeon E5310 Quad Core (Clovertown)

Memory: 4GB DDR Registered ECC Hd1 RAM

Drives: Dual 32GB SSD (solid state drives) Hardware Raid 1

Hd2: 120GB 7200RPM SATA / 8MB Cache RemoteBackup

OS: Linux - CentOS 4 (+64 Bit)

This new server setup promised to be a screamer, with solid-state drives that could theoretically be orders of magnitude faster than SCSI or ATA drives for some operations. The only question was the amount of memory that went down from 6 to 4GB, but everything else had improved. The ATA drives in particular are slower to seek and deliver data as they work sequentially, not in parallel like SCSI drives can. Solid state drives are relatively new, so we chose to configure a RAID 1 of two of them, in case one failed (highly unlikely). The new sever benchmarked as follows, tested at the same time of day under similar loads (see Figure 4).

Figure 4: Response Time Graph - After Server Upgrade

Figure 4 shows the results from Pylot load testing the new server by ramping up responses per second and agents. The response time averaged 11.87 seconds (34% faster), and once ramped up to 2 or more requests per second about 30 seconds, about 36% faster. In timed tests after the upgrade the analyzer itself ran about twice as fast when analyzing large home pages like CNN.com. The difference was not as significant for smaller pages such as Useit.com. Figure 5 shows the corresponding throughput for the initial benchmark test of the new server.

Figure 5: Throughput Graph - After Server Upgrade

Conclusion

Upgrading your dedicated server can make a significant difference in response times for your website. However, the performance improvement of your site will not be linearly related to the performance improvement of your processors or hard drives.

By upgrading our server to have more and faster processors and faster hard drives, we realized about a 35% improvement in performance. Even though the drives were 10 to 100 or more times faster than normal ATA or SCSI drives for some operations and the effective number of processors grew by a factor of 4 (not to mention the clock speed) the overall performance of our site and tool grew by 35% to 50% respectively. Next, we plan on testing a separate server to host the analyzer tool to see the effects on the main website.

Futher Reading