By: Jim Armstrong

All clouds are not created equal and when you choose to hand over your core infrastructure to a public cloud provider it is important to know exactly what you are getting from that service, both in terms of price and performance. VMware is sharing benchmarking results showing that VMware vCloud Air customers get 20-40% greater compute performance from their applications running on VMware vCloud Air than when those applications are running on comparably sized instances on Microsoft Azure and Amazon Web Services while being cost competitive to both. The testing methodology and summary results are presented here (a full whitepaper will be available soon).

With the adoption of cloud there is a shift in IT roles from capacity management to budget and performance management. Many of the customers we talk to today are looking at multiple cloud providers but it seems every major cloud solution has a slightly different set of pricing metrics. On top of this, the very essence of “cloud” is that you no longer have to worry about the hardware, but this leads many new cloud customers to questions concerning performance. “If my application currently runs on a VM with 2 CPUs and 8 GB of RAM, which VM should I choose from my cloud vendor? Will that VM on vendor A perform the same as a similar VM on vendor B?” The performance variable usually appears after testing begins, or worse, after the application is put in to production and the first major spike in utilization hits. Once it does, you are left with another choice: do you scale out with more VMs? Or scale up the initial VM to a larger size? Or do you “shoot” your underperforming VM and start another one in the hopes that you just got unlucky with the placement of the first VM? All of these choices have implications not just for performance but also for cost: more VMs or bigger VMs cost more money.

In order to test performance VMware recently commissioned Principled Technologies to perform CPU benchmark testing on the latest generation of compute services from VMware, Amazon Web Services (AWS), and Microsoft Azure. For this round of tests, we looked at VMware vCloud Air VPC OnDemand, the new M4 series of instances on AWS, and the D series instances on Azure. We chose two VM sizes for testing, aligning the instance sizes with typical VM sizes our customers might use on-premises: 2 vCPUs x 8 GB vRAM and 4 vCPUs x 16 GB vRAM. Unlike previous generation AWS instances, the new M4 instance sizes align with these VM sizes and others we see customers running on-premises today, making direct comparisons easier to run. Azure’s “t-shirt size” instances still do not perfectly align in vCPU and memory, however they offer instances that are close approximations. In contrast, VMware vCloud Air users can create virtual machines of any configuration they need and “hot” add compute resources to those configurations without having to shut off the virtual machine (assuming the guest OS supports hot-adding resources). The table below shows the virtual machine configurations tested:



Using a common Linux guest operating system and the industry standard SPEC CPU benchmark, we looked at both integer and floating point performance with the following results (higher numbers are better):

Comparing Performance and Cost of Similar Virtual Machines

While performance varies across these three platforms, so does price. Below are the list prices per hour for the following VMs directly from each vendor’s website [i], [ii],[iii]:

[Note: Azure D2 and D3 instances include 100 and 200 GB of SSD, respectively, so we added similar storage to the base AWS and VMware vCloud Air instances as well]

If we simply consider the list price per VM the VMware vCloud Air instances are competitively priced but a little higher than both AWS and Azure. However this simple pricing exercise ignores the effect of performance. When we run applications in the cloud, performance differences show up as a need for a greater number of VMs or larger VMs to perform the same amount of work so total cost comparisons should also account for these performance differences. To put it another way, if it takes 13 VMs in one public cloud system to produce the same work as 10 VMs in VMware vCloud Air we need to account for this in the total cost of running that application. It is all about price/performance, and not just price alone.

The chart below summarizes this comparison using a hypothetical 10 VM application on VMware vCloud Air. Our performance comparisons show that to produce the same performance on AWS you would need approximately 30% more VMs, 13 VMs total, and on Azure you would need about 20% more VMs, or 12 total, to get the same performance as 10 VMs running that application on VMware vCloud Air. Looking at it this way, VMware vCloud Air pricing is very close to AWS and almost 10% lower than the price of Azure.

There are other cost benefits that do not show up in simple pricing comparisons, too, including:

Flexible VM sizing : As mentioned earlier, VMware vCloud Air customers have the freedom to select any configuration VM meaning less “wasted” resources in choosing an instance. If you find your application only really uses 6 GB of RAM you can resize your VM to fit, and thus lower your cost. Need extra RAM for a short period of time? Size up the VM for a short period and then cut the RAM when it is no longer needed.

: As mentioned earlier, VMware vCloud Air customers have the freedom to select any configuration VM meaning less “wasted” resources in choosing an instance. If you find your application only really uses 6 GB of RAM you can resize your VM to fit, and thus lower your cost. Need extra RAM for a short period of time? Size up the VM for a short period and then cut the RAM when it is no longer needed. Metering by the minute : Along with Azure, VMware vCloud Air VPC OnDemand only meters for resources used by the minute. AWS meters by the hour, which can make a significant cost difference over the course of a month. Thinking back to the RAM sizing example above, if we increase the RAM for 30 minutes and then reduce it, VMware vCloud Air only charges for 30 minutes of burst capacity. On AWS you would be charged for a full hour of the larger VM size. And with VMware vCloud Air you could hot add the RAM to your VM, reducing the need for downtime. On AWS you would have to shut down your VM and attach the storage to a larger VM to resize.

: Along with Azure, VMware vCloud Air VPC OnDemand only meters for resources used by the minute. AWS meters by the hour, which can make a significant cost difference over the course of a month. Thinking back to the RAM sizing example above, if we increase the RAM for 30 minutes and then reduce it, VMware vCloud Air only charges for 30 minutes of burst capacity. On AWS you would be charged for a full hour of the larger VM size. And with VMware vCloud Air you could hot add the RAM to your VM, reducing the need for downtime. On AWS you would have to shut down your VM and attach the storage to a larger VM to resize. Built-in HA capacity and live migration : Every VMware vCloud Air environment includes spare HA capacity for free. That means that every single virtual machine is protected against failure with an automatic restart of the virtual machine. We also leverage vMotion to migrate running applications between hosts during maintenance windows and to optimize performance. This enables VMware vCloud Air to offer SLAs for VM up-time, and not just service up-time like other cloud providers.

: Every VMware vCloud Air environment includes spare HA capacity for free. That means that every single virtual machine is protected against failure with an automatic restart of the virtual machine. We also leverage vMotion to migrate running applications between hosts during maintenance windows and to optimize performance. This enables VMware vCloud Air to offer SLAs for VM up-time, and not just service up-time like other cloud providers. Compatibility with your on-premises workloads: Unlike the other clouds that run their own hypervisors underneath, VMware vCloud Air is built on vSphere, making it fully compatible with the workloads you’re running on-premises today.

An Enterprise-Ready Platform

These benchmark results further validate that vCloud Air is an optimal platform for the toughest, most intensive workloads out there. Built on vSphere, the world’s leading virtualization solution, the underlying technology behind vCloud Air is battle tested against millions of virtual machines running in our customers’ data centers. We’ve honed and improved vSphere over many years to get near bare-metal performance over a wide variety of workloads, even new kinds of workloads like containers (there is an interesting article comparing Docker Containers on vSphere 6.0 here). And with vCloud Air, we are providing an environment that can extend this performance to the cloud.

Further Reading:

Principled Technologies, “CPU Performance Comparison: VMware vCloud Air, Microsoft Azure, and Amazon Web Services.” Aug 15, 2015.

Jim is a product marketing manager responsible for competitive analysis for vCloud Air. He has been in the software and hardware industry for 19 years, as a system engineer and marketing manager working with a range of customers and partners on infrastructure, operations, and security projects.

If you’re ready to get started with the hybrid cloud, visit vCloud.VMware.com.

Be sure to check out the vCloud Air Community, where you can join or start a discussion, watch our latest vTech Talk video, enter for a change to win swag in our monthly giveaways and more. Get started here!

For future updates, follow us on Twitter and Facebook at @vCloud and Facebook.com/VMwarevCloud.

_bizo_data_partner_id = “7871”;

(function() {

var s = document.getElementsByTagName(“script”)[0];

var b = document.createElement(“script”);

b.type = “text/javascript”;

b.async = true;

b.src = (window.location.protocol === “https:” ? “https://sjs” : “http://js”) + “.bizographics.com/insight.min.js”;

s.parentNode.insertBefore(b, s);

})();