Recently, I have found myself in several intense discussions about “cloud testing”: what it is, what it isn’t, and what it means for testing and QA professionals. However, there is much confusion in these discussions around the definition of cloud itself. If you attempt to look up cloud computing on the Internet, you will find it difficult to get a formal definition. Wikipedia says it outright: “Cloud computing is a jargon term without a commonly accepted non-ambiguous scientific or technical definition." Duh!

Since we want to talk about cloud applications and testing thereof in this article, I've tried to distill some common traits for "cloud applications" based on aforementioned and other similar articles. (Let me know if you disagree!)

Common for "cloud applications" is that they

Run on a virtualized hardware and a software stack that can be moved and replicated between physical machines as needed.

Share common physical resources with other cloud applications (disk, network, data stores, etc.).

Are built to be highly scalable in real-time – meaning that they can handle high increases in load by dynamically scaling to more physical resources as needed.

Are predominately accessed using standard network protocols.

Use HTML and other web technologies for providing both their front-end and management UIs.

Provide APIs for integration and management – possibly made available to users or third-party application vendors.

Consume third-party APIs for providing common services and functionality – things like authentication (OpenID, Facebook), storage (DropBox, Google Drive), messaging (Twitter, Gmail), geo-functionality (GooglMaps) etc.

Tend to use NoSQL data stores – primarily for managing large amounts of unstructured data.

The “cloud” itself comes down to being the infrastructure that hosts a “cloud application”; it is usually either public (Amazon, Rackspace, etc.), private, or a combination of the two – and can offer many different levels of service (IaaS, PaaS, SaaS, etc.).

So, given these basic characteristics, what should testers be thinking about when testing a “cloud application” – or more likely, a web application or API that is running “in the cloud”? Are there any specific traits related to a cloud application that mandate extra consideration as opposed to if the application is deployed on an old, dusty, dilapidated server in the corner of your office (running Windows 2000)?

I used to immediately answer this with, "No". A web application or API needs to be tested in the same way no matter how it is deployed. It still has to perform and function as required, and testing doesn't change for different deployment scenarios. Or so I thought – thanks to my colleagues’ persistence, I have started to open up to a more nuanced answer. Perhaps there are a number of aspects that a tester needs to consider carefully when tasked with testing a “cloud application” – many of which are related to the infrastructural nature of the cloud. For example:

Performance – Applications running in a cloud run on hardware that you might not have any control over, and they share with other applications. Therefore, ensuring both performance and required scalability is crucial. Be sure to test performance in a cloud environment similar to the one you will be using in production. If you know that your application shares resources with other applications under your control, run load tests on both at the same time to see if they affect each other. In production, be sure to use monitoring as a means to continuously validate both performance and functionality while your application is in production – ensuring that it scales as required. Using cloud resources to scale under load can be costly, so knowing where that breakpoint is and monitoring to see how close you are to it can also help you budget correctly for your infrastructure needs. Security – Since cloud applications usually share resources and infrastructure with others, you have to give extra consideration to data privacy and access control issues. Is sensitive data encrypted when stored? Are access control mechanisms in place in all possible situations and at all levels? This is just as valid for applications hosted in a private cloud; data intrusion and “theft” could even happen “by accident” if, for example, a backup for one cloud application happens to access resources or data related to another application. Third-party dependencies – Cloud applications are likely to consume external APIs and services for providing some of their functionality. You should consider testing and monitoring these as if they were part of your own solution (since that’s what they are from your users’ point of view). You want to make sure they work as you need them to and you want to be the first to know when they don’t.

One common interpretation of “cloud testing” that many vendors adhere to is using the cloud to run or manage the tests themselves. For example, testers can use the cloud to generate massive distributed load tests, simulate a large number of mobile devices, or run functional and performance monitors from all over the world. Although these offerings are highly valuable, they are not very specific for testing cloud applications. So, calling it “cloud testing” can, in some situations, be a bit of a stretch.

So, did this initial insight help me in understanding what “cloud testing” is? Well, to be honest, not really. Although I do agree that there are things to be aware of when testing an application in the cloud, I still struggle with “cloud testing” being something separate than “regular” performance, integration or security testing.

All of these just need some special consideration and understanding when applied to an application running in the cloud.

Perhaps my words have cleared up the confusion regarding "cloud testing" for you – or perhaps I just created more confusion. Please don't hesitate to share your opinions on what "cloud testing" means for you and your organization. Did you have the same issues I did, or am I prehaps being overly technocratic?

This post originally ran on Ole’s Software Quality column in Network World.

See also: