Functional testing for Web applications

Using Selenium, Windmill, and twill to test Google App Engine applications

As applications move further away from an individually hosted model into the cloud, reliability and predictability become even more important. The cloud moves many factors outside of our control, so having solid, tested code is more important than ever before.

Notes on twill and Google App Engine From Related topics, you can view a more detailed article on getting a "Hello World" application running from scratch. On the OS X platform you can download the Google App Engine Launcher, which will create the boilerplate files for a "Hello World" application for you. To get started using twill, you only need to run the command: sudo easy_install twill . Next, simply invoke the twill-sh scripting tool to interactively test your application.

Most developers, whether they test their code or not, have at least gotten a lecture about testing code at some point. Web developers—more so than most developers—need to deliver applications quickly, so unit testing often takes a backseat to a deadline. In some circles, it is never all right to skip unit testing any code, as a unit test tests actual components of the application and provides a way of explaining the internal workings of the code to other developers. Functionally testing your Web application is quite a different story though, and for some reason hasn't gotten as much of a buzz.

In this article, I explore several different tools that help you perform functional testing on your Web application. I use Google App Engine here, but the testing techniques will apply to any Web application. I will also argue that it is never all right to forgo functional testing, because it is so quick and so easy to perform, at least a minimal level of functional testing. In this article, I explore three functional testing tools: Windmill, Selenium, and twill. Both Windmill, and Selenium are functional Web testing frameworks that allow automation of user interface testing in a Web browser for JavaScript and Asynchronous JavaScript and XML (Ajax) applications. Twill is a lightweight Web scripting tool that deals with non-JavaScript functional testing.

Functional testing with twill

Easy Install Easy Install is a tool that helps install Python packages, either locally or on the network. Most often, the easy_install command is used to install packages from the network.

I'll start the discussion of functional testing with the lightweight command-line Web browser and scripting tool, twill, and a default Google App Engine project.

The first thing you do is establish a connection to your application. To do that, use the go command, as shown in Listing 1. Note, if you enter show , it then shows the actual output.

Listing 1. Sample show output

# twill-sh -= Welcome to twill! =- >> go localhost:8087 ==> at http://localhost:8087 current page: http://localhost:8087 >> show Hello World! current page: http://localhost:8087

Another interesting feature of twill is its ability to check http status codes.

Listing 2. Sample twill output of http status codes.

> go http://localhost:8087 ==> at http://localhost:8087 current page: http://localhost:8087 >> code 200 current page: http://localhost:8087 >> code 400 ERROR: code is 200 != 400 current page: http://localhost:8087

As you can see from the output of the command, it only returns an error if it gets a status code that doesn't match what is expected. Twill also supports the ability to run these actions as a script. You can name the file anything you want and pass it to twill-sh . If you place those commands in a file called test_twill.script, you will see something like Listing 3.

Listing 3. More sample twill output

# twill-sh test_twill.script >> EXECUTING FILE test_twill.script AT LINE: test_twill.script:0 ==> at http://localhost:8087 AT LINE: test_twill.script:1 -- 1 of 1 files SUCCEEDED.

Twill is a handy tool for automated testing of the non-JavaScript portions of your Web application, and it has more features than what I covered, such as the ability to work with variables, cookies, forms, http authentication, and more. If you would like to know about more advanced use cases, see Related topics.

Functional testing with Selenium

Selenium is a heavier-weight testing tool that allows cross-platform testing in the browser. Writing cross-platform JavaScript code is a regrettable cross that Web developers must bear. Writing Web applications is difficult enough, and just when you think you're done, you inevitably run into some obscure bug that is only present on one browser.

Unfortunately, a unit test does not catch this type of bug. In fact, these bugs often make a Web developer skeptical of doing any testing in the first place. They figure that testing is yet another hoop to jump through, that it gets in the way of their already tight deadline, and it doesn't even reliably work. So why bother?

Selenium (as well as other browser testing tools) is an answer to this problem. You can write functional tests that run in each browser, and then implement some form of continuous integration system that runs the functional tests upon each check-in of the source code. This allows potential bugs in the browser to quickly get caught, and has an immediate payoff.

One of the most basic things Selenium does is record browser actions so they can be replayed as a test. Looking at the sample in Figure 1, you see a window where you place a base url. From there you simply record the actions of the Web site you are testing. In this case, I am testing a Google App Engine site: http://shell.appspot.com, which is a demo Ajax Python interpreter. After the session has been recorded, you can then export the tests in Python and run them using Selenium.

Figure 1. Selenium IDE window

To run the tests, simply save your tests as Python code (or whatever language you choose), download and run the Selenium RC test server, and then run your tests. Listing 4 shows an example of what that test might look like.

Listing 4. Example Selenium test

from selenium import selenium import unittest, time, re class NewTest(unittest.TestCase): def setUp(self): self.verificationErrors = [] self.selenium = selenium("localhost", 4444, "*chrome", "http://shell.appspot.com") self.selenium.start() def test_new(self): sel = self.selenium sel.open("/") sel.click("link=source") sel.wait_for_page_to_load("30000") def tearDown(self): self.selenium.stop() self.assertEqual([], self.verificationErrors) if __name__ == "__main__": unittest.main()

You can launch Selenium RC, which acts as a proxy for testing, on multiple browsers, and then run your functional test. Listing 5 shows what the output of Selenium RC looks like.

Listing 5. Sample of Selenium RC output

# java -jar selenium-server.jar 01:18:47.909 INFO - Java: Apple Inc. 1.5.0_16-133 01:18:47.910 INFO - OS: Mac OS X 10.5.6 i386 01:18:47.915 INFO - v1.0-beta-1 [2201], [1994] 01:18:48.044 INFO - Version Jetty/5.1.x 01:18:48.045 INFO - Started HttpContext[/,/] 01:18:48.047 INFO - Started HttpContext[/selenium-server] 01:18:48.047 INFO - Started HttpContext[/selenium-server/driver] 01:18:48.055 INFO - Started SocketListener on 0.0.0.0:4444 [output suppressed for space]

I encourage you to read the full Selenium RC FAQ to understand how it interacts with multiple browsers (see Related topics). As you can see, using Selenium to automate cross-platform functional tests is quite easy, and there is support for many languages including plain HTML, Java™ code, C#, Perl, PHP, Python, and Ruby.

Windmill

Windmill is a Web testing framework that is very similar to Selenium, although there are some differences. One main difference is that Windmill is written in Python and JavaScript, and was originally developed to test the Chandler Web client, an ambitious open source competitor to Microsoft® Outlook, but has continued on as an independent open source project.

To get started using Windmill, simply run the command: sudo easy_install windmill . This installs the windmill testing framework. Next, if you enter windmill firefox , the Windmill IDE opens (see Figure 2). Then the testing page opens, as shown in Figure 3. In Figure 2 you can see that the IDE records actions much like Selenium. You can then save the test file, which looks like the output in Listing 6.

Figure 2. Windmill IDE window

Figure 3. Windmill tutorial

Listing 6. Sample test file output

# Generated by the windmill services transformer from windmill.authoring import WindmillTestClient def test_recordingSuite0(): client = WindmillTestClient(__name__) client.click(id=u'recordedClickId') client.click(id=u'textFieldOne') client.type(text=u'foo bar', id=u'textFieldOne') client.click(id=u'btnSub')

There is an option to save a test as JSON or Python, and in this case I save it as Python. Next, to actually run the test, you simply run the test file from the command line with the test option. Listing 7 shows how this looks.

Listing 7. Running the test

# windmill firefox test=windmill_test.py http://tutorial.getwindmill.com/ Started ['/Applications/Firefox.app/Contents/MacOS/firefox-bin', '-profile', '/var/folders/1K/ 1KgyCzqJHButzT6vq8vwHU+++TI/-Tmp-/tmp_ovtnN.mozrunner', 'http://tutorial.getwindmill.com/windmill-serv/start.html'] Server running...

You can easily adapt this example to your Google App Engine application or any other Web application, for that matter. Windmill also has great documentation that explains how to do more advanced tasks such as extending windmill with plug-ins.

Conclusion

Functional testing is essential to the process of Web development. Without functional testing, Web development becomes a guessing game that can be filled with frantic, error-prone deployments and refactoring.

So, should functional testing be mandatory for all Web developers? I would argue, yes, all Web applications should be tested, especially if they are destined for a cloud environment. Lack of functional testing for Web applications is a definite red flag, considering how easy it is to doâat least a minimum of testingâwith Selenium, Windmill, or twill. To borrow a tag line from the author of twill, Dr. Titus Brown, "if you don't test your code, how you can say it works?".

Downloadable resources

Related topics