Test automation and continuous integration with STAF/STAX

Shorten development time and boost software quality with a STAF/STAX framework

About regression and incremental tests

Regression testing intends to ensure that software changes do not introduce new problems, or failures, in a software. Also, it ensures that you don't reintroduce previously fixed bugs. The introduction of new problems is especially common if the software has many dependencies to third-party components and libraries, like XML processing APIs or database abstraction APIs like JPA. In these cases, a subtle change in the database schema might cause a behavior change in the application that is difficult to predict. If the software is inherently complex, one small change in a common piece of code can cause unforeseen side effects.

A scenario that requires continued support of previous versions of a specific middleware software, or different platforms, sharpens the probability that you might introduce problems during maintenance or development. A fix or function that works in one version of the database middleware might have a disastrous outcome in a different version. Also, when you add functions, an API available only for newer versions of the middleware might break clients that use previous versions. When you support different platforms, thoroughly test functions dependent on command-line interfaces or scripts. Otherwise different behaviors can arise in different operating systems like Windows® and Linux®.

Regression tests aim at the software as a whole, and consider its interdependencies with other components or applications. During development, focus on the specific modules or functions under development. You can incrementally test modules or functions as you implement them, and sometimes use stubs that refer to other functions not yet implemented. If incremental tests are sufficiently structured and independent that you can repeat them given a certain environment (such as required middleware software, machines, or configuration), you can reuse them in future regression tests. This structure and its independence reduce rework and improve testing quality as every piece of code is tested with the same scrutiny used when it was developed.

Consider automated incremental tests when you test the function under development on a set of architectures, operating systems, and middleware versions. For example, imagine a command-line interface that is supported for Linux, AIX®, Windows 7 and Windows XP. During development, the execution of tests in each one of these operating systems might take considerable resources and time. In some cases, the time that is required to test the new feature can overcome the time necessary to develop it. Considering that many features are concurrently developed in a project by different developers, this scenario becomes even more difficult. An added complication is that there are normally a limited number of machines, even if virtual ones, for testing. Often, tests require a pre-configured environment, including database servers, application servers, and more. The more machines that are dedicated for testing, the more time spent updating and configuring each of them. With automated incremental tests run by a tool in a set of environments, or test bed, you save the time that is needed to manually run a test on each environment, and allow better sharing of resources.

Automated regression tests can use the same test bed as incremental tests. By setting regression tests to run once a day, for example, you do not consume resources that you might otherwise use for development or incremental testing. You can set lengthy regression tests to run at longer intervals, such as during weekends.

STAF/STAX Framework

The Software Testing Automation Framework (STAF), an open source framework, contains a set of built-in services that are specially suited for the construction of an automation solution. Services are reusable components that provide all the capability in STAF. Each STAF service provides a specific set of functions such as:

Logging

File system operations

Security and process invocation

Definition of a set of requests

Additional services, such as Email Service, FTP Service, and Timer Service, among many others, are not built-in, but are easily plugged into the framework.

STAX service is a special service that is implemented in a Java™ environment, or execution engine, that allows the invocation of other STAF services through XML. It also brings support for:

Parallel execution

User-defined granularity of execution control

Support for nested test cases

The ability to control the length of execution time

The ability to import modules at runtime

Support for existing Python and Java modules and packages

The XML elements that are defined by STAX work like a programming language, containing all the essential elements of a language:

Loops

Conditions

Error handling

Function invocations

The element script can be used to embed Python scripts, allowing more complex logic and access to predefined Python modules.

The Python code is run by Jython. Jython takes the Python programming language syntax and enables it to run on the Java platform. This allows integration with the use of Java libraries and other Java applications, which increases productivity in a team of Java developers.

One interesting feature of STAX is it allows importation of existing STAX files that contain auxiliary functions. This feature improves organization of automation code and reuse of code.

STAF runs as a daemon, listening to requests from a pre-configured port. The STAF services are accessed through these requests, which are instructions for required commands. Stafcmd is the STAX-defined element that allows use to represent a request. A request, as in Listing 1, contains:

The location where the command must run

The service that runs it

The actual command

Listing 1. An example of a request

<stafcmd> <location>'lab01.mydomain.com'</location> <service>'FS'</service> <request>'CREATE DIRECTORY /tmp/CVT_TEMP' </request> </stafcmd>

The code in Listing 1 requests the creation of directory /tmp/CVT_TEMP on machine lab01.mydomain.com. The service that defines this command is 'FS' (File System service). If the location where the command must be run is not 'local', the STAF instance that requests the command sends the request to the specified location. This communication is not apparent to the caller. A request for creating a directory in the local machine has the same syntax. The only difference is that you specify the location 'local'. STAF then understands that the command must be run locally, not in other machine.

Using STAF/STAX for test automation

The steps that are needed to automate the execution of tests depend on the software that you test. The main steps of most automation solutions are to:

Prepare the environment for the test case Create remote directories Copy required images and prerequisites to remote environments Place or uncompress images and prerequisites in the correct locations Start database or application servers Create a database Install the application Run the test case Collect information (results, coverage information, logs) Free up resources Delete temporary folders Drop database Build reports with results (JUnit, Coverage or HTML reports) Store the results locally or in a remote server (using FTP, a shared folder, or other tools) Send notifications (emails)

You can use STAF/STAX to perform all of these steps, as seen in the following examples.

Prepare environment for the test case

One way to organize your testbed is to create a property file for each environment, where you would place all properties that make sense for your test cases, such as:

Database ports

Server ports

Software installation directories

Any other properties that are needed by the test cases

Each property file represents a test environment that is on a specific machine. A machine can have more than one environment. For example, one environment might define a DB2 database installation and a WebSphere v8.5 server. Another environment on the same machine contains the properties of an Oracle installation with a WebSphere v7.0 server. Each property file represents a test environment that is on a specific machine.

In the preparation phase, the developer must iterate over the set of environments, and start a set of commands to prepare for the execution of tests in the remote machine where each environment is defined. Listing 2 shows a basic loop over the environments:

Listing 2. Basic loop over

<script> env_list = ['LinuxDB2Websphere8','WindowsDB2Webshere85'] </script> <paralleliterate var="env_name" in="env_list"> <sequence> <message log="STAXMessageLog"> 'Preparing to launch tests on %s' % env_name </message> <tcstatus result="'info'"> 'Reading environment information from %s' % env_name </tcstatus> <script> file_path = "%s/CVT/config/%s.properties" % (temp_dir, env_name) </script> <call function="'read_properties'"> {'properties_file' : file_path} </call> <script> props_dict = STAXResult try: mach_name = props_dict.get('STAF_MACHINE') except: raise Exception('Property STAF_MACHINE not found in Env %s' % env_name) </script> … prepare remote environment for tests </sequence> </paralleliterate>

Create remote directories

The preparation for the execution of tests involves the creation of one or more directories where both the STAX scripts, as necessary libraries or product images are placed. One suggested structure is:

<TEMP_DIR>/CVT_TEMP/PRODUCT → directory where libraries and required images are placed.

→ directory where libraries and required images are placed. <TEMP_DIR>/CVT_TEMP/CVT → auxiliary STAX scripts and other resources necessary for testing.

To retrieve the location of the system temporary directory, a STAF variable with this value must be resolved, as in Listing 3.

Listing 3. Resolution of STAF variable

<stafcmd> <location>mach_name</location> <service>'VAR'</service> <request>'RESOLVE STRING {STAF/Env/TEMP}'</request> </stafcmd> <script> temp_dir = STAFResult product_dir = 'CVT_TEMP/PRODUCT/%s' % temp_dir </script>

The directory is created with the file system service (FS), as in Listing 4.

Listing 4. Creating a directory

<stafcmd> <location>mach_name</location> <service>'FS'</service> <request>'CREATE DIRECTORY %s FULLPATH' % (product_dir)</request> </stafcmd>

Copy required images and prerequisites to remote environments

When you prepare the remote environment, copy compressed images and prerequisites, and then extract them at the destination directories. To copy a file, use the COPY request in Listing 5:

Listing 5. Copying compressed images and prerequisites

<stafcmd> <location>'LOCAL'</location> <service>'FS'</service> <request>'COPY FILE %s/%s TODIRECTORY %s TOMACHINE %s' % \ (local_directory, zip_file, remote_temp_dir, mach_name)</request> </stafcmd>

Place or uncompress images and prerequisites in the correct locations

For uncompressing the image, use the ZIP service in Listing 6:

Listing 6. Uncompressing an image

<stafcmd name="'Unzip %s on %s' % (zip_file, mach_name)"> <location>mach_name</location> <service>'ZIP'</service> <request>'UNZIP ZIPFILE %s/%s TODIRECTORY %s REPLACE' % \ (source_directory, zip_file, target_directory)</request> </stafcmd>

Start database or application servers

The process element represents a STAF process that is run on a specified machine. It can be used to start a script or executable code and wait for it to complete. Among other options, you can specify:

The command itself

Its parameters

The working directory

Environment variables that are used by the process

See Listing 7.

Listing 7. A process element

<script> ... cmd = 'C:/PROGRA~1/IBM/WebSphere85/profiles/Dmgr01/bin/startManager.bat' parms = '-username %s -password %s' % (user, passwd) </script> <process name="'cmd %s %s' % (cmd, parms)"> <location>mach_name</location> <command mode="'shell'">cmd</command> <parms>parms</parms> <workdir>work_dir</workdir> <envs>envs</envs> <console use="console" /> <stdout>stdoutFile</stdout> <stderr>stderrFile</stderr> <returnstdout /> <returnstderr /> </process>

The process element provides the basic tool to run a script or executable code. When you run dozens of different scripts during a test execution, it is advisable to create an auxiliary STAX function to wrap the call, log the command that ran plus its parameters, and check exit codes.

Create a database

The process element can also be used to start a script that creates a database. The code in Listing 8 creates a DB2 database:

Listing 8. Creation of a DB2 database

<script> script_name = 'db2Setup.bat' db_script_dir = 'C:/CVT_TEMP/PRODUCT/bin' cmd_str = 'db2cmd -w -i -c %s %s -d %s -u %s -p %s' % \ (script_name, migrate, db_name, db_user, db_pswd) </script> <process name="'cmd %s %s' % (cmd, parms)"> <location>mach_name</location> <command mode="mode">cmd</command> <workdir>db_script_dir</workdir> <console use="console" /> <returnstdout /> <returnstderr /> </process>

Install the application

The installation process normally involves calling a script or executable code in non-interactive mode. The process is similar to how you called the scripts that set up the database, as in Listing 8.

Run the test case

Although STAF/STAX is useful for setting up the test environment, the test code (STAX and python scripts) becomes complex, or repetitive when you test variations of the same scenario. This complexity makes maintenance difficult, especially if not all testers and developers have deep programming skills. While STAF/STAX provides a solid foundation framework, you might reuse previous tests written in other languages, and consolidate reporting tools instead of requiring testers to scan through sometimes non-intuitive STAX logs. For the Java language, JUnit is the most commonly used testing framework. JUnit is integrated with most development software, and allows for the use of interesting related frameworks that increase the productivity of writing test code. For example, you can separate test logic from test data with DDTUnit. JUnit reports generated by ANT are also useful for organizing reports and finding failures in a test case.

Calling JUnit from a STAX script

Although you can start JUnit directly as a simple Java class, it is easier to call it by using ANT. The JUnit ANT task encapsulates the call to JUnit, sets the class path, and records the execution of the test in an XML file format. This XML file can then be used by another task called junitreport to generate a report in an HTML format.

The function run_junit_ant, defined in the sample file JUnitUtil.xml, can be used to call JUnit for a set of test cases or suites. It creates a temporary ANT project that contains a classpath definition junit.tmp.classpath, which contains all the JAR files that are specified in parameters lib_dirs and libs. After that, it starts the target junit.run defined in the sample file JUnitRunner.xml, referencing the classpath. (To get the JUnitRunner.xml and JUnitUtil.xml sample files, see Download.) The target starts the task junit as in Listing 9:

Listing 9. Starting the task JUnit

<project name="cvt.junit.runner" default="junit.run" basedir="."> <import file="${junit.tmp.classpath.import}" /> <target name="junit.run" description="Runs the unit test cases."> <echo message="Generating JUnit output at ${output.dir}"/> <junit dir="${work.dir}" fork="yes" forkmode="once" printsummary="on" haltonfailure="no" haltonerror="no" showoutput="true" failureproperty="ut.failed"> <jvmarg value="-Djava.util.logging.config.file=${java.util.logging.config.file}"/> <sysproperty key="net.sourceforge.cobertura.datafile" file="{net.sourceforge.cobertura.datafile}" /> <classpath refid="junit.tmp.classpath" /> <test name="${cvt.test.class}" haltonfailure="no" outfile="TEST-${cvt.test.class}" todir="${output.dir}"> <formatter type="xml"/> </test> </junit> <fail if="ut.failed" message="Some of the tests failed. Check the JUnit report for details." /> </target> </project>

The property net.sourceforge.cobertura.datafile can be used to generate coverage information (see Cobertura project for more information). You can provide this property value in parameter jvm_args from function run_junit_ant.

Collect information (results, coverage information, logs)

Again, the transference of compressed files is more efficient than transferring single files one by one. To compress a directory in a remote machine, use the ZIP request. In Listing 10, the directory 'C:/CVT_TEMP/CVT/logs' into ZIP file 'C:/CVT_TEMP/testcase_123_results.zip' is compressed in machine mach_name.

Listing 10. Using the ZIP request to compress a directory in a remote machine

<script> zip_file = 'C:/CVT_TEMP/testcase_123_results.zip' results_dir = 'C:/CVT_TEMP/CVT/logs' </script> <stafcmd name="'Compressing results'"> <location>mach_name</location> <service>'ZIP'</service> <request>'ADD ZIPFILE %s DIRECTORY %s RECURSE RELATIVETO %s' % \ (zip_file, results_dir, results_dir)</request> </stafcmd>

After compression, you can retrieve the results with the request COPY FILE in Listing 11:

Listing 11. Using the request COPY FILE

<script> local_results_dir = 'C:/CVT_TEMP/results' </script> <stafcmd> <location>mach_name</location> <service>'FS'</service> <request>'COPY FILE %s TODIRECTORY %s' % (zip_file, local_results_dir)</request> </stafcmd>

Firewall problems might arise when you run tests on machines that are spread across different networks. A safer approach is to use the GET BINARY request to retrieve the whole file in binary form as a result from the call. The advantage is that the remote machine is not required to resolve and access the local machine. The disadvantage is that you might not be able to retrieve large files without breaking them into smaller pieces before the transference.

Free up resources

The request to delete directories is straightforward. Specify the directory to be removed and the machine where the directory is located, as in Listing 12:

Listing 12. Requesting directory deletion

<stafcmd name="'Delete remote directory'"> <location>mach_name</location> <service>'FS'</service> <request>'DELETE ENTRY %s RECURSE CONFIRM' % \ (file, ignore_param)</request> </stafcmd>

To delete databases, stop servers, or take other actions, use the process element as explained in Listing 12.

Build reports with results (JUnit reports)

For generating a JUnit report, employ the same approach as for running JUnit test cases: Create an auxiliary ANT project to start the tasks that create the JUnit report, as in Listing 13:

Listing 13. Creating an auxiliary ANT project

<project name="cvt.junit.report" basedir="."> <target name="junit.report"> <echo message="Generating JUnit Report at ${junit.report.todir}"/> <echo message="Reading results from ${junit.report.dir}"/> <mkdir dir="${junit.report.todir}" /> <junitreport todir="${junit.report.todir}"> <fileset dir="${junit.report.dir}"> <include name="TEST-*.xml" /> </fileset> <report todir="${junit.report.todir}" /> </junitreport> <echo message="JUnit Report Generated"/> </target> </project>

To start the script, call it through the process element:

Listing 14. Starting a script through the process element

<script> from java.io import File f = File(STAXJobXMLFile) ant_script = File(f.getParentFile(),'utilities/JUnitReport.xml').getAbsolutePath() ant_script = ant_script.replace('\\','/') java_envs = '' java_envs = '%s -Djunit.report.todir=%s' % (java_envs, output_dir) java_envs = '%s -Djunit.report.dir=%s' % (java_envs, work_dir) parms = '%s -f %s junit.report' % (java_envs, ant_script) </script> <process name="'cmd %s %s' % (ant_cmd, parms)"> <location>mach_name</location> <command mode="'shell'">ant_cmd</command> <parms>parms</parms> <workdir>work_dir</workdir> <console use="console" /> <stdout>stdout</stdout> <stderr>stderr</stderr> <returnstdout /> <returnstderr /> </process>

Store the results locally or in a remote server (using FTP or a shared folder)

After you run all reports useful for verifying the tests results, compress them in one or more files. Access to the results might be easier if you store them on a web server so developers and testers have immediate access to them. See an example in Listing 15.

Listing 15. Compressing reports

<stafcmd> <location>'local'</location> <service>'FTP'</service> <request>'PUT HOST %s URLPATH %s FILE %s USER %s PASSWORD %s' % \ (CVT_FTP_SERVER, remotePath, localPath, ftp_user, ftp_passwd) </request> </stafcmd>

Send notifications (emails)

You can use the EMAIL service to send notifications about incremental or regression tests to developers or testers. The configuration of the service in staf.cfg (STAF configuration file at bin directory) is simple. It requires just an email server and the path to the JAR file that contains the service implementation. You can download the files from the STAF download page, as follows:

SERVICE email LIBRARY JSTAF EXECUTE c:/staf/services/email/STAFEmail.jar \ PARMS "MAILSERVER na.relay.ibm.com"

Listing 16 shows the structure of the send email request:

Listing 16. Structure of the send email request

<stafcmd name="'send_email'"> <location>'local'</location> <service>'EMAIL'</service> <request>email_cmd</request> </stafcmd>

where email_cmd is in the format (from the Email Service User's Guide) as in Listing 17:

Listing 17. Format for email_cmd

SEND < TO <Address> | CC <Address> | BCC <Address> >... [FROM <user@company.com>] [CONTENTTYPE <contenttype>] < MESSAGE <Message> | FILE <File> [MACHINE Machine] > [SUBJECT <subject>] [NOHEADER] [TEXTATTACHMENT <file>]... [BINARYATTACHMENT <file>]... [ATTACHMENTMACHINE <machine>] [RESOLVEMESSAGE | NORESOLVEMESSAGE] [AUTHUSER <User> AUTHPASSWORD <Password>]

Using continuous integration with STAF/STAX

From a developer perspective, continuous integration consists of frequently integrating new code and changes into the code repository, and continuously verifying that the changes do not break existing code.

Verification is normally performed by an automated build system, which can detect when changes occur or run builds at scheduled times. The build verifies possible code breaks, and preferably runs a set of unit tests to verify that basic functions were maintained. Builds can happen often or take a long time to run. You normally do not want to include component tests in a continuous integration system, since it is too much to run every build.

One possible solution for running component tests continuously is a separate system. The system runs automated tests against the current code base or runs specific tests upon request. Consider running a single loop, through STAX, that:

Retrieves the latest production and test image Verifies whether a prescheduled regression is scheduled Detects changes in the test code (STAX script for each test case) Verifies whether a request from an incremental test is scheduled

Perform regression tests once a day, comprising all tests. Incremental tests can be run whenever there is a code change, like a fix or new feature, that must be tested. For example, to test a new parameter to an existing command-line interface, run an incremental test that is specific for this command-line interface. Running incremental tests increases team productivity as they provide for identification of failures sooner that waiting for a nightly regression.

To verify whether an image (bundle containing software to be tested, or test resources) was updated, retrieve the file modification time and compare it with a previous modification time stored in a local file. The Python command to retrieve the modification time of a file in an FTP server is:

ftp.cwd(ftp_url_path) ftpMDTM = ftp.sendcmd('MDTM %s' % file) #Example of result: '213 20101129060511'

If the file was modified since the last verification, you can download it using either STAX FTP service, or the Python FTP module. The verification of whether it is time for running a regression can be as simple as comparing the current time to the schedule. If the current time is older than the time of schedule, run the regression.

A more complex function is the ability to run incremental tests. An incremental test is a request for running a list or a predefined group of test cases in a chosen set of environments. One possible implementation is to periodically verify whether new requests are in a file that is shared with the web server. You can use an HTML page made available by a local web server to include requests in this file, containing information such as:

The request (a predefined group or a list of individual test cases)

The product version

The list of environments where to run the test cases

sessionID

Identification of the request

Other internal information that is needed to handle the request

In the following example, the code displays on multiple lines to meet format restrictions. Normally it is a single line of concatenated code.

t||testcase_nop||FRS-1.0.0||LinuxDB2Websphere85,ZLinuxDB2Websphere8||false||true| |4EF60DC3B825707D6BAE349047B4E664||fnegre@br.ibm.com||ready||20121023220123||default

Figure 1 shows an example of page that is used to request incremental tests.

Figure 1. Request page

Using a web server like Apache Tomcat as an interface with the STAF/STAX script running the continuous integration allows expansion of the number of options for the final user, for example:

Add the ability to specify customized images (images that are generated locally by a developer/tester/support analyst)

Add the ability to download images directly from an FTP server

Cancel a request under execution

Visualize STAX user logs for the current execution

Visualize logs and history, and other reports

Figure 2 shows an implementation of the main page with links to the main functions in the system. A web server is also useful to show the status of the integration system, such as:

The test case under execution

The start time

The requester

The environments under test

Links to the results (when the test is already finished)

Figure 2. Main index of the continuous integration system

Figure 3 shows the page where the user can access the results from previous executions, which are organized by environment name. The results contain information like test case logs, JUnit reports, coverage reports, and any other useful information that is generated by the tests. Notice the six environments in the report. All test cases run nightly in these environments, and are available during the day for incremental tests. Among the available environments are a Linux for System z, Windows, and Linux (Red Hat) machines. New environments can be plugged into the system by creating a properties file for each new environment.

Figure 3. Access to regression logs

A continuous integration system that contains the basic functions can gradually build following this sequence:

STAX script to run individual test cases in a machine STAX script to run multiple test cases in multiple environments after updating/preparing them STAX script to run the script above nightly Ability to run incremental test through an HTTP page and status page to visualize current execution

Conclusion

One significant advantage of using STAF/STAX to build both the test cases and the infrastructure for continuous integration is the flexibility to grow the test automation system as needed, by creating new functions and scripts that you can integrate into it. When a project adopts agile development with the capability to automate all tests with STAF/STAX and integrate them in a continuous integration system, you can test new functions or fixes in a timely fashion. Also, the execution, upon user request, of incremental tests across a set of different environments in parallel can shorten development time and boost software quality.

Downloadable resources

Related topics