Create an ooRexx build environment on Linux KVM

Kernel Virtual Machine improves build performance

Recently, the Open Object Rexx project (ooRexx; see Related topics later in this article for more info) converted its old on-demand software build system from VMware-hosted guest operating systems to Linux Kernel Virtual Machine (KVM)-hosted guests. This change provided a more efficient build environment and reduced the build times for users.

The ooRexx software build system allows developers to build ooRexx software packages for multiple x86-based platforms and operating systems. Currently, the supported guest operating systems include Windows® XP (i386), Fedora 10 (i386 and x86_64), and Ubuntu 8.04 (i386). These guest operating systems can produce ooRexx installation and documentation packages for Windows (EXE), Fedora and openSUSE (RPM), and Ubuntu (DEB). Other x86-based operating systems will come online as demanded by the developers and users of ooRexx.

This article shows how to create your own software build system, taking the ooRexx development team's setup as an example and including tips and direction as needed for developers experienced with ooRexx, Apache, and Linux. You can download the server and guest scripts at the end of this article. As designed, the system is specifically for building the ooRexx software, but the concepts can be applied to a general purpose build system for any software.

The broad requirements for this system include the following:

You need a Web interface for generating the build request.

You need a Web interface for retrieving the build results.

You need support for multiple guest operating systems.

Guest operating systems must perform fully automated builds.

An e-mail should be generated and sent to the requesting user at the end of the build.

To accommodate these requirements, the development team and I used a quad-core Xeon-based server. The server contains 4GB of memory and 250GB of disk space. We chose the Fedora 10 x86_64 distribution as the main operating system mainly due to the stability and up-to-date release of KVM used by the distribution. Your choice of hardware and software may be different, but the main hardware criterion is that your processor should have hardware virtualization available—it's a must in order to use KVM.

Set up the server

The first step in setting up the build server is to determine the partitioning scheme. We decided to segregate the Web store and the images for the guest operating systems into separate partitions. We set aside 50GB for the Web store and about 150GB for the /var partition where the guest images are placed. The rest was divided between the /home partition and the /root partition.

General requirements of a build system Some of the more basic requirements of a build system include: Frequent builds to catch problems as early as possible

Faster builds (the faster they are, the more you can do)

Incremental build processing (or build avoidance) to reflect small development updates

Support for managing (at least the low-level) source code dependencies to keep your system as flexible as possible

Extraction/reporting capabilities on build, compile, and link usage

A reporting system that can trace source to binary matching (efficiently comparing the old to the new)

The ability to report on build status and things like testing pass or fail

The ability to create release notes and system documentation

Next, we installed the main operating system using the Fedora 10 x86_64 release distribution. If you're setting up your own system, you'll save yourself a lot of trouble if you do the following:

Enable hardware virtualization via the machine's BIOS prior to starting the installation so that Fedora will see that KVM is available.

Perform a custom install of the software components so that you can select the Fedora virtualization options.

After we installed the server operating system, we configured it for access by the guest operating systems. This involved enabling Samba for the Windows guests and NFS for the Linux guests. This gives the guests access to the build results partition so they can store their build files for access by the user. Both the main Samba share and the main NFS export point to the same location for all guests.

Next, we configured the Apache Web server to provide access to the build request system, which I discuss below under The build requests, and the build results repository.

One configuration decision you will need to make concerns the networking option for the guests. The default installation is configured to supply a private internal network for all guests. A class C network is provided along with a DHCP server to provide IP addresses to the guests. The other option is to set up the system so that one of the network devices acts as a bridge to the server external network. This will need to be configured by hand. You can see an example of how to configure your server for this option at the libvirt Wiki (see Related topics for a link).

Create the guests

There are two approaches to creating guests for use by KVM. In the first approach, you create just the guests you need to meet your requirements. The second way takes a more long-range approach to creating guests. This is the method we used to create guests, and we recommend it as a standard approach if the necessary resources are available.

We first determined the number and types of guests from the requirements. We needed operating systems to create software built for those environments and another operating system for creating the documentation. It turned out that in our case, the documentation and i386 RPM tasks could be combined and handled by a single guest. Here are the guests and tasks assigned to them:

Windows XP (i386): Build the Windows install executable.

Fedora 10 (i386): Build the i386 RPM file and the documentation ZIP file.

Fedora 10 (x86_64): Build the x86_64 RPM file.

Ubuntu 8.04 (i386): Build the DEB file.

The approach we used created the previously mentioned guests as images that could be cloned at a later time. So, each guest had a basic version that we kept for later cloning and a customized clone of that guest that performed the actual build task.

Cloning a KVM guest is really easy. The virt-clone script provided by Fedora 10 can completely automate the task.

Listing 1. Fedora 10's virt-clone script

$ virt-clone --original=Fedora10-i386-Base \ --name=Fedora10-i386-Build \ --file=/var/lib/libvirt/images/Fedora10-i386-Build.img

The original option specifies the name of the guest operating system as it is known to the virtual machine manager. The name option specifies the name of the new guest. The file option specifies the file name of the new image file for the guest. This will completely clone an existing guest and copy it to a new version of the guest. It will also modify the MAC address of the new guest as well as the UUID. Thus, the original guest is preserved for further cloning, if necessary, and a new guest is provided for your customization.

One thing about the virt-clone script should be noted: If the original guest has an unconnected device, such as a CD-ROM, the script will fail. All unconnected devices must be removed prior to cloning the guest. If necessary, you can add them back after the cloning operation.

Once the basic build guests were created, each was then customized for the build task. The Windows XP guest was the most challenging in this area, because the Visual C++ compiler and Software Development Kit needed to be installed as part of the customization. The Linux guests required practically no customization except for installing NFS on the Ubuntu guest.

The build requests

All the build guests require one additional customization: Open Object Rexx version 3.2 must be installed on each of them. Why? Because a script written in ooRexx controls the request and build process on each guest. The script accesses a TCP/IP port on the build machine server to search for build requests. If a request is found, the build process is invoked and a set of installation files (RPM, DEB, EXE), along with a build report, is created and stored on the build machine server for access via HTTP by the user.

The ooRexx build request script is smart enough to not duplicate an already built version of the software. It also notifies the user via e-mail that the build is finished and provides a URL to the location on the build machine server where the build was stored.

Build requests are generated via a CGI script on the build machine server. The user visits an HTTP Web page and requests a build on their choice of operating system. The Apache server then invokes the CGI script, which places the request in a queue. The build guests are then responsible for accessing those requests and performing the requested operation. The results of the build are also available via the Web site for download.

As mentioned above, the code for both the server and the build guest clients is available for download below. All the scripts are written in ooRexx and should be easy to follow. The following files are included:

buildserver.rex : This script runs on the Web server and maintains the queues that the guest operating systems use to look for work.

: This script runs on the Web server and maintains the queues that the guest operating systems use to look for work. buildd.rex : This script runs on each guest operating system and accesses the queues on the Web server looking for work. Each guest has only one queue for which it is responsible and thus will build only one kind of output file.

: This script runs on each guest operating system and accesses the queues on the Web server looking for work. Each guest has only one queue for which it is responsible and thus will build only one kind of output file. simq.cls : This file is not a script but is a class file used by the buildserver.rex script.

: This file is not a script but is a class file used by the buildserver.rex script. socket.cls : This class file is used by the buildd.rex script.

: This class file is used by the buildd.rex script. streamsocket.cls: This class file is also used by the buildd.rex script.

The build results

The results of a build are placed on the Web site by the build guests. The builds are organized such that each Subversion revision has its own subdirectory to contain all the platform guest builds for that revision. Also, each platform guest produces a build report that is also stored in the revision subdirectory.

The build report is an absolute necessity for all this to really work. If an error occurs during the build, and the final output files are not produced, the user must have a way to discover what went wrong. So, the build process on each guest always produces a build report and stores it in the build repository.

The build repository location segregates the ooRexx software builds from the documentations builds. The documentation build produces a large number of output files, so it was deemed convenient to separate those builds from the software builds.

Currently, there is no automated build cleanup process. The project does not produce enough output to make that necessary at this time.

Conclusion

By switching from VMware build guests to Linux KVM guests, build times for each guest are reduced by as much as 50 percent. This means the development cycle for ooRexx has also been reduced.

The ooRexx development team is by far the largest user group of the build machine, but during the alpha and beta cycles for major releases, a large number of ooRexx users also make use of the environment. This allows users to provide timely feedback for any bug fixes or enhancements made to ooRexx.

One of the most-used build guests is the documentation builder. All the ooRexx documentation is coded in DocBook. This makes the documentation hard to build on a Windows system without some experience with DocBook in that environment. This is a real problem for most of our developers and users. By providing a documentation build process that everyone can access, we ensure that developers and users can contribute to and maintain the documentation. This has produced documentation for ooRexx that is much better than most open source projects, a source of pride for our development team.

Downloadable resources

Related topics