Gwydion: An Integrated Software Environment for Evolutionary Software Development and Maintenance

Scott E. Fahlman School of Computer Science Carnegie Mellon University Pittsburgh, PA 15217 Internet: sef+@cs.cmu.edu Phone: 412 268-2575 Fax: 412 681-5739 March, 26, 1994

Overview

Gwydion will be built around Dylan [2], a new object-oriented DYnamic LANguage being designed by Apple Computer, with some input from our group and from other collaborators. Dylan incorporates many ideas from AI-oriented languages such as Smalltalk, Scheme, and Common Lisp, but with a new emphasis on practical, real-world programming. Dylan is being designed to appeal to mainstream programmers who are now working in static languages such as C and C++. (See section [*].)

As part of this project, we will develop a Dylan compiler that will produce efficient executable code whose size and speed are competitive with C and Fortran.

The following key ideas will guide our work on Gwydion:

A single, unified development environment should be used throughout the life-cycle of a software system, from initial prototyping though testing, refinement, deployment of a high-quality system, and post-deployment modification. (See section [*].) Fast initial prototyping and easy modification at later stages can best be achieved in an object-oriented dynamic language such as Dylan. (Section [*].) In order to support software projects all the way to deployment and post-deployment maintenance, the Gwydion environment must include a Dylan compiler that produces executable code whose speed and size are competitive with C and Fortran. (Section [*].) While Gwydion will primarily support the development of new code in the Dylan language, it must also provide excellent support for combining Dylan code with modules written in existing languages, particularly C, C++, and Ada. A dynamic, object-oriented language can serve as very flexible glue for creating heterogeneous software systems. (Section [*].) Programmers will be more productive and programs will be more reliable if the programming environment provides a very large library of pre-existing software modules, plus some simple, flexible glue for linking these pieces together. Gwydion will support this model, and will provide excellent "librarian" software to maximize software re-use. (Section [*].) A program should no longer be thought of as a linear string of ASCII characters with (maybe) a few supporting documents. The Gwydion environment will deal in hypercode, analogous to hypertext. A program in Gwydion will be a complex linked data structure stored in an object-oriented database. This data structure will contain source code linked with documentation and all the other information needed to support the software system throughout its entire life-cycle. (Section [*].) For hypercoding to be effective, Gwydion must provide multiple ways of viewing and editing hypercode, customized to match the user's preferences and immediate needs. (Section [*].) To aid in fast prototyping and ongoing maintenance, Gwydion will provide advisors, demon processes that continually monitor the consistency of the software being developed, keeping track of missing modules, interface violations, and so on. (Section [*].) Program execution, profiling, and debugging will be monitored via animated views relating the program's activity to specific locations in the hypercode. (Section [*].)

Evolutionary Software Development A single language integrated with an environment that produces efficient and maintainable code has great advantages for evolving software that can be used with code developed in other languages.

The Need For An Evolutionary Approach

For some tasks, especially for very large programs and those that must meet strict safety requirements, this strict division of labor may be appropriate and even necessary. For such tasks, conventional software-engineering methodologies and languages like Ada and Pascal are the best tools available.

However, there is a growing realization that most software systems cannot be fully specified in advance. Usually, the ``customer'' begins with only a vague idea of what he wants. A working prototype is developed, and experience with this prototype enables the user to understand more clearly what it is that he really wants. Changes are made, the revised prototype is tested, and so on. This cycle of testing and revision often continues throughout the lifetime of the software. For most successfully deployed programs, more money is spent on post-deployment modification than was spent during the program's initial development. Even if the specification writers were perfect, deployed programs would still have to be modified as the mission, the environment, and the computing systems change over time.

If we accept the idea that most software development is evolutionary, whether or not it is intended to be, then we need software tools designed to support the evolutionary process, not to suppress it. Instead of emphasizing program ``correctness,'' in the sense of matching a fixed specification, we need tools that make it fast and easy to get a prototype program running, to test and debug it, and to modify the program's behavior after it is running.

A Single Language for the Entire Software Life-Cycle As people have gained an appreciation for the necessity of evolutionary programming, some have proposed the development of separate ``fast prototyping'' languages, perhaps specialized for use in a particular application domain. We think that this idea is a bit off the mark.

The prototype-language paradigm suggests that each program should be written two or three times, using a different language each time. First, a specification of some kind is written, either in very precise English or in some formal specification language. Second, an executable prototype is created using some special language designed for fast prototyping. This version is tested and changed until the program is close enough to what the customer really wants. However, the revised prototype often cannot be deployed ``as is'' because it is slow and perhaps not very reliable. So the prototype is translated into a more conventional language for deployment, and this time more attention is paid to efficiency issues. Program documentation may be considered a fourth version of the program's description, since the original specification may not be readable by the average user or maintainer, and may no longer be up to date.

Most programmers resist this methodology. They prefer to write the program only once, in their favorite programming language and using their favorite tools. They believe that they can get the program right in that environment, and that the translation from one version to another is likely to introduce more problems than it solves. In many cases, they will be right. If multiple human-maintained definitions of a complex system exist, they are almost certain to be inconsistent with one another.

The idea of using separate languages for prototyping and delivery has another major problem: the transition is designed to be one-way. Once you move from the prototyping language to the deployment language, you have to stay there. If you want to make substantial changes to the production version of the program, you lose all the advantages of working the prototyping environment.

Our view is that a single language and environment should be used throughout the program's life cycle, from the initial sketch of the program, through a series of executable prototypes, and on to final deployment and maintenance phases. The transition from a flexible, easily modified prototype program to an efficient version suitable for deployment would not require translating the program by hand, but instead would be a mechanical transformation--a sort of compilation. Changes would always be made and tested in the prototyping environment.

Initially, the program would have a simple structure dictated by the desired behavior. Even this simple early sketch should be executable, using default choices for implementation decisions the user has not yet specified. For testing purposes, any modules that are not yet fully specified would be represented by "dummy" versions, perhaps with a value being supplied by the human operator. As program development progresses, the programmer will begin to make more detailed choices relating to implementation and efficiency questions rather than functionality.

We believe that Dylan will be an excellent choice for this central evolutionary language. The ``AI languages'' such as Lisp, Smalltalk, and Scheme have always emphasized fast-prototyping and evolutionary development, but without paying much attention to the concerns of mainstream programmers working on real projects. Dylan incorporates many of the best ideas from these ``AI languages,'' but with a new emphasis on the development of practical applications.

High-Quality Compilation for Delivery The success of this ``one language, one environment'' approach depends critically on the ability of the system to produce Dylan programs that are as efficient and robust as programs developed in static languages. If the quality demanded for delivery is not easily attainable, then there will be continual pressure to translate the code (irreversibly) into a language with adequate efficiency and static error detection.

There is a difficult problem here: a dynamic language like Dylan allows the user to change the definition of a function or a class at any time, perhaps even while the program is running. This is one of the reasons that such dynamic languages are attractive for evolutionary programming. But the ability to replace a module at any time means that the seams between redefinable parts of the program cannot be welded as tightly as they are in conventional programs. Furthermore, as the user makes changes, he may want to call on new routines from the library of available functions, so this library must be kept around at all times. The result is a system that is excellent for fast prototyping and easy modification, but one that is rather slow and that requires a huge runtime environment.

The solution, we believe, is to cleanly separate development mode from delivery mode. While developing and modifying a Dylan program, the programmer will use the Gwydion environment in development mode, with full access to the extensive libraries of existing code. Code can be compiled and run in this environment, but the functions and classes will be bound together rather loosely, so that changes can be made easily. Code speed can be reasonably good in the development environment, but it will never be quite as good as in conventional languages (given comparable compiler technology).

However, at any point, the user can ask Gwydion to compile the program in delivery mode. The delivery version will never be changed (though it may be replaced with a later version), so the program can be compiled as a single block, taking full advantage of opportunities for global optimization. No new functionality will be added to the delivery module, so there is no need to carry along the full library. The delivery module will include only those parts of the code library that the module actually uses. Some runtime error checking code will normally be included (the user can specify how much safety he wants), so that any problems can be detected and diagnosed, but fixes and changes are never made in the delivery module. For any changes or repairs the user goes back to the Gwydion development environment, which provides full support for modification, testing, and logging of changes.

Since the delivery module is not designed for easy modification, there is no excuse for it to be slower or larger than an executable produced by C or Fortran. In fact, since the entire program can be analyzed as a whole, comprehensive inter-module optimization and static error checking can be done, resulting in better efficiency and correctness guarantees than are provided by conventional file-at-a-time compilers.

Compiler Technology. At this point, code quality just comes down to the quality of the optimizing compiler. Our experience in developing the Python compiler for CMU Common Lisp gives us confidence that we can develop a Dylan compiler that will offer good performance in development mode and excellent performance in delivery mode. Python performs extensive flow analysis and type-inference on the program, which allows for a high degree of optimization. Floating-point code is particularly fast under Python, as is ``safe'' code with a high degree of runtime error checking. If the programmer has left out information (e.g. type declarations) that the compiler needs in order to produce optimal code, the compiler will (optionally) engage the user in a sort of dialog, explaining what additional information is needed and the performance consequences of various implementation decisions. Python's code-generation machinery is largely table-driven, which makes it easy to retarget the compiler for new machines.

Automatic Storage Allocation.

Gwydion's Dylan implementation will employ the latest and best GC technology, probably using the facilities of the GC toolkit developed by Eliot Moss at the University of Massachusetts. This advanced generation-scavenging GC can be very efficient and almost invisible to the user.

However, even this advanced GC technology may not be the right answer for embedded real-time applications. Gwydion will provide a special static storage allocation mode for developing and testing such applications. The GC machinery in the Gwydion development environment will be used to verify the operation of code that, in delivery mode, will run without a garbage collector. The GC can be used to catch exactly the kinds of subtle storage-allocation bugs described above before the system is deployed. This technique has been used successfully in some of the more advanced commercial C++ environments such as Object Center.

The Glue and Library Model The best existing tools for fast prototyping and evolutionary development employ what we call the ``glue and library'' model. This means that instead of programming everything at the lowest level, the user selects existing pieces of software from a very extensive library, and combines these pieces using some very simple, flexible syntax -- a sort of ``universal glue.'' We see something like this even in the C/Unix world, in which small programs with a standard interface are strung together into a larger system by means of Unix pipes. This is a rather awkward system, since the pipes carry only bytes of ASCII text and the grain size of the modules is rather large. Nevertheless, this primitive system is responsible for much of Unix's celebrated power and flexibility.

The Lisp world provides a better sort of universal glue in the Lisp function call. This can handle data objects (arguments) of any type, and can be compiled into an efficient form. A key point is that the same ``universal glue'' syntax is used at every level of the program, whether you are tying an expert system to a display package or just accessing an array. Object-oriented programming, as implemented in CLOS and Dylan, is just a more flexible kind of glue. The program is built up from classes of objects (organized in a hierarchy), along with the specialized operations that affect each class.

Robust software comes, in part, from building large systems in small, manageable steps. An individual Lisp function or Dylan method is usually much shorter than a screenful of code. Small functions are easy to build and easy to get right. Each function is relatively self-contained; it has a stated purpose (text attached to the definition) and an interface that adheres to the standard conventions. Each function can be tested before it is combined with others into larger structures.

Larger pieces of code can similarly be assembled without viewing the internal structure of library elements. A visual programming model can be used by the librarian to paste together entries into more powerful components, with user knowledge limited to the functionality and advertised interface of each module.

The point of all this is that both fast prototyping and later modification of programs can be much easier, and less prone to errors, if the code is organized into small independent functions connected in a uniform way. It becomes very easy to pop the parts of a program apart and to rearrange them in some new way. Combine this idea with a large library of pre-existing functions, and you have a very powerful evolutionary programming environment. Add a compiler smart enough to compile this call-intensive code efficiently, and you will have a good deployment language as well.

Dylan's universal glue (object-oriented function call) is very similar to that of Common Lisp with CLOS, but a clear distinction is made in the Dylan language between the core language and the libraries. Gwydion will be designed to support arbitrarily large libraries of hypercode, with librarian software to help the user find the code module that best fits his current needs. Searches might be performed using an index of keywords, some of which may be automatically extracted from the text associated with each module. It might be possible to look for all programs that have certain structural features or that are used in certain ways. The library might be kept on the local machine, or it might exist on a set of server machines somewhere on the Internet. Some research will be required to refine the interface for this library facility.

The idea of application-specific programming languages is very popular in some quarters, but there are problems with this approach: first, with separate languages it is hard to mix and match code written for different applications; second, there is much duplication of effort in creating development tools for each of these many languages. Dylan may be viewed as a single, general-purpose language which can be made application specific though the addition of libraries. Or, if you prefer, it may be viewed as a common toolkit in which any number of applications-specific languages can easily be created. In any case, the programs written in Dylan will all share a common set of development tools and a common syntax and structure that allows easy mixing and matching.

Other languages might also serve this "common toolkit" function, but Dylan has several advantages: First, the object-oriented model used in Dylan makes it easy to combine libraries of code and object-classes that were developed separately. Second, Dylan provides a very rich and extensible set of data structures and styles of programming that can support the specialized needs of any application domain. Third, Dylan's powerful macro system makes it possible to alter the surface syntax of the language to meet any specialized needs, and even to add new compiler optimizations.

Heterogeneous Programming It is our goal to produce an implementation of Dylan that is so easy to use and so efficient that it will be the natural choice for most tasks. However, we recognize that a very large amount of useful code will still be produced in languages such as C++ and Ada. The environment must therefore provide excellent facilities for integrating pieces of foreign code with code written in Dylan. In a sense, this is a natural extension of the ``glue and library'' philosophy, but with the library modules written in many different languages.

We will support function calls in both directions and the sharing of structured data objects from (at least) C, C++, and Ada. One important technique is to automate the building of interface code for foreign objects by reading the foreign-language source files--the C record declarations and routine headers, for example.

The Dylan Language

Object oriented programming (OOP), in one form or other, has been used in AI languages like Smalltalk and Lisp for almost 20 years--long before the current object-oriented fad took off in mainstream languages. OOP is an important tool for evolutionary programming. It provides a more flexible and powerful form of ``universal glue'' than the standard function call. OOP is especially valuable in task involving simulation and graphics. The real world, and (and its image on a computer display) is naturally described in terms of objects, classes, attributes, relationships, and class-specific operations. The object-oriented paradigm preserves this natural structure.

But while object-oriented programming is an important tool for evolutionary programming, it is not the whole story. It is equally important to provide a dynamic software development environment. A language like C++, in which some object-oriented facilities are grafted onto an old-fashioned static language, does not deliver the full benefit of object-oriented programming.

By ``dynamic'' we mean four things:

Dynamic Linking and Loading. During program development, it must be possible to change the definition of a specific function or method without stopping to recompile and re-link the entire system. Dynamic Type System. During development, it must be possible to create new classes or change the definition of old ones, again without stopping to recompile and rebuild the universe. It must be possible to create variables and operations that can accept objects of any type, or of a specified range of types. Dynamic Memory Management. For fast prototyping, it is essential to have automatic storage allocation for new objects and some sort of garbage collection of old, unused objects. This frees the programmer from having to worry about these issues, and eliminates a common source of very subtle bugs. Dynamic Debugging and Program-Development Tools. For maximum effectiveness, all parts of the programming environment must be designed with a dynamic program-development style in mind. This includes program editors, interactive debugging tools, performance measurement tools, etc.

In our initial planning for the Gwydion project, we examined existing programming languages and came to the conclusion that a new language was needed if we were to bring the full benefit of evolutionary programming to a wider range of programmers and real-world software projects. Of existing languages, Common Lisp with CLOS is perhaps the best example of an object-oriented dynamic language. CLOS provides more advanced object-oriented facilities than are found in Smalltalk or C++, and Common Lisp was fully dynamic from the start. However, Common Lisp has some serious problems as well:

The CLOS system was grafted into Common Lisp late in the game, so the object system is not well-integrated with some other parts of the language.

Common Lisp is considered by many programmers to be too large for everyday use, both in the size of delivered applications and in the overwhelming amount of material a programmer must learn. It is one thing to have an extensive library of available functions, but quite another to present it all as a single monolithic language.

The Common Lisp syntax, with its nested parentheses, is considered by many programmers to be unreadable.

Common Lisp provides poor support for modularity and ``programming in the large,'' and poor separation between the development and runtime environments.

While some of these problems can be fixed, others are now locked in due to the ANSI standardization of Common Lisp. In any case, many potential users are now convinced that these problems are inherent in Common Lisp.

We were about to propose the development of a new object-oriented dynamic language that would eliminate these problems and thus be attractive to a wider range of programmers. However, we were worried that yet another language, especially coming from a university group without solid industrial backing, would not be well received. Then we learned about the Dylan project at Apple's Cambridge Research Laboratory, then under the direction of Ike Nassi.

Upon investigation, we found that Apple's goals for Dylan (the language itself, not the programming environment) were nearly identical to our goals for a new evolutionary-programming language. Dylan has the solid backing of Apple, and is also attracting interest from other companies and research groups. A group at Digital Equipment Corporation has implemented and released Thomas, a ``quick and dirty'' prototype implementation of Dylan on top of Scheme. Harlequin (a Common Lisp vendor based in England) is developing a Dylan implementation with Esprit funding. Apple has sent out several thousand hardcopies of the Dylan manual to individuals who have requested them, and there is lively discussion of the language design on the Internet. Several stories about Dylan have appeared in the industry press.

While Dylan is a trademark of Apple, they have stated that this control will be used only for quality-control purposes. They intend to license the use of the Dylan name to any implementation that passes their test suite. Apple plans to complete the Dylan language specification in the spring of 1994.

The Gwydion Environment

The Gwydion environment is designed to support the flexible interleaving of the design, implementation, testing, and tuning phases of software development, before and after deployment. As part of its programmer support it captures knowledge about the requirements and design rationale. The Gwydion environment can also provide reverse-engineering tools for advising the programmer the best way to make some modifications with minimal ripple effects on the rest of the system.

Hypercode

Levels of Detail.

Hypercode is not just text; it also will contain graphical objects that illustrate various concepts, data structures, and abstractions, linked to one another and to code strings. The code viewer allows consistent manipulation of code strings, templates, and modules in one framework, all the while preserving the programmer's ability to view and edit the code at different levels of abstraction.

The relationship between objects, for example the call dependencies between functions, are designed to be part of the graphical interface used by Gwydion, and these relationships can be traversed and selectively viewed. The user could jump directly from a function to a list of its callers, and from there to some specific caller. People who have maintained large object-oriented systems are aware of the problem of ``object spaghetti'': If you look at the DRAW method for the BATTLESHIP class, you may find a routine that draws a gun turret and then defers to the more general DRAW method for WARSHIP. It can take a great deal of searching through the type hierarchy to find all the modifiers and the routine that actually does the bulk of the work (if there is such a central routine). Upon command, Gwydion will collect all of the code relevant to a certain operation or class for inspection by the user. When the user makes a change, Gwydion will keep track of what other parts of the program are affected by that change.

Integrated Documentation and Testing.

The hypercode view that a programmer finds most useful will be easily adjustable and will track the user's actions even across sessions. This means that the tools and modes that the system developer finds most useful will be readily available whenever they are needed. There should be no need to maintain a ``profile'' or ``initialization file'' by hand.

Past views, a software history.

Programming in the Large.

Given the facilities for logging changes and re-building consistent versions of the system, Gwydion should be an excellent environment for supporting cooperative work by a large group of people. Instead of locking files or directories while changes are being made, it should be possible to lock semantically meaningful parts of the applications, such a ``lock this function and every function that calls it.''

Advisors and Animated Views

The environment includes a set of advisor demons (co-routines or processes) that run in the background as the user works on a software system. These demons continually monitor the user's actions and the state of the system. They can provide advice at any time, but the user is not required to follow it, so the user is free to develop the hypercode in his own way and in any order. Rather than following the dictates of a rigid structure editor, the programmer may choose to build code that is temporarily inconsistent or incomplete. When the time comes for final cleanup, the advisors will help him to locate all the loose ends.

At the lowest level, an advisor will observe the user's code, noting any syntactic or data-type mismatches, keeping track of any functions or data structures that have been used but not yet defined, and suggesting consistent default values for some parts of the program. A debugging advisor can keep track of breakpoints, testing code, and temporary changes, with explanations for each. A module-level advisor will monitor interface consistency and name conflicts. A system-level advisor may keep track of large-scale system configuration issues and possible problems introduced when many programmers work the same system at once. All of these observations are linked into appropriate places in the hypercode, so that the user can jump directly to the location or locations in question.

While Gwydion will primarily support the development of new code in the Dylan language, it must also provide excellent support for combining Dylan code with modules written in existing languages, particularly C, C++, and Ada. The advisors will have knowledge of the interface requirements for linking to libraries and call-outs to a limited set of other languages.

As a programmer incrementally compiles, loads and runs code fragments the advisory process is a set of snapshots of portions of the code database. When a larger subset of the object database and code is compiled, the environment will string these together to animate the progression. This can be combined with graphic displays to profile the activities of the system in a more human readable form than conventional profiling tools.

The Gwydion Project

We are in the startup phase of a three-year project to create the Gwydion environment and make it widely available to the research community and to industry. We want the system to be widely used on real application projects so that we will get good feedback on the design. We have already implemented a prototype Dylan implementation built on top of CMU Common Lisp which is tracking (and sometimes anticipating) the Dylan design as it evolves. We expect to produce a good-quality stand-alone Dylan implementation within the next year. After an initial period of bootstrapping, Gwydion will be written in Dylan and later versions of Gwydion will be developed under early versions of the Gwydion environment.

Resources

The Gwydion project team currently consists of Scott Fahlman plus five full-time staff members, all veterans of the CMU Common Lisp and the Gandalf projects. The Gandalf project explored advanced software-development environments for more traditional programming languages. In addition, the project currently employs four very good undergraduates as part-time programmers.

This team brings unique skills and experience to a project such as Gwydion. The CMU CL project (known in the early days as Spice Lisp) has been involved in the implementation of advanced Lisp systems and programming environments on standard workstations for over a decade. We played a central role in the design, implementation, and popularization of Common Lisp as that language grew from a mere suggestion by ARPA to a widely-used standard that forms the foundation of the worldwide AI industry. Our code made it possible for many companies to get into the Common Lisp game while minimizing their initial cost and risk.

Much of our effort over the past few years has been spent developing and integrating our new Python compiler for Common Lisp. Python sets a new standard for Lisp compilers in a number of areas (see section [*].) Another area of expertise is in the integration of static and dynamic languages: our Alien facility embeds conventional data structures into Lisp's dynamic type system in a way that is both natural and highly efficient (see section [*].) We also have considerable experience in programming environment development: the Hemlock editor is tightly integrated with the compiler and run-time system, and provides many incremental development and browsing capabilities that have only recently become available in commercial products.

With the release of CMU CL for Sparc/SunOS in the fall of 1991, we suddenly found ourselves supporting a user community of several hundred individuals and projects worldwide. In addition to the impoverished university researchers and a few large projects for whom unrestricted access to our sources is important, we also find that CMU CL is being used for small "bootleg" projects in many companies and government labs. Apparently these people need to prove that Lisp is the right tool for some task before trying to persuade their management to buy a commercial Common Lisp license.

We intend to play a similar role in the growth and widespread acceptance of the Dylan language, which we see as a significant improvement over Common Lisp for many kinds of programming. In fact, we have already been told by a number of other groups that our participation in Dylan gives them confidence that the language will be a widespread success, and not just an internal project of Apple. We believe that the Gwydion environment will showcase the capabilities of Dylan in the best possible light.

Our group is still supporting CMU Common Lisp because we feel a certain responsibility to our external users and because we are using CMU CL as a cross-development platform in the initial phases of Gwydion. However, this support is at a much-reduced level, since CMU CL is no longer our primary task.

We are currently starting up a Gwydion consortium for companies that want to work with us to commercialize the products of our research. Consortium members will receive a license to use our code and documents commercially and will have easy access to the Gwydion design team. We want to work closely with these industrial partners to make sure the system meets their real-world needs. In particular, our decisions about which machines and operating systems to support will be strongly influenced by the desires of our industrial partners.