Problems for Users

Any veteran GNU/Linux user has, at one point or another, run across a package which used the autoconf/automake toolset. There is a lot to be said in favor of this emerging standard. Running "./configure && make && make install" usually results in a working installation of whatever package you are attempting to compile. The autoconf tools are also portable to almost every *nix platform in existence, which generally makes it easier to release your program for a large variety of systems. However, despite these few pluses, the auto* tools are constantly a thorn in the side of users and developers alike.

Let's take a typical autoconf package. I'll call it package-xyz. Joe GNU/Linux User has just downloaded and untared this package. Like most users, the first thing he does is change directory to the newly-unpacked source tree and run a quick "ls" to see what files are there. To his delight, he discovers a "configure" script, indicating that he probably doesn't have to do any editing of Makefiles or other such craziness. Little does he realize the troubles awaiting him.

He again does the typical thing and runs "./configure --prefix=/opt". The configure script runs for a while, then exits with an error which basically translates to "You have an autoconf version which is three weeks old; please upgrade", but this is displayed in the most cryptic manner possible. He won't realize this is indeed what the error message means until he runs a few quick Google searches. He really wants to install this program, so he doesn't give up quickly. A few minutes later, he's run apt-get upgrade (or run whatever auto update his distribution uses).

He decides that he would like to customize his package, so he runs "./configure --help". A list of options puke themselves all over his screen. Undeterred, he runs configure again, but this time pipes the output to his favorite pager. He's forced to parse this output because there is no real standard for the options to pass to configure. He wants to use GTK, but how does he do it? Is it "--with-gtk" or "--enable-gtk"? Perhaps this time it's "--enable-gnome", or maybe "--with-toolkit=gtk". In any case, he finally "gets" it and runs the configure script, customizing the package to his heart's delight.

Then he discovers he's missing lib-lzw-3.2.3.4. He takes the time to go through the same rigamarole with that package and, after a bit of tinkering, gets it to install. Then he discovers that configure still thinks that the library is not there. He does a bit of investigation and discovers that he needs to delete config.cache. That's if he's lucky and the "cached" option isn't in some other random directory or file (in which case he would remove the source tree and start again from scratch).

The configure script now runs properly and outputs a nice Makefile, so Joe runs "make". To his surprise, configure, for no apparent reason, decides to run again. That's ok; Joe sits patiently through the same tests as they run over and over (this time, hopefully, cached). While looking through the log messages, he happens to wonder if there is, perhaps, maybe, some way to not have to run the same 50 tests over and over and over. After a while, make runs, starts compiling the program, and errors out. Joe reruns make to see what the error is, redirecting stderr so he can scroll up and see the original error message of the 95 presented. To his amazement, while configure picked up the fact that he did indeed have lib-lzw-3.2.3.4 installed, it failed to realize that the header files were located in /usr/include/lzw, not in /usr/include. At this point, Joe has a real problem.

Our friend Joe is a seasoned *nix user, so his gut reaction is to start Emacs (one thing the FSF got right) and edit the makefile. He does this only to discover a 50,000-line-long monstrosity. He greps it, looking for the right variable to edit. Of course, there are 500 different $(INCLUDE) settings spattered throughout. Little does he realize that the variable he's looking for is three directories down and called $(YOUD_NEVER_GUESS_THIS_VARIABLE_NAME_HAHA). He finally finds it and edits it to the proper value. configure again decides to run for no apparent reason (even though "make" skipped it the last five times) and overwrites all this hard work.

Now, Joe is presented with an interesting problem. He realizes that he needs to edit something besides the Makefile. But what does he look at? configure.in? Makefile.am? Makefile.in? Makefile.yoyo? Makefile.banana?

By now, the average user has done one of the following:

Given up and tried a different package. Shot himself in the head with a twelve gauge shotgun.

Joe is feeling masochistic, and continues trying. I'll save you the pain of talking about his later problems and the pain of hearing about Sally FreeBSD user, who wants to configure a package to run under GNU/Linux on her iPaq.

Problems for Developers

Your average developer has no clue what m4 is or how it works. Most Unix people (except the old diehards) have not even heard of m4. That didn't stop the autoconf guys from using it.[1]

Here's the issue: In order to be able to write your autoconf setup files properly (without a considerable amount of pain and suffering), you must first learn m4. Let's follow the path of a GNU/Linux developer, a creative lady named Jane.

Jane is a seasoned developer. She is accustomed to writing Makefiles, but decides she wants the extra portability that autoconf allows. She isn't familiar with m4, but thinks she can wade through building her files anyway.

First, she looks for a tutorial on how to use autoconf. Sadly, she will spend a lot of time looking. If she's lucky, she'll discover a five-year-old online copy of a book that might help her accomplish 5% of what she needs to do.

The GNU info system isn't much help, either. The documentation she needs is spread over 50 info pages, divided into three packages. These divisions don't help much, as configure.in sometimes makes Makefile.am do strange things. Eventually, she gives up writing the files and does what every other developer does with the auto* tools: copy someone else. There are perhaps 30 developers worldwide who understand the autoconf tool chain. (I doubt there is anyone who quite understands the braindead Makefile that is puked out.) This very problem is what caused many of Joe's issues. Well-written configure scripts are rare things of beauty. (They are mostly found in GNU projects, but I've seen a few others.)

The Root of the Problem

The autoconf/configure system is clever, but in the end, it is just a very creative hack to work around the deficiencies of Makefiles. Those who have used *nix for a long while have seen several such workarounds. There's the Sleepy-Cat libdb approach of having 50 Makefiles, one for each architecture. There's the "config.h" approach seen in several packages. There's "imake", used by X11 projects.[2]

It's easy to see why autoconf resorts to outputting 50,000-line-long Makefiles. Makefiles lack support for complex dependencies and complicated error checking. Many of the above tools use the same hacks, like using "touch" to create files used for dependencies, with complicated build rules for making those files. A paper on the various Makefile techniques could fill volumes.

Solutions

There are several unique and innovative projects being developed to help sort out the quagmire in build processes.

Two such solutions are "SCons" and "Cons". Both try to replace Make with something far more flexible. However, they both depend on tools which are a bit less standard than sh and m4: Perl and Python. Still, both languages are quickly becoming standard on the major Unixes. SCons is a bit less mature, but is preferred by the author of this paper. Using either allows a much clearer build process for complicated software. There is no requirement to generate a set of build instructions from a template.

Another interesting piece of software is A-A-P. It allows a user to create "recipes" for building software.

Any of these is cleaner and easier than autoconf. I like the familiarity that Make brings to the table, but it's time to face up to the fact that Makefiles introduce far more complexity than needs to be.