Buildapp - Create executables with SBCL or CCL

Buildapp is an application for SBCL or CCL that configures and saves an executable Common Lisp image. It is similar to cl-launch and hu.dwim.build. Buildapp is available under a BSD-style license. The latest version is 1.5.6, released on November 7th, 2015. Download shortcut: http://www.xach.com/lisp/buildapp.tgz

Contents

Installation

Buildapp does not require any libraries. To compile it with SBCL you simply run make install . To compile with a different lisp, just specify it after the make command: make install LISP=<my_favorite_lisp> . For example, you wanted to compile it under CCL you'd say make install LISP=ccl By default, it is installed in /usr/local/bin ; to use another location, use make DESTDIR=/path install .

You can also create the buildapp binary by loading the buildapp system with asdf and running (buildapp:build-buildapp) . Example Use

Here's a small application:

$ buildapp \ --eval '(defun main (argv) (declare (ignore argv)) (write-line "Hello, world"))' \ --entry main \ --output hello-world [undoing binding stack and other enclosing state... done] [saving current Lisp image into hello-world: writing 6352 bytes from the read-only space at 0x20000000 writing 4064 bytes from the static space at 0x20100000 writing 44834816 bytes from the dynamic space at 0x1000000000 done] $ ./hello-world Hello, world

The following creates a toy curl-like application. (It's not quite practical, because any errors will land you in the interactive debugger.)

$ buildapp --output lisp-curl --asdf-path ~/src/clbuild/systems/ \ --load-system drakma \ --eval '(defun main (args) (write-string (drakma:http-request (second args))))' \ --entry main ;; loading system sb-grovel (needed by drakma) ;; from /usr/local/lib/sbcl/sb-grovel/ ;; loading system sb-posix (needed by cl+ssl) ;; from /usr/local/lib/sbcl/sb-posix/ ;; loading system trivial-gray-streams (needed by chunga, cl+ssl, flexi-streams) ;; from /home/xach/src/clbuild/source/trivial-gray-streams/ ;; loading system flexi-streams (needed by drakma, cl+ssl) ;; from /home/xach/src/clbuild/source/flexi-streams/ ;; loading system alexandria (needed by cffi, babel) ;; from /home/xach/src/clbuild/source/alexandria/ ;; loading system trivial-features (needed by cffi, babel) ;; from /home/xach/src/clbuild/source/trivial-features/ ;; loading system babel (needed by cffi) ;; from /home/xach/src/clbuild/source/babel/ ;; loading system cffi (needed by cl+ssl) ;; from /home/xach/src/clbuild/source/cffi/ ;; loading system cl+ssl (needed by drakma) ;; from /home/xach/src/clbuild/source/cl+ssl/ ;; loading system sb-bsd-sockets (needed by usocket) ;; from /usr/local/lib/sbcl/sb-bsd-sockets/ ;; loading system usocket (needed by drakma) ;; from /home/xach/src/clbuild/source/usocket/ ;; loading system chunga (needed by drakma) ;; from /home/xach/src/clbuild/source/chunga/ ;; loading system cl-base64 (needed by drakma) ;; from /home/xach/src/clbuild/source/cl-base64/ ;; loading system puri (needed by drakma) ;; from /home/xach/src/clbuild/source/puri/ ;; loading system drakma ;; from /home/xach/src/clbuild/source/drakma/ [undoing binding stack and other enclosing state... done] [saving current Lisp image into lisp-curl: writing 6352 bytes from the read-only space at 0x20000000 writing 5472 bytes from the static space at 0x20100000 writing 61722624 bytes from the dynamic space at 0x1000000000 done] $ ./lisp-curl http://xach.com/ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd"> <HTML> <HEAD> <TITLE>www.xach.com</TITLE> </HEAD> ...

Here's how the l1sp.org redirection service application is built:

$ make buildapp --output l1sp --entry redirector:main \ --asdf-path /opt/l1sp/systems \ --require sb-aclrepl \ --eval '(pushnew :hunchentoot-no-ssl *features*)' \ --load-system swank \ --eval '(setf swank:*log-output* nil)' \ --load-system redirector ;; loading system sb-grovel ;; from /usr/local/lib/sbcl/sb-grovel/ ;; loading system sb-bsd-sockets ;; from /usr/local/lib/sbcl/sb-bsd-sockets/ ;; loading system sb-introspect ;; from /usr/local/lib/sbcl/sb-introspect/ ;; loading system sb-posix ;; from /usr/local/lib/sbcl/sb-posix/ ;; loading system sb-cltl2 ;; from /usr/local/lib/sbcl/sb-cltl2/ ;; loading system swank ;; from /opt/l1sp/src/slime/ ;; loading system html-template (needed by redirector) ;; from /opt/l1sp/src/html-template-0.9.1/ ;; loading system sb-rotate-byte (needed by sb-md5) ;; from /usr/local/lib/sbcl/sb-rotate-byte/ ;; loading system sb-md5 (needed by redirector) ;; from /usr/local/lib/sbcl/sb-md5/ ;; loading system cl-who (needed by redirector) ;; from /opt/l1sp/src/cl-who-0.11.1/ ;; loading system cl-ppcre (needed by hunchentoot, redirector) ;; from /opt/l1sp/src/cl-ppcre-2.0.0/ ;; loading system url-rewrite (needed by hunchentoot) ;; from /opt/l1sp/src/url-rewrite-0.1.1/ ;; loading system rfc2388 (needed by hunchentoot) ;; from /opt/l1sp/src/rfc2388/ ;; loading system md5 (needed by hunchentoot) ;; from /opt/l1sp/src/md5-1.8.5/ ;; loading system cl-fad (needed by hunchentoot) ;; from /opt/l1sp/src/cl-fad-0.6.2/ ;; loading system cl-base64 (needed by hunchentoot) ;; from /opt/l1sp/src/cl-base64-3.3.2/ ;; loading system trivial-gray-streams (needed by flexi-streams) ;; from /opt/l1sp/src/trivial-gray-streams-2006-09-16/ ;; loading system flexi-streams (needed by chunga) ;; from /opt/l1sp/src/flexi-streams-1.0.7/ ;; loading system chunga (needed by hunchentoot) ;; from /opt/l1sp/src/chunga-0.4.3/ ;; loading system hunchentoot (needed by redirector) ;; from /opt/l1sp/src/hunchentoot-0.15.7/ ;; loading system redirector ;; from /opt/l1sp/src/redirector/ [undoing binding stack and other enclosing state... done] [saving current Lisp image into l1sp: writing 6176 bytes from the read-only space at 0x20000000 writing 4064 bytes from the static space at 0x20100000 writing 61042688 bytes from the dynamic space at 0x1000000000 done] $ ./l1sp ;; Swank started at port: 7717. CL-USER(1):

The (setf swank:*log-output* nil) eval is needed to avoid problems when the image restarts.

redirector::main looks like this:

(defun main (argv) (declare (ignore argv)) (load "/opt/l1sp/etc/init.lisp") (sb-impl::toplevel-repl nil))

Here's an example of the --dispatched-entry option, which was inspired by the desire to have a dozen different small utilities embedded in one big executable and called based on the binary name. First, the support files:

;;;; utils.lisp (defpackage #:utils (:use #:cl)) (in-package #:utils) (defun main (argv) (let ((name (pathname-name (first argv)))) (format *error-output* "Unknown binary name ~S, try using cl-echo, cl-ls, or cl-true~%" name) (sb-ext:quit :unix-status 1))) ;;;; ls.lisp (defpackage #:ls (:use #:cl)) (in-package #:ls) (defun main (argv) (declare (ignore argv)) (dolist (file (directory "*.*")) (write-line (namestring file)))) ;;;; echo.lisp (defpackage #:echo (:use #:cl)) (in-package #:echo) (defun main (argv) (format t "~{~A~^ ~}~%" (rest argv))) ;;;; true.lisp (defpackage #:true (:use #:cl)) (in-package #:true) (defun main (argv) (declare (ignore argv)) (sb-ext:quit :unix-status 0))

Buliding it all together looks like this:

$ buildapp --output utils \ --load utils.lisp --dispatched-entry /utils:main \ --load ls.lisp --dispatched-entry cl-ls/ls:main \ --load echo.lisp --dispatched-entry cl-echo/echo:main \ --load true.lisp --dispatched-entry cl-true/true:main ;; loading file #P"/tmp/demo/utils.lisp" ;; loading file #P"/tmp/demo/ls.lisp" ;; loading file #P"/tmp/demo/echo.lisp" ;; loading file #P"/tmp/demo/true.lisp" [undoing binding stack and other enclosing state... done] [saving current Lisp image into utils: writing 6352 bytes from the read-only space at 0x20000000 writing 4064 bytes from the static space at 0x20100000 writing 45223936 bytes from the dynamic space at 0x1000000000 done] $ ln -sf utils cl-ls $ ln -sf utils cl-echo $ ln -sf utils cl-true $ ./cl-ls /tmp/demo/cl-true /tmp/demo/echo.lisp /tmp/demo/ls.lisp /tmp/demo/true.lisp /tmp/demo/utils /tmp/demo/utils.lisp $ ./cl-echo Hello world Hello world $ ./cl-true && echo $? 0

Overview

Here is the usage output of buildapp:

Usage: buildapp --output OUTPUT-FILE [--flag1 value1 ...] Required flags: --output OUTPUT-FILE Use OUTPUT-FILE as the name of the executable to create Entry-point flags: --entry NAME Use the function identified by NAME as the executable's toplevel function. Called with SB-EXT:*POSIX-ARGV* as its only argument. If NAME has a colon, it is treated as a package separator, otherwise CL-USER is the implied package. --dispatched-entry DNAME Specify one possible entry function, depending on the name of the file that is used to start the application. The syntax of DNAME is APPLICATION-NAME/ENTRY-NAME. If the name used to start the executable matches APPLICATION-NAME, use ENTRY-NAME as the entry point. This can be used to choose one of many possible entry points by e.g. symlinking names to the application executable. If APPLICATION-NAME is empty, the specified ENTRY-NAME is used as a default if no other application names match. There may be any number of dispatched entry points, but only one default. Action flags: --load FILE Load FILE. CL:*PACKAGE* is bound to the CL-USER package before loading --load-system NAME Load an ASDF system identified by NAME --require NAME Use CL:REQUIRE to load NAME --eval CODE Use CL:EVAL to evaulate CODE. The code is read with CL:READ-FROM-STRING in the CL-USER package There may be any number of load/load-system/require/eval flags. Each is executed in command-line order before creating an executable. Load path flags: --load-path DIRECTORY When handling a --load, search DIRECTORY for files to load --asdf-path DIRECTORY When handling a --load-system, search DIRECTORY for ASDF system files to load --asdf-tree DIRECTORY When handling a --load-system, search DIRECTORY and all its subdirectories for ASDF system files to load --manifest-file FILE When handling a --load-system, read a list of ASDF system file pathnames from FILE as possible matching systems. There may be any number of load-path/asdf-path/asdf-tree/manifest-file flags. They take priority in command-line order. Other flags: --compress-core Compress the core or executable; requires configuration support in SBCL --core-only Make a core file only, not an executable --dynamic-space-size MB Pass a --dynamic-space-size option to SBCL when building; value is megabytes --help Show this usage message --logfile FILE Log compilation and load output to FILE --sbcl PATH-TO-SBCL Use PATH-TO-SBCL instead of the sbcl program found in your PATH environment variable For the latest documentation, see http://www.xach.com/lisp/buildapp/

Limitations

Buildapp is limited in scope. It aims to make the following steps easy:

Define an application environment by loading files, loading ASDF systems, evaluating code, and using CL:REQUIRE

Dump an executable image with an arbitrary startup function

By design, it does not handle the following tasks:

Initialization tasks at app startup (use SB-EXT:*INIT-HOOKS* for that)

Debugger management (use *DEBUGGER-HOOK* for that)

for that) Command-line processing (use a library such as COMMAND-LINE-ARGUMENTS for that)

Obtaining libraries (use clbuild, LibCL, cl-librarian, asdf-install, etc. for that)

Implementation

Buildapp works like this:

It processes the command-line and creates an object that captures the command-line requirements: the output file, any eval/load/load-system/require actions, the entry function, etc. It creates a new Lisp file (the dumpfile) with all the commands needed to implement the command-line options. It runs either "sbcl" with sb-ext:run-program or "ccl" with ccl:run-program . In either case lisp is invoked with no init files (it doesn't read user or system rc files) and loads the dumpfile: The first few commands of the dumpfile establish a specialized loading environment: The debugger is changed with *invoke-debugger-hook* to simply quit with a special exit code instead of entering the normal debugger (in CCL the same behavior is accomplished using the :application and :error-handler arguments to ccl:save-application )

to simply quit with a special exit code instead of entering the normal debugger (in CCL the same behavior is accomplished using the and arguments to ) Most output (ASDF system compilation output, low-level error messages, etc) is redirected to a log stream; that stream can be directed to a file with the --logfile argument

argument Stale compiled files are automatically recompiled with an :around method on asdf:perform There are some sanity checks: Is the output file writable? Does this version of sbcl support the required :save-runtime-options argument? The dumpfile performs the eval/load/load-system/require actions. Each operation is evaluated in the cl-user package and with a fresh binding of sb-ext:*invoke-debugger-hook* in sbcl or ccl::*debugger-hook* in ccl to the buildapp debugger function. If the binding is modified, the new value is saved. The dumpfile clears itself out of the environment: Remove extra ASDF methods with remove-method

Reset the debugger hook to the saved value or NIL if no value was saved.

Delete the dumpfile package with delete-package The dumpfile then creates an executable with save-lisp-and-die in sbcl or ccl:save-application in ccl. This ends the lisp subprocess. It deletes the dumpfile.

The ASDF central registry is temporarily extended with the --asdf-path and --asdf-tree arguments at load time, and reverts back to the default central registry value after that. To avoid conflicts with this behavior, changes to the central registry should be done at startup time instead of application load time. Feedback

If you have any questions or comments about buildapp, please email me, Zach Beane. License

Copyright © 2010 Zachary Beane, All Rights Reserved

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR 'AS IS' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.