Vendoring with Quicklisp, Make, Git, and Nix Max Rottenkolber <max@mr.gy>

I have a pet project called Athens (source), it is a server application built with CCL. What it does is entirely irrelevant for the topic at hand. What follows are my thoughts on dependency management for Lisp applications, and my experience with some tools—old and new—that solved my woes.

Dependency hell For being a seemingly simple application, Athens has an awful lot of dependencies of different kinds. A dozen primary dependencies include both third-party systems from Quicklisp, as well as homegrown systems not in Quicklisp. These primary dependencies pull in countless secondary dependencies from Quicklisp, which in turn depend on C libraries provided by the OS such as OpenSSL and LibXML2. Ouch. I want building Athens to be trivial and predictable. No chasing down dependencies. I also want to be in control of when dependencies are upgraded, and be able to reproduce build and test environments faithfully. I am fine with only supporting the Unix-like operating systems supported by CCL, so platform support is restricted to those.

Making things easy Athens has a human-readable Makefile that describes how the application is built. You can cd into its source directory and type make to produce an executable (which includes the CCL kernel and the Athens Lisp image) which will act like a reasonable Unix citizen. $ cd ~/git/athens/ $ make mkdir bin ccl -Q -b -n -l quicklisp/setup.lisp -l build/athens.lisp To load "athens": Load 1 ASDF system: athens ; Loading "athens" ... du -h bin/athens 65M bin/athens $ bin/athens --help Usage: athens init <configuration> athens start <configuration> athens -h|-help|--help Building Athens. Right now, its build dependencies (minus ubiquitous core utilities and GNU Make) are CCL, OpenSSL and LibXML2, the latter two of which are also runtime dependencies that are dynamically loaded as shared objects. bin/athens: quicklisp bin build/athens.lisp $(SOURCE_OBJECTS) ccl -Q -b -n -l quicklisp/setup.lisp -l build/athens.lisp du -h bin/athens Rule to build Athens. The Make rule to build Athens invokes ccl to run the actual build script which is written in Lisp. It also loads a project-local Quicklisp installation! Athens ships with Quicklisp, and includes all its Lisp dependencies checked into the repository as part of the Quicklisp installation. The release manager (me) makes sure all is in order on the master branch. ASD = $(shell find src/ -regex '[^\#]*\.asd' -printf '%p ') VENDOR_ASD = $(shell find lib/ -regex '[^\#]*\.asd' -printf '%p ') quicklisp: ccl -Q -b -n -l lib/quicklisp/quicklisp.lisp \ -e '(quicklisp-quickstart:install :path "quicklisp/")' \ -e '(quit)' for asd in $(ASD) $(VENDOR_ASD); \ do ln -s -v ../../$$asd quicklisp/local-projects/; done Rule to vendor in a Quicklisp dist. Note: there is another, officially supported way to do essentially the same thing via Quicklisp’s bundle-systems. Athens vendors a bunch of homegrown and pinned dependencies (currently xmls-1.7 , because its API changed and I have not found the time to restore compatibility in my downstream library, yet) in lib/ . The remaining third-party dependencies are managed via the fabulous Quicklisp. The above rule installs Quicklisp into the source tree and links all non-Quicklisp systems to its local-projects/ . In order to upgrade to a new Quicklisp dist I would do $ make tidy && make …and check in the resulting quicklisp/ tree into the repository (after testing that the resulting Athens binary built with the new dist works as expected.) To avoid excessive repository bloat, Athens has some Quicklisp specific patterns in its .gitignore file. quicklisp/tmp quicklisp/dists/quicklisp/archives Patterns in .gitignore specific to Quicklisp. I manage the vendored dependencies in lib/ (including quicklisp-bootstrap) with git-subtree when possible.