Comprehensive Haskell Sandboxes Posted on February 10, 2013

For an alternative approach, see Comprehensive Haskell Sandboxes, Revisited instead.

Serious Haskell developers will want to test their packages against different versions of ghc, different versions of libraries, etc. It is therefore useful to have multiple Hackage development environments, or sandboxes. A comprensive sandbox contains:

An installation of ghc

A package database

cabal configuration (for instance, so that we can pin packages)

Moreover, good sandboxes are

isolated from each other: only the active sandbox can be modified relocatable: it should be possible to copy the sandbox ~/env/ghc762 to ~/env/client1 to use as the basis for a new sandbox, or to copy the entire set of sandboxes (~/env) from one machine to another (suitably similar) machine (this means that paths in config files or baked into executables should never mention the name of the sandbox) self-contained: there should be no need for a system wide install of ghc transparent: we should be able to use the standard tools (ghc, cabal) as they are

There are existing tools for Haskell sandboxes (such as cabal-dev or hsenv), but none satisfy all of the above requirements (to the best of my knowledge). However, it is easy enough to set up our own.

The basic approach is:

We are going to keep our sandboxes in ~/env/. For instance, we might have ~/env/ghc742 and ~/env/ghc762. The active sandbox is a symlink ~/env/active → ~/env/ghc742 We set up three “root” symlinks: ~/.cabal → ~/env/active/dot-cabal for cabal

~/.ghc → ~/env/active/dot-ghc for ghc

~/local → ~/env/active/local for software installed in this sandbox (for instance, we will have ghc as ~/local/bin/ghc).

That’s basically it. The devil, of course, is in the details. In the discussion below we assume that the user’s home directory is /Users/dev (we can’t use ~ everywhere because some tools require an absolute path).

Bash prompt

I find it useful if my bash prompt tells me which development environment is active. I therefore add

prompt() { ACTIVE= ` readlink ~/env/active ` PS1= "

[\e[31m ${ACTIVE} \e[0m \e[32m\w\e[0m]

# " } PROMPT_COMMAND= prompt

to my ~/.profile. This will give me a prompt such as

[ghc762 ~/project/cloud-haskell] #

Procedure for adding a new development environment

Create a new directory in ~/env/foo, and create subdirectories ~/env/foo/local, ~/env/foo/dot-cabal and ~/env/foo/dot-ghc. Activate the new sandbox by symlinking ~/env/active to ~/env/foo. Install ghc from bindist ./configure --prefix=/Users/dev/local make install Install cabal-install from source, making sure to install profiling libraries (if you want them) EXTRA_CONFIGURE_OPTS=--enable-library-profiling ./bootstrap.sh Run cabal update Modify ~/.cabal/config ; enable library profiling and pin ghc dependencies (see below) Use cabal install to install standard useful tools such as alex, happy, cabal-dev (cabal-dev 0.9.1 does not compile with ghc 7.6.1; install from github instead), hlint, hasktags, haddock.

It is also possible to copy the “cabal” binary from a different sandbox, so that you don’t have to build it from scratch. This is not necessarily true for other binaries though, unless you also copy some other data; for instance, utilities like haddock and alex rely on data installed in ~/.cabal.

Pinning ghc dependencies

Since I often want to link against the ghc package I find it useful to pin all of ghc’s direct dependencies to avoid the diamond dependency problem. For instance, suppose that we have ghc 7.6.1, linked against containers-0.5.0.0. Without pinning ghc’s dependencies, if we later installed package X, and package X got linked against containers-0.5.2.1, we would not be able to install package Y which requires both X (compiled against containers-0.5.2.1) and ghc-api (compiled against containers-0.5.0.0). Of course we can reinstall packages at this point, but simply pinning ghc’s dependencies avoids these problems.

If there is a handy way to find all of a package (transitive) dependencies using ghc-pkg, then I am not aware of it. I use the following script instead (AllDeps.sh):

#!/bin/bash echo $1 for pkg in ` ghc-pkg field $1 depends ` do if [ " $pkg " != "depends:" -a " $pkg " != "builtin_rts" ] then unqualified= ` echo $pkg | rev | cut -d- -f2- | rev ` $0 $unqualified fi done

Running AllDeps.sh ghc-7.4.2 will give you a big long list of all of ghc-7.4.2’s transitive dependencies, with lots of duplication in it. Run the result through sort and then uniq to remove these duplicates, and then pin these dependencies. For ghc-7.4.2 that means adding these lines to ~/.cabal/config (which is now really ~/env/ghc742/dot-cabal/config, so that we can have different cabal configurations in different development environments):

constraint: Cabal==1.14.0 constraint: array==0.4.0.0 constraint: base==4.5.1.0 constraint: bin-package-db==0.0.0.0 constraint: binary==0.5.1.0 constraint: bytestring==0.9.2.1 constraint: containers==0.4.2.1 constraint: deepseq==1.3.0.0 constraint: directory==1.1.0.2 constraint: filepath==1.3.0.0 constraint: ghc==7.4.2 constraint: ghc-prim==0.2.0.0 constraint: hoopl==3.8.7.3 constraint: hpc==0.5.1.1 constraint: integer-gmp==0.4.0.0 constraint: old-locale==1.0.0.4 constraint: old-time==1.1.0.0 constraint: pretty==1.1.1.0 constraint: process==1.1.0.1 constraint: template-haskell==2.7.0.0 constraint: unix==2.5.1.1

Or for ghc 7.6.2:

constraint: Cabal==1.16.0 constraint: array==0.4.0.1 constraint: base==4.6.0.1 constraint: bin-package-db==0.0.0.0 constraint: binary==0.5.1.1 constraint: bytestring==0.10.0.2 constraint: containers==0.5.0.0 constraint: deepseq==1.3.0.1 constraint: directory==1.2.0.1 constraint: filepath==1.3.0.1 constraint: ghc==7.6.2 constraint: ghc-prim==0.3.0.0 constraint: hoopl==3.9.0.0 constraint: hpc==0.6.0.0 constraint: integer-gmp==0.5.0.0 constraint: old-locale==1.0.0.5 constraint: old-time==1.1.0.1 constraint: pretty==1.1.1.0 constraint: process==1.1.0.2 constraint: template-haskell==2.8.0.0 constraint: time==1.4.0.1 constraint: unix==2.6.0.1

Even if you never link anything against the ghc package, you might still want to pin template-haskell, because upgrading that without upgrading ghc will almost certainly not work.

The Haskell Platform

If you want to install a sandbox for the Haskell platform, start by creating a sandbox with the appropriate ghc compiler and the appropriate cabal-install (the Haskell Platform changelog should tell you what the appropriate versions are). When installing cabal-install, modify the bootstrap.sh script so that it installs the appropriate versions of the libraries needed for cabal-install itself. For instance, for HP 2012.4.0.0, we have

PARSEC_VER="3.1.3"; PARSEC_VER_REGEXP=${PARSEC_VER} DEEPSEQ_VER="1.3.0.0"; DEEPSEQ_VER_REGEXP=${DEEPSEQ_VER} TEXT_VER="0.11.2.3"; TEXT_VER_REGEXP=${TEXT_VER} NETWORK_VER="2.3.1.0"; NETWORK_VER_REGEXP=${NETWORK_VER} CABAL_VER="1.14.0"; CABAL_VER_REGEXP=${CABAL_VER} TRANS_VER="0.3.0.0"; TRANS_VER_REGEXP=${TRANS_VER} MTL_VER="2.1.2"; MTL_VER_REGEXP=${MTL_VER} HTTP_VER="4000.2.5"; HTTP_VER_REGEXP=${HTTP_VER} ZLIB_VER="0.5.4.0"; ZLIB_VER_REGEXP=${ZLIB_VER} TIME_VER="1.4" TIME_VER_REGEXP=${TIME_VER} RANDOM_VER="1.0.1.1" RANDOM_VER_REGEXP=${RANDOM_VER}

Next we want to pin all the platform packages – not much point in having a sandbox for a Haskell Platform version to test our package if cabal installs different versions of the platform packages. Download the .cabal file for the Haskell Platform (with a bit of luck the Haskell Platform page itself will give you a link to the appropriate .cabal file; otherwise you might find it in the github repository) and copy its dependencies as constraints to your cabal config (like we did above) – don’t forget to enable library profiling too, if you want the profiling libraries available. Next I would use cabal install to install the appropriate versions of alex and happy (the HP cabal file tells you which versions to use). Finally, you can install all platform packages by running

cabal install --only-dependencies haskell-platform.cabal

(or you install them “lazily” as they as needed, relying on the cabal config configuration to install the right versions). If you have cabal-install 1.16 or higher to might want to use the “-j” flag to enable parallel builds.

Switching quickly

To switch quickly between two environments, you could create the following script and save it as “chenv” somewhere in your PATH:

#!/bin/bash if [ $# -ne 1 ] then echo "Usage: $0 env" exit fi if [ -d ~/env/ $1 ] then pushd . > /dev/null cd ~/env rm active ln -s $1 active popd > /dev/null else echo $1 is not a directory fi hash -r

And if you wanted to get really fancy you could add bash autocompletion support by adding this to your ~/.profile:

function CompleteChenv() { local envs= ` ls ~/env ` local word=${COMP_WORDS[COMP_CWORD]} COMPREPLY=( ` compgen -W " $envs " -- " $word " ` ) } complete -F CompleteChenv chenv

Compiling GHC

If you want to create an additional build environment with a ghc compiled from source, there is a little twist you need to take care of, since building ghc requires ghc. Here’s how you can do it:

Make sure ~/env/active points to a working ghc compiler, let’s say ~/env/ghc762 Create a new environment, let’s say ~/env/ghc762src, and create the necessary subdirectories (local, dot-cabal and dot-ghc). Do not activate the new sandbox. Unpack ghc in ~/env/ghc762src/local/src. Follow the usual build instructions, but configure with ./configure --prefix=/Users/dev/local This makes sure that we bake in the right paths, even though ~/local is currently pointing to ~/env/ghc762/local, which is not where we want to install our new compiler. Once everything has been built, install with make install prefix=/Users/dev/env/ghc762src/local This puts the files in the right place but all the paths refer to ~/local. (You might think you can run the “make install” without overriding “prefix” by activating the sandbox at this point – that will not work, however; make will try and fail to rebuild stuff.) At this point you can change ~/env/active to ~/env/ghc762src to start using your new compiler.

If you want to hack on ghc you probably don’t want to do the install step at all, and just use ~/local/src/ghc-7.6.2/inplace/bin/ghc-stage2 directly; either use cabal’s –with-ghc option or (more conveniently) create a handful of symlinks (mutatis mutandis):

~/local/bin/ghc → ~/local/src/ghc-7.6.2/inplace/bin/ghc-stage2

~/local/bin/ghc-7.6.2 → ~/local/src/ghc-7.6.2/inplace/bin/ghc-stage2

~/local/bin/ghc-pkg → ~/local/src/ghc-7.6.2/inplace/bin/ghc-pkg

~/local/bin/hsc2hs → ~/local/src/ghc-7.6.2/inplace/bin/hsc2hs

If you want to hack on ghc now, you can simply go to ~/local/src/ghc-7.6.2, make changes, and as long as you don’t need the stage 0 compiler (the bootstrap compiler) you can just compile without changing your sandbox. For example, if you wanted to make a change to the ghc package itself, run “make 2” in the compiler/ directory (which should be very quick) and then just relink your own package and you’ll have the modified ghc package (as long as you are using the in-place ghc).

Note that if you use the in-place compiler the sandbox is not relocable (the ghc script will include explicit references to the full path of the sandbox).

Building GTK

(This section is Mac OS X specific.) The GTK libraries themselves are independent of our Haskell environment, of course, but I don’t like polluting my top-level directory with all the directories that the standard installation procedure creates. To avoid too much pollution, you can make the following changes.

Create a directory ~/env/gtk and download gtk-osx-build-setup.sh there. Then modify that script and change SOURCE to $HOME/env/gtk/Source Create a directory ~/env/gtk/dot-local and a corresponding symlink ~/.local. Add ~/.local/bin to your PATH. Run the script Modify ~/.jhbuildrc-custom and add checkoutroot = os.path.expanduser("~/env/gtk/Source/gtk") prefix = os.path.expanduser("~/env/gtk/inst") Run the jhbuild incantations from the GTK build instructions to complete your GTK installation. For Haskell purposes it is also useful to install librsvg jhbuild build librsvg

This gives us the GTK libraries, but not yet their Haskell bindings. The Haskell bindings will of course be relative to whatever development environment is currently active. Now inside a jhbuild shell:

cabal install gtk2hs-buildtools cabal install gtk -fhave-quartz-gtk cabal install svgcairo # Optional, requires librsvg

At this point you should have the Haskell GTK libraries available, and you should be able to exit the jhshell and install other Hackages that rely on Gtk2Hs (such as threadscope) without the jhshell. (If you do install svgcairo, you’ll want to install graphviz too.)

Known Limitations

The sandboxes are designed to completely isolated from each other. If you activate sandbox A, you will not be able to use package DB from another sandbox, even if you specify the right paths. The problem is that the package registrations (use ghc-pkg describe to see them) point to ~/.cabal ; so if you activate a different sandbox (i.e., change what ~/.cabal points to) then these paths in the package registrations will be incorrect. You can change this by setting

install-dirs user prefix: /Users/dev/env/someSandbox/dot-cabal

in your ~/.cabal/config for someSandbox; this will make sure that the package registrations will use absolute paths instead. Of course, this means that the sandbox is no longer relocatable because you are now hardcoding paths (it might be possible to resolve these conflicting goals with the use of $pkgroot, but Cabal support for that is still limited).