November 27, 2019

A few months ago, I converted a server I’m administering to run Guix System. I wouldn’t recommend you do the same at this point (the subject of a long and tragic story yet to be told), but I did learn a lot along the way; in particular, how to get an Elm web app built and up and running, which I want to talk about here.

Aims:

document the state of my Elm packaging work, so others can build on it

show how to bypass Elm’s “highly opinionated” packaging infrastructure

provide an introduction to guix packaging

I’ve split this into a series of articles:

In this first part, we’ll go over the Guix packaging of the Elm compiler proper, a Haskell project. This work has already made it to Guix upstream:

$ guix package --show=elm-compiler name: elm-compiler version: 0.19.0 outputs: out systems: x86_64-linux i686-linux dependencies: ghc-ansi-terminal@0.8.0.4 ghc-ansi-wl-pprint@0.6.8.2 ghc-edit-distance@0.2.2.1 + ghc-file-embed@0.0.10.1 ghc-http-client-tls@0.3.5.3 ghc-http-client@0.5.13.1 ghc-http-types@0.12.3 + ghc-http@4000.3.12 ghc-language-glsl@0.3.0 ghc-logict@0.6.0.2 ghc-network@2.6.3.6 + ghc-raw-strings-qq@1.1 ghc-scientific@0.3.6.2 ghc-sha@1.6.4.4 ghc-snap-core@1.0.3.2 + ghc-snap-server@1.1.0.0 ghc-unordered-containers@0.2.9.0 ghc-utf8-string@1.0.1.1 ghc-vector@0.12.0.1 + ghc-zip-archive@0.3.3 location: gnu/packages/elm.scm:40:2 homepage: https://elm-lang.org license: Modified BSD synopsis: The `elm' command line interface, without {elm reactor description: This includes commands like `elm make', `elm repl', and many others for helping make Elm + developers happy and productive.

An overview of Elm tooling

The elm tool (sources at github.com/elm/compiler) is a command line program that provides a variety of subcommands related to building Elm applications. It’s written in Haskell, and our first step will be to build this tool with Guix (or enough of this tool to be able to run elm make successfully). The sources are at [github.com/elm

The packaging here is for the 0.19.0 version of Elm. In the meantime, a bugfix release 0.19.1 has been released.

To avoid some problems with the upstream 0.19.0 release, I initially based the packaging off the CurrySoftware fork, which is where I found some of the patches that we’ll apply to the elm sources below.

It will be useful below to have a rough idea of how the elm tool works. Elm projects come in two forms: packages and applications. Packages are distributed via package.elm-lang.org, and can be used as libraries from other packages and applications. Applications on the other hand are the “end product”, compiling to Javascript to be served to a web browser.

Both such projects are defined by a file elm.json . The elm tool is then run from a directory containing such a file. It provides a couple of commands; the relevant ones for us are:

elm make Download package dependencies to ~/.elm from package.elm-lang.org. Compile dependencies within ~/.elm to an internal format. Compile Elm modules within ./elm-stuff to the internal format. If this is an application, compile the internal format to Javascript. elm reactor This serves a web app to interactively debug your Elm application. It is itself implemented as an Elm application.

We’ll face issues with both of these: elm make ’s all-in-one design is great for ease of use if you’re doing things the way they’re meant to be done. But when building in a sandboxed environment without internet access, we’ll need to trick it to believe it actually built its package database by talking to package.elm-lang.org. Then it’ll grudgingly fall back to an offline mode and work for us.

Concretely, we’ll construct a package database in ~/.elm/0.19.0/package/ by unpacking dependency sources there and generating a versions.dat database.

The issue with elm reactor isn’t with its use, but with its build: Since it is in Elm application itself, we need to have elm make (or its logic) available to build it. The clean way to do this is with a two-stage build, where we first build the Elm compiler, then use it to build the Elm reactor. Which is all good, except the upstream build solves this is a different and rather hacky way: It calls out to itself via Template Haskell, generating the web application Javascript during the build of the elm tool.

This one we’ll address by tearing the build apart: In a first stage, we’ll patch out the reactor and build a version of the elm tool that primarily supports elm make . In a second stage we’ll re-enable the reactor, providing Javascript files built using the first stage elm make tool.

Nix deals with both of these issues in a different way. See part 2 for a brief comparison.

Packaging the elm compiler (minus reactor)

The steps we’ll follow to package elm make are:

make sure the Haskell project builds with stack and the right LTS version generate a package definition using guix import hackage -s < elm.cabal import unpackaged Haskell dependencies replace the source field to refer to the github release apply patches

We’ll go through this process explicitly below, not least to share the pain inherent in Guix packaging work.

The first step is easy enough. I found things built fine with a GHC 8.4 stackage release after applying the patch elm-compiler-relax-glsl-bound.patch to relax version bounds.

guix import hackage converts a cabal file into an expression of the form

(package (name "ghc-elm") (version "0.19.0") (source (origin (method url-fetch) (uri (string-append "https://hackage.haskell.org/package/elm/elm-" version ".tar.gz")) (sha256 (base32 "failed to download tar archive")))) (build-system haskell-build-system) (inputs `(("ghc-ansi-terminal" ,ghc-ansi-terminal) ... ("ghc-zip-archive" ,ghc-zip-archive))) (home-page "https://elm-lang.org") (synopsis "The `elm` command line interface") (description "This includes commands like `elm make`, `elm repl`, and many others for helping make Elm developers happy and productive.") (license bsd-3))

This is a scheme record, with a number of named fields. The important ones are

source This declares the package source archive. inputs This declares a list of dependencies. Typically, those are other packages. build-system This declares the build system that’s used. It’s pretty much up to the build system to define what the source and input fields actually are.

The dependencies in inputs are given as an association list of pairs ("package-name" ,package) , where the name appears to be mostly irrelevant, while the package itself is a package record like we’re defining.

Not knowing Scheme well before, I thought the comma was part of tuple syntax. Instead, it’s part of Scheme syntax for defining data that’s a mix of literal and evaluated data: quasiquote. When starting out with Guix, it’s tempting (and mostly fine) to do things by copy-and-pasting existing package definitions, but what with the sorry state of Guile error messages, there’s really no way around learning the language.

Let’s try to build this package definition: We’ll write it to a file called elm.scm , and call guix build :

$ guix build -f elm.scm ice-9/eval.scm:223:20: In procedure proc: error: package: unbound variable hint: Did you forget `(use-modules (guix packages))'?

Indeed, putting the hinted line at the top of the file helps.

The Guile module (guix packages) is defined in guix/packages.scm within the guix git repository. Generally, the guix tooling itself and packaging infrastructure is defined within guix/ , while the packages themselves are defined under gnu/ .

The next failure relates to the broken source field:

$ guix build -f elm.scm guix build: error: exception thrown: #<condition &invalid-base32-character [character: #\e string: "failed to download tar archive"] 29c9f00>

Unsurprisingly, "failed to download tar archive" isn’t a valid base32 hash. This happened because elm isn’t actually published on Hackage, so guix import failed to download the source archive and compute a hash. To fix this, we edit the file to refer to a correct URI for the elm compiler sources, put in some arbitray valid hash copied from another package definition, and eventually fix the hash to the correct version.

$ guix build -f elm.scm ice-9/eval.scm:223:20: In procedure proc: error: url-fetch: unbound variable hint: Did you forget `(use-modules (guix build download))'?

Well, not “forget” as such, but indeed: adding the import gets us further.

$ guix build -f elm.scm ice-9/eval.scm:223:20: In procedure proc: error: haskell-build-system: unbound variable hint: Did you forget a `use-modules' form?

Yes, we also need (use-modules (guix build-system haskell)) .

$ guix build -f elm.scm ice-9/eval.scm:223:20: In procedure proc: error: bsd-3: unbound variable hint: Did you forget `(use-modules (guix licenses))'?

Oh no. We did.

$ guix build -f elm.scm /home/rob/blog-test/elm.scm:17:3: In procedure inputs: error: ghc-ansi-terminal: unbound variable hint: Did you forget a `use-modules' form?

Oh, our dependencies aren’t yet imported. We can find this one:

$ guix search ghc-ansi-terminal name: ghc-ansi-terminal version: 0.8.0.4 outputs: out systems: x86_64-linux i686-linux dependencies: ghc-colour@2.3.4 location: gnu/packages/haskell-xyz.scm:288:2 homepage: https://github.com/feuerbach/ansi-terminal license: Modified BSD synopsis: ANSI terminal support for Haskell description: This package provides ANSI terminal support for Haskell. It allows cursor movement, screen clearing, color + output showing or hiding the cursor, and changing the title. relevance: 20

It’s defined in the module (gnu packages haskell-xyz) , so we’ll add an import for that. There’s a few more similar missing imports, which we fix by also importing (gnu packages haskell-check) and (gnu packages haskell-web) .

The Elm dependencies have by now all been packaged, but usually at this point you’d expect to find a few libraries that are still missing. Then we’d have to package those first. guix import has a recursive mode that can help with this.

$ guix build -f elm.scm Backtrace: In guix/store.scm: 623:10 19 (call-with-store _) In guix/scripts/build.scm: 914:26 18 (_ #<store-connection 256.99 e32ae0>) In ice-9/boot-9.scm: 829:9 17 (catch _ _ #<procedure 1bf1800 at ice-9/boot-9.scm:104…> …) In guix/ui.scm: 415:6 16 (_) In guix/scripts/build.scm: 879:5 15 (_) In srfi/srfi-1.scm: 679:15 14 (append-map _ _ . _) 592:17 13 (map1 ("x86_64-linux")) 679:15 12 (append-map _ _ . _) 592:17 11 (map1 (#<package elm@0.19.0 /home/rob/blog-test/elm.sc…>)) In guix/scripts/build.scm: 840:18 10 (_ _) In guix/packages.scm: 936:16 9 (cache! #<weak-table 358/443> #<package elm@0.19.0 /ho…> …) 1255:22 8 (thunk) 1188:25 7 (bag->derivation #<store-connection 256.99 e32ae0> #<<…> …) In srfi/srfi-1.scm: 592:29 6 (map1 (("haskell" #<package ghc@8.4.3 gnu/packages…>) …)) 592:17 5 (map1 (("source" #<origin "https://github.com/elm/…>) …)) In ice-9/boot-9.scm: 829:9 4 (catch srfi-34 #<procedure 3743870 at guix/packages.sc…> …) In guix/packages.scm: 1003:18 3 (_) In guix/store.scm: 1803:24 2 (run-with-store #<store-connection 256.99 e32ae0> _ # _ …) 1673:13 1 (_ _) In guix/build/download.scm: 741:0 0 (url-fetch _ _ #:timeout _ #:verify-certificate? _ # _ # …) guix/build/download.scm:741:0: In procedure url-fetch: Invalid keyword: #vu8(73 77 243 55 36 34 67 7 214 226 180 208 179 66 68 140 201 39 144 20 131 56 78 228 248 207 238 44 179 142 153 60)

Oops. I’ve never seen this error before going through these steps again while writing the article. It turns out that the helpful hint to import (guix build download) was misleading: We want to import the url-fetch from (guix download) instead.

$ guix build -f elm.scm [...] downloading from https://github.com/elm/compiler/archive/0.19.0.tar.gz... 0.19.0.tar.gz 608KiB/s 00:01 | 439KiB transferred sha256 hash mismatch for /gnu/store/pcxygjzvmals9yramliz2bzznz1lh9i1-elm-0.19.0.tar.gz: expected hash: 1111111111111111111111111111111111111111111111111111 actual hash: 0g4risrjrvngz3j4wf432j82gjcc8i1b7l5lwbb0fhr24hvz6ka9 hash mismatch for store item '/gnu/store/pcxygjzvmals9yramliz2bzznz1lh9i1-elm-0.19.0.tar.gz' [...]

Progress! We copy and paste the actual hash into elm.scm , and we’re actually trying to build. Though as expected, we run into trouble in the Haskell configure phase:

$ guix build -f elm.scm [...] starting phase `configure' running "runhaskell Setup.hs" with command "configure" [...] [...] Configuring elm-0.19.0... Setup.hs: Encountered missing dependencies: language-glsl >=0.0.2 && <0.3 [...]

This is where we need to apply the patch elm-compiler-relax-glsl-bound.patch . We store it in the current directory, and add it to the source definition:

(source (origin (method url-fetch) (file-name "elm-0.19.0.tar.gz") (uri "https://github.com/elm/compiler/archive/0.19.0.tar.gz") (sha256 (base32 "0g4risrjrvngz3j4wf432j82gjcc8i1b7l5lwbb0fhr24hvz6ka9")) (patches `("elm-compiler-relax-glsl-bound.patch"))))

$ guix build -f elm.scm [...] ui/terminal/src/Develop/StaticFiles.hs:92:3: error: • Exception when trying to run compile-time code: /homeless-shelter: createDirectory: permission denied (Permission denied) Code: bsToExp =<< runIO Build.compile • In the untyped splice: $(bsToExp =<< runIO Build.compile) | 92 | $(bsToExp =<< runIO Build.compile) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [...]

Finally, we run into trouble with the convoluted Elm build: It’s trying to run the compiler itself while building, which tries to write to $HOME , which is read-only inside the guix sandbox. (We could get around this by providing a writable home directory, but then we’d fail soon after when it tries to access the network.)

We fix this by adding another patch, elm-compiler-disable-reactor.patch.

$ guix build -f elm.scm [...] successfully built /gnu/store/8xn1irs377mk6n17bcrxr293qpzr6224-ghc-elm-0.19.0.drv /gnu/store/542im7drgr0p37ymaiqkh31k5ff4ghaj-ghc-elm-0.19.0 $ /gnu/store/542im7drgr0p37ymaiqkh31k5ff4ghaj-ghc-elm-0.19.0/bin/elm Hi, thank you for trying out Elm 0.19.0. I, Evan, hope you like it! [...]

Success! Wrapping this up as a package for Guix upstream requires a little bit more work:

Bind the package definition to a name, and place it in a module within the (gnu packages) hierarchy.

hierarchy. Move the patches to gnu/packages/patches/ , and add the files to the list in gnu/local.mk .

, and add the files to the list in . Rewrite the guix import -generated synopsis and description to fit the Guix style and markup format.

You can look at the complete patch in the guix repository.

In the next article, we’ll see how to use the Elm compiler within Guix to package Elm applications.