I am trying nix - Part 2

This is an update on how I am getting along with nixos/nix since last time. At that point I had only installed nixos, added a few essential packages and even started editing and generating this blog on the new system. Now I want to play around with bleeding edge Haskell code and not pay a heavy price in terms of system configuration. If I can continue on from there to develop production code I will consider the experiment a success and may migrate to using nixos full-time.

Update 07–09–2016: nix continues to evolve. So much so that a blog post like this or even the nix wiki often provides out-of-date information. I am working on an updated post about nix, but until then and after, I recommend you rely on the nix manual for authoritatve informantion about nixos and nix. If you encounter problems head straight the github repository for nix and check for related issues.

About nix

Nixos is a linux distribution that uses the nix package manager to manage your system, while nix is a package manager that can be used on a variety of systems, perhaps as a replacement for homebrew on mac, or in addition to apt-get on ubuntu. There is also a cloud management tool called nixops that is very interesting in its own right.

One of the big draws of nix is the idea that it sandboxes everything. That means every package you install keeps track of all its inputs and outputs in /nix/store . Nix manages your environment by adding and removing packages from the various paths that make up your environment. Conflicting packages can easily coexist on the system, since every package can specify all its dependencies in complete ignorance of other packages. Some Haskell devs have moved to nix to avoid so called “cabal hell”. Cabal sandboxes, especially the newer version, work fine for many users, but nix also covers code outside of Haskell and makes it easier, in my opinion at least, to support multiple versions of ghc on your dev box.

I have never gotten the hang of running multiple ghc versions on my ubuntu dev box and have never seen a solution that looked easy to manage. Frankly, I am a coward, since I use my dev box to produce production code, and I don’t want to break anything besides my own code. Of course breaking things is probably the best way to learn. Nix promised to make it easy for the cowardly and fearless me to work on the same box. That’s a very enticing claim to test.

First Some Tips

So far the learning curve for nix has been steep, though maybe less steep than learning Haskell. Before doing some more serious hacking, here are some suggestions that a “normal” Haskell developer like me might wish to consider to make the learning curve that much easier.

Do check out some nix resources.

The documentation for nix is fairly complete, but sometimes seems written by experts, for experts. Fortunately there are a few good introductory blog pieces around, many of them written by devs who appear to be heavily involved with the nix project. Here are a few in no special order that I found helpful:

Special mention goes to a couple of blog posts by Ollie Charles, which convinced me to finally invest some serious time in nix.

It is also worth highlighting a few of the most helpful spots in the offical docs.

Do refer to nix packages by their attribute path rather than their name whenever possible.

The installation command nix-env -i lets you refer to packages either through their “attribute path” or their package name. The latter is very slow due to the way the query tool currently works.

As a result, for me

nix-env -iA nixpkgs.haskellPackages.SomeHaskellPkg

is usually dramatically faster than

nix-env -i haskell-some-haskell-pkg

Where to find these attribute paths? You can do nix-env -qaP to get the attribute paths. For a while I was saving the output to a file and searching for packages there until I learned the next trick.

Do install nix-repl.

It is as easy as

nix-env -iA nixpkgs.nix-repl

Having the repl available does a number of things for you. First, since nix uses the nix expression language for all its build recipes, nix-repl is like ghci for that language. You can play around with simple expressions to get used to the language. That will come in handy later.

nix-repl is also handy for inspecting available packages. Try this…

$ nix-repl nix-repl> nixpkgs = import <nixpkgs> nix-repl> :a builtins nix-repl> attrNames nixpkgs builtins.attrNames pkgs [ "AgdaSheaves" "AgdaStdlib" "DisnixWebService" "LASzip" "MPlayerPlugin" "OVMF" "PPSSPP" "PatolineEnv" "R" "SDL" "SDL2" "SDL2_gfx" "SDL2_image" "SDL2_mixer" "SDL2_net" "SDL_gfx" "SDL_image" "SDL_mixer" "SDL_net" "SDL_sound" "SDL_ttf""TotalParserCombinators" "VisualBoyAdvance" "Xaw3d" "ZopeInterface" "_915resolution" "a2jmidid" "a52dec" "aacskeys" "aalib" "aangifte2006" "aangifte2007" "aangifte2008" "aangifte2009" "aangifte2010" "aangifte2011" "aangifte2012" "aangifte2013" "abc" "abc-verifier" "abcPatchable" "abcde" "abduco" "abiword" ... ]

A total of 4430 packages appear in the output on my system. Let’s hunt for versions of ghc that are available.

nix-repl> attrNames nixpkgs.haskellP<TAB> nix-repl> builtins.attrNames nixpkgs.haskell nixpkgs.haskell nixpkgs.haskellPackages nixpkgs.haskellPackages_ghc6104 nixpkgs.haskellPackages_ghc6123 nixpkgs.haskellPackages_ghc704 nixpkgs.haskellPackages_ghc722 nixpkgs.haskellPackages_ghc742 nixpkgs.haskellPackages_ghc763 nixpkgs.haskellPackages_ghc783 nixpkgs.haskellPackages_ghc783_no_profiling nixpkgs.haskellPackages_ghc783_profiling nixpkgs.haskellPackages_ghcHEAD nix-repl> exit

This shows all the versions of ghc available including the latest bleeding edge version. The default is currrently 7.63 and ghcHEAD corresponds to an unreleased version of ghc. Many of these also have binary cached versions available, which install much faster and are exactly what you would get if you built them yourself.

You can also hunt specifically for Haskell packages. Let’s see what’s available for ghc 7.6.3.

$ nix-repl nix-repl> nixpkgs.haskellPackages_ghc763.<TAB> nixpkgs.haskellPackages_ghc763.ACVector nixpkgs.haskellPackages_ghc763.AES nixpkgs.haskellPackages_ghc763.Agda nixpkgs.haskellPackages_ghc763.Allure nixpkgs.haskellPackages_ghc763.BNFC nixpkgs.haskellPackages_ghc763.BlogLiterately nixpkgs.haskellPackages_ghc763.Boolean nixpkgs.haskellPackages_ghc763.BoundedChan nixpkgs.haskellPackages_ghc763.CCdelcont nixpkgs.haskellPackages_ghc763.Cabal nixpkgs.haskellPackages_ghc763.Cabal_1_16_0_3 nixpkgs.haskellPackages_ghc763.Cabal_1_18_1_3 nixpkgs.haskellPackages_ghc763.Cabal_1_20_0_2 ... nixpkgs.haskellPackages_ghc763.zippers nixpkgs.haskellPackages_ghc763.zlib nixpkgs.haskellPackages_ghc763.zlibBindings nixpkgs.haskellPackages_ghc763.zlibConduit nix-repl> exit

If you are using the default version of ghc, currently 7.6.3, you can omit reference to the compiler version and install a package like zippers this way.

nix-env -iA nixpkgs.haskellPackages.zippers

Do fork nixpgs.

Your nix installation already contains a complete copy of all the nix packages under ~/.nix-defexpr/channels/nixpkgs/ . This is called a channel, which bundles the ability to build all packages from source with the potential to install prebuilt binaries. These binaries are called a substitute in nix parlance and are a pure win if they are available, since your built-from-source package would be identical.

You can also have your own complete set of nix packages to play with. Just fork clone the repo over at https://github.com/NixOS/nixpkgs.

One reason for forking nixpkgs is that you can update the repo to obtain the very latest versions. It’s perfectly ok to install packages either from your forked repo or your nix channel.

If you want to run different versions of ghc there is another reason to fork. I have a fork stored at /timsears/code/nixpkgs . I can open nix-repl again and point to the fork.

nix-repl> mynixpkgs = import /home/timsears/code/nixpkgs {} nix-repl> mynixpkgs.haskell mynixpkgs.haskell mynixpkgs.haskellPackages mynixpkgs.haskellPackages_ghc6104 mynixpkgs.haskellPackages_ghc6123 mynixpkgs.haskellPackages_ghc704 mynixpkgs.haskellPackages_ghc722 mynixpkgs.haskellPackages_ghc742 mynixpkgs.haskellPackages_ghc763 mynixpkgs.haskellPackages_ghc783 mynixpkgs.haskellPackages_ghc783_no_profiling mynixpkgs.haskellPackages_ghc783_profiling mynixpkgs.haskellPackages_ghcHEAD

Right now the list is the same as the one from my channel. But a couple of weeks ago, ghc 7.83 was only available through my forked repo.

Either way, this is cool. I would not dare to support so many versions of ghc on my ubuntu dev box.

Do poke around the nix store.

Everything to do with building packages ends up in /nix/store . When you first poke around in there it is a little bewildering. Do take a few nix pills and learn what’s what. The .drv files are what nix calls derivations. These files relate all the input and output files connected to building a package. All those inputs and outputs are isolated from each other and stored in their own subdirectory under /nix/store . Any dependencies of installed packages are there too, along with dependencies of dependencies, and so forth. You may have several versions of the same package if two downstream packages need different versions. It is probably not too surprising that all the outputs are there, but what about the inputs? Why keep them? Nix keeps references to everything needed to build and reproduce a build exactly. Having nix know the inputs essentially provides the proof that the output will be the same. Nix can even locate binary caches of those builds if available elswhere. That’s the benefit of purity. Nix starts to look very much like a memoized Haskell function.

Do understand profiles.

There at least three “locations” to install packages in nixos: the system, user-profile and per-project.

System level install.

You can add packages to /etc/configuration.nix as I have done here. This makes them available to all user profiles. There’s probably only one real user on your dev box, but this is still a useful option. For example, you may want to have multiple user profiles to segregate packages that don’t play nice, but you still don’t want to reinstall packages like emacs everywhere. If you prefer not to edit this file you can create a custom set of packages that every user profile can install. I won’t talk about this further here, but I am interested to hear other nix users’ comments on easy ways to manage sets of common packages.

User profile installs

You can also install packages in your default profile nix-env -iA nixpkgs.somePackage and that’s fine. Eventually you will need to manage groups of packages that do not play nice, such as different versions of ghc and their corresponding libraries. Let’s create two new user profiles. In so2013 we will run ghc 7.42 and in the profile letitbleed we will run ghcHEAD which corresponds to an unreleased ghc 7.9.

Nix was designed by folks unafraid of long pathnames, so here’s how you are supposed to do it.

$ nix-env --switch-profile /nix/var/nix/profiles/per-user/timsears/so2013 $ nix-env -q --confirm it's an empty profile $ nix-env -iA nixpkgs.haskellPackages_ghc742.ghc

Run ghc --version to confirm that it worked.

Let’s add an old favorite, hmatrix, to the profile. It’s known to work with ghc 7.4.

$ nix-env -iA nixpkgs.haskellPackages.hmatrix

You should now be able to see something like this.

$ nix-env -q ghc-7.4.2-wrapper haskell-hmatrix-ghc7.4.2-0.16.0.6

We can now repeat the process to set up our letitbleed profile.

$ nix-env --switch-profile /nix/var/nix/profiles/per-user/timsears/letitbleed $ nix-env -iA nixpkgs.haskellPackages_ghcHEAD.ghc $ nix-env -iA nixpkgs.haskellPackages_ghcHEAD.hmatrix

The system responds with

installing `haskell-hmatrix-ghc7.9.20141106-0.16.0.6-shared' ... output omitted ... Creating package registration file: /nix/store/d0wccnkgjgxikh1l4hs9c3s4jnyqfzgq-haskell-hscolour-ghc7.9.20141106-1.20.3-shared/lib/ghc-7.9.20141106/package.conf.d/hscolour-1.20.3.conf ghc-pkg: ghc no longer supports single-file style package databases (/nix/store/d0wccnkgjgxikh1l4hs9c3s4jnyqfzgq-haskell-hscolour-ghc7.9.20141106-1.20.3-shared/lib/ghc-7.9.20141106/package.conf.d/hscolour-1.20.3.installedconf) use 'ghc-pkg init' to create the database with the correct format. [qbuilder for `/nix/store/vvf2ml647ms6c07885i190ay8b5gznhb-haskell-hscolour-ghc7.9.20141106-1.20.3-shared.drv' failed with exit code 1 cannot build derivation `/nix/store/0b0dgss2dpl0vnfqmvdnbr66n4vzk3g7-haskell-hmatrix-ghc7.9.20141106-0.16.0.6-shared.drv': 1 dependencies couldn't be built error: build of `/nix/store/0b0dgss2dpl0vnfqmvdnbr66n4vzk3g7-haskell-hmatrix-ghc7.9.20141106-0.16.0.6-shared.drv' failed

Boom! Looks like the dependency hscolour does not install, confirming we have indeed arrived on the bleeding edge. We’ll leave that for another day.

Project level installs

When building your own code with nix you can hide your system and user profiles to build code in a more or less clean environment using nix-build and nix-shell . You don’t really install packages, you declare dependencies. I will get into more detail in a later post.

Next Time