shell-conduit: Write shell scripts in Haskell with Conduit

As part of my series of write-about-personal-projects, my latest obsession is writing shell scripts with Michael Snoyman’s Conduit.

Here is my package, shell-conduit. It’s still in the experimental phase, but I don’t forsee any changes now for a while.

Bash is evil

I hate writing scripts in Bash. Until now, it was the easiest way to just write unix scripts. Its syntax is insane, incredibly error prone, its defaults are awful, and it’s not a real big person programming language.

Perl/Python/Ruby are also evil

If you’re going to go as far as using a real programming language, why bother with these dynamically typed messes? Go straight for Haskell.

Like a glove

I had an inkling a while back that conduits mirror the behaviour of bash pipes very well. I knew there was something to the idea, but didn’t act on it fully for a while. Last week I experimented somewhat and realised that the following Haskell code

does indeed accurately mirror

And that also the following

is analogous to

We’ll see examples of why this works later.

I must Haskell all the things

Another trick I realised is to write some template Haskell code which will calculate all executables in your PATH at compilation time and generate a top-level name that is a Haskell function to launch that process. So instead of writing

you could instead just write

There are a few thousand executables, so it takes about 10 seconds to compile such a module of names. But that’s all.

Again, we’ll see how awesome this looks in a minute.

Modeling stdin, stderr and stdout

My choice of modeling the typical shell scripting pipe handles is by having a type called Chunk :

All Left values are from stderr . All Right values are either being pulled from stdin or being sent to stdout . In a conduit the difference between stdin and stdout is more conceptual than real.

When piping two commands, the idea is that any Left values are just re-yielded along, they are not consumed and passed into the process.

A process conduit on chunks

Putting the previous model into practice, we come up with a type for launching a process like this:

Meaning the process will be launched, and the conduit will accept any upstream stdin ( Right values), and send downstream anything that comes from the actual process (both Left and Right values).

Process conduits API

I defined two handy functions for running process conduits:

One to launch via a shell, one to launch via program name and arguments. These functions can be used in your shell scripts. Though, we’ll see in a minute why you should rarely need either.

Executing a shell scripting conduit

First we want something to consume any remainder chunks after a script has finished. That’s writeChunks :

This simply consumes anything left in the pipeline and outputs to the correct file handles, either stderr or stdout .

Now we can write a simple run function:

First it yields an empty upstream of chunks. That’s the source. Then our script p is run as the conduit in between, finally we write out any chunks that remain.

Let’s try that out:

Looks good. Standard output was written properly, as was stderr.

Returning to our mass name generation

Let’s take our earlier work of generating names with template-haskell. With that in place, we have a process conduit for every executable in PATH . Add to that variadic argument handling for each one, we get a list of names like this:

The real types when instantiated will look like:

Putting it all together

We can now provide any number of arguments:

We can pipe things together:

Results are outputted to stdout unless piped into other processes:

Live streaming between pipes like in normal shell scripting is possible:

(Remember that grep needs --line-buffered if it is to output things line-by-line).

Error handling

By default, if a process errors out, the whole script ends. This is contrary to Bash, which keeps going regardless of failure. This is bad.

In Bash, to revert this default, you run:

And the way to ignore erroneous commands on case-by-case basis is to use || true :

Which means “do foo, or otherwise ignore it, continue the script”.

We can express the same thing using the Alternative instance for the ShellT type:

String types

If using OverloadedStrings so that you can use Text for arguments, then also enable ExtendedDefaultRules , otherwise you’ll get ambiguous type errors.

But this isn’t necessary if you don’t need to use Text yet. Strings literals will be interpreted as String . Though you can pass a value of type Text or any instance of CmdArg without needing conversions.

Examples of script files

Quick script to reset my keyboard (Linux tends to forget these things when I unplug my keyboard):

Cloning and initializing a repo (ported from a bash script):

Script to restart a web process (ported from an old bash script I had):

You’ve seen Shelly, right?

Right. Shelly’s fine. It just lacks the two killer things for me:

All names are bound, so I can just use them as normal functions.

shell-conduit also, due to its mass name binding, prioritizes commands. For example, Shelly has a group of functions for manipulating the file system. In shell-conduit, you just use your normal commands: rm "x" and mv "x" "y" .

and . Not based on conduit. Conduit is a whole suite of streaming utilities perfect for scripting.

Piped is not the default, either. There’re a bunch of choices: Shelly, Shelly.Lifted, Shelly.Pipe. Choice is good, but for a scripting language I personally prefer one goto way to do something.

Also, Shelly cannot do live streams like Conduit can.

Conduits as good scripting libraries

You might want to import the regular Conduit modules qualified, too:

Which contains handy functions for working on streams in a list-like way. See the rest of the handy modules for Conduit in conduit-extra.

Also of interest is csv-conduit, html-conduit, and http-conduit.

Finally, see the Conduit category on Hackage for other useful libraries: http://hackage.haskell.org/packages/#cat:Conduit

All of these general purpose Conduits can be used in shell scripting.

Using it for real scripts

So far I have ported a few small scripts to shell-conduit from Bash and have been happy every time. I suck at Bash. I’m pretty good at Haskell.

The next test is applying this to my Hell shell and seeing if I can use it as a commandline shell, too.

My friend complained that having to quote all arguments is a pain. I don’t really agree that this is bad. In Bash it’s often unclear how arguments are going to be interpreted. I’m happy just writing something predictable than something super convenient but possibly nonsense.

Summary

I set out a week ago to just stop writing Bash scripts. I’ve written a bunch of scripts in Haskell, but I would still write Bash scripts too. Some things were just too boring to write. I wanted to commit to Haskell for scripting. Today, I’m fairly confident I have a solution that is going to be satisfactory for a long while.

© 2014-09-21 Chris Done