btest is a minimal, language-agnostic test runner originally written for testing compilers. Brian, an ex- co-worker from Linode, wrote the first implementation in Crystal (a compiled language clone of Ruby) for testing bshift, a compiler project. The tool accomplished exactly what I needed for my own language project, BSDScheme, and had very few dependencies. After some issues with Crystal support in containerized CI environments, and despite some incredible assistance from the Crystal community, we rewrote btest in D to simplify downstream use.

How it works

btest registers a command (or commands) to run and verifies the command output and status for different inputs. btest iterates over files in a directory to discover test groups and individual tests within. It supports a limited template language for easily adjusting a more-or-less similar set of tests. And it supports running test groups and individual tests themselves in parallel. All of this is managed via a simple YAML config.

btest.yaml

btest requires a project-level configuration file to declare the test directory, the command(s) to run per test, etc. Let's say we want to run tests against a python program. We create a btest.yaml file with the following:

test_path: tests runners: - name: Run tests with cpython run: python test.py

test_path is the directory in which tests are located. runners is an array of commands to run per test. We hard-code a file to run test.py as a project-level standard file that will get written to disk in an appropriate path for each test-case.

On multiple runners

Using multiple runners is helpful when we want to run all tests with different test commands or test command settings. For instance, we could run tests against cpython and pypy by adding another runner to the runners section.

test_path: tests runners: - name: Run tests with cpython run: python test.py - name: Run tests with pypy run: pypy test.py

An example test config

Let's create a divide-by-zero.yaml file in the tests directory and add the following:

cases: - name: Should exit on divide by zero status: 1 stdout: | Traceback (most recent call last): File "test.py", line 1, in <module> 4 / 0 ZeroDivisionError: division by zero denominator: 0 templates: - test.py: | 4 / {{ denominator }}

In this example, name will be printed out when the test is run. status is the expected integer returned by running the program. stdout is the entire expected output written by the program during execution. None of these three fields are required. If status or stdout are not provided, btest will skip checking them.

Any additional key-value pairs are treated as template variable values and will be substituted if/where it is referenced in the templates section when the case is run. denominator is the only such variable we use in this example. When this first (and only) case is run, test.py will be written to disk containing 4 / 0 .

templates section

The templates section is a dictionary allowing us to specify files to be created with variable substitution. All files are created in the same directory per test case, so if we want to import code we can do so with relative paths.

Here is a simple example of a BSDScheme test that uses this feature.

Running btest

Run btest from the root directory (the directory above tests ) and we'll see all the grouped test cases that btest registers and the result of each test:

$ btest tests/divide-by-zero.yaml [PASS] Should exit on divide by zero 1 of 1 tests passed for runner: Run tests with cpython

Use in CI environments

In the future we may provide pre-built release binaries. But in the meantime, the CI step involves downloading git and ldc and building/installing btest before calling it.

Circle CI

This is the config file I use for testing BSDScheme:

version: 2 jobs: build: docker: - image: dlanguage/ldc steps: - checkout - run: name: Install debian-packaged dependencies command: | apt update apt install -y git build-essential ln -s $(which ldc2) /usr/local/bin/ldc - run: name: Install btest command: | git clone https://github.com/briansteffens/btest cd btest make make install - run: name: Install bsdscheme command: | make make install - run: name: Run bsdscheme tests command: btest

Travis CI

This is the config Brian uses for testing BShift: