Writing tests is an essential part of software development. There are a plenty of frameworks to do that. In this post I'd like to introduce Outthentic - an universal script engine with embedded testing facilities, allow one to create BDD/Integration tests for command line applications easy.

Install

We will need the the cutting edge of Outthentic to see some recent features created especially for test development.



$ cpanm https://github.com/melezhik/outthentic.git

Application to test

Say, we have an application script - application.bash - which does all useful job.

Our goal to test this script properly:



$ nano /usr/local/bin/application.bash #!bash echo "Hello world"

Let's outline our simple testing plan:

Ensure that script exit code is 0

Ensure that script output certain string to STDOUT

Skip test for a certain environment

Abort test if script is not installed in the system

Ensure that script exit code is 0

This exit code is checked by default when external script gets through Outthentic, we just need to add a thin layer to enable this test:



$ mkdir check-exit-code $ nano check-exit-code/story.bash #!/bash application.bash

Now let's run out first test and see the output:



$ strun --story check-exit-code 2018-09-26 16:35:47 : [path] /check-exit-code/ /root/projects/outthentic-dev.to//check-exit-code/story.bash: line 1: /usr/local/bin/application.bash: Permission denied not ok scenario succeeded STATUS FAILED (2)

Well, the test has failed. And we see the reason. We forgot to set execution bit, it's easy to fix and re-run test again. This is what we have write our tests for - to see things that do not work, rather then see things that are fine.



$ chmod a+x /usr/local/bin/application.bash 2018-09-26 20:07:27 : [path] /check-exit-code/ Hello world ok scenario succeeded STATUS SUCCEED

A few words about status code emitted by Outthentic. As you might have noticed the first failed test produces status code 2 , which generally means test failures, there are 3 exit codes provided by Outthentic:

0 tests are passed, everything is ok

tests are passed, everything is ok 2 tests are failed, something definitely went wrong

tests are failed, something definitely went wrong 1 some tests are passed, some are not, there is something wrong or there are some warnings

The last case makes it possible to use Outthentic tests as Consul check scripts

Ensure that script outputs some string to STDOUT

Before going into details, let's think why we would need it?

A few reasons ( I wonder if a reader would provide more ) :

Some external scripts do not provide a sane exit code, the only reasonable thing we can test is an output

None zero exit code might not means that program works unexpectedly. Again we can check output rather then exit code

Program emits zero exit code ( finishes successfully ) but makes different output messages when being called with different parameters. So to check various test cases we need to run the same program differently and check different output

As our application is just an example, the check for "Hello world" string is trivial:



$ nano check-exit-code/story.check Hello World

Now run:



$ strun --story check-exit-code 2018-09-26 17:23:29 : [path] /check-exit-code/ Hello world ok scenario succeeded ok text has 'Hello world' STATUS SUCCEED

There are more things you can check with Outthentic. Imagine we want to check the program's output consists of sequential numbers from 1 to 10 :



$ nano check-exit-code/story.check begin: generator: <<CODE [ map { "regexp: ^\\d+$_" } (1..10) ] CODE end:

But let's go further.

Splitting tests for different cases

In real projects we can eventually have many test cases mapped to many test scenarios. Outthentic is flexible enough to adopt for such a scheme, running multiples tests as a suite.

Let's reorgonize our test structure a bit:



$ mkdir check-exit-code $ nano check-exit-code/story.bash #!/bash application.bash $ mkdir check-stdout $ nano check-exit-code/story.bash #!/bash application.bash $ nano check-stdout/story.check Hello World

Now we have two test scenarios. To check exit code and to check returned output.

These tests overlaps by the fact they both runs application.bash however for our purposes it is not critical, we just create different tests to emphasize what we what to test.



$ tree . ├── check-exit-code │ └── story.bash └── check-stdout ├── story.bash └── story.check

Finally Outthentic allows us to run all our tests recursively:



$ strun --recurse 2018-09-26 17:34:52 : [path] /check-exit-code/ Hello world ok scenario succeeded STATUS SUCCEED 2018-09-26 17:34:52 : [path] /check-stdout/ Hello world ok scenario succeeded ok text has 'Hello world' STATUS SUCCEED STATUS SUCCEED

As we have and more and more tests, we might need to narrow down the output, it is achievable by --format option of Outthentic test runner:



$ strun --recurse --format=production 2018-09-26 17:36:14 : [path] /check-exit-code/ 2018-09-26 17:36:14 : [path] /check-stdout/

Now let's go to the two last points of our testing plan.

Sometimes we need to skip our tests for some reasons or even raise exception if some preliminary conditions are not met, so we don't waste our time and run tests polluting console with unnecessary messages, as we already know that we might not run test suite.

Quit test for some environment

Say, we don't want run tests for production environment which is defined by passing environment variable:



$ export environment=production

Outthentic allows to *immediately * quit test execution phase by using quit function, let's see an example of it:



$ nano check-exit-code/hook.bash #!bash if test "$environment" = "production"; then quite "production tests are disabled, please use dev environment" fi

$ strun --story check-exit-code 2018-09-26 18:10:54 : [path] /check-exit-code/ ? forcefully exit: production tests are disabled, please use dev environment STATUS SUCCEED

The opposite idea is to let your tests fail immediately upon a certain condition. You should choose outthentic_die function for this:



$ nano check-exit-code/hook.bash #!bash which application.bash 2>/dev/null || \ outthentic_die "application.bash is not installed. You should install it to run tests"

$ unlink /usr/local/bin/application.bash $ strun --story check-exit-code 2018-09-26 18:16:49 : [path] /check-exit-code/ !! forcefully die: application.bash is not installed. You should install it to run tests STATUS FAILED (2)

Now our project structure looks like this:



. ├── check-exit-code │ ├── hook.bash │ └── story.bash └── check-stdout ├── story.bash └── story.check

Hook.bash is an example Outthentic hooks - small script gets run before main test run, you can read more about hooks on Outthentic documentation pages.

Decentralized or centralized testing model

As you could have noticed, we would have to add hook.bash to every test where we want to ensure preliminary conditions are met. In this scheme every test is treated as independent unit, and this follows the pattern we can see in many testing frameworks. While it seems easier to implement, it also results in duplication of code. Now we have hook.bash scripts doing the same job for every story.

Alternatively we might check those preliminary conditions ( like environment and application being installed ) once, in the very beginning, before we run any test.

This approach leads us to the opposite scheme, where all test become deepened on some "main" entry point where we can:

perform initialization steps ( preliminary conditions check )

call tests as functions in order

In Outthentic this type of design could be easily implemented through so called "story modules". Laterally when you call tests as a functions.

Let's slightly refactor our project to implement the idea:



$ nano hook.bash #!bash # this a main entrypoint if test "$environment" = "production"; then quite "production tests are disabled, please use dev environment" fi which application.bash 2>/dev/null || \ outthentic_die "application.bash is not installed. You should install it to run tests" run_story "check-exit-code" run_story "check-stdout" $ # we don't need hook.bash per story anymore $ rm check-exit-code/hook.bash # we make our tests - modules or functions, just when we copy those ones to modules/ folder $ mkdir modules $ mv check-exit-code check-stdout modules/

So we end up with this structure with one main entrypoint ( hook.bash ) and two dependable tests ( check-exit-code/story.bash , check-stdout/story.bash ), also notice that we run preliminary conditions check inside main entrypoint and control how and what tests to run.



. ├── hook.bash └── modules ├── check-exit-code │ └── story.bash └── check-stdout ├── story.bash └── story.check

Now we are ready to run tests, note that we don't need --recurse option anymore, because all the tests sequence is defined through the hook.bash file.



$ strun 2018-09-26 20:24:20 : [path] /modules/check-exit-code/ Hello world ok scenario succeeded 2018-09-26 20:24:20 : [path] /modules/check-stdout/ Hello world ok scenario succeeded ok text has 'Hello world' STATUS SUCCEED

The end

This was just a brief introduction into testing capabilities provided by Outthentic framework. If you like it - go to GH pages to see all the details.

Feel free to share you opinions here in comments.