Sparrow6 is a Raku automation framework with embedded tool that allows to create blackbox testing scenarios.

The feature is called Task Checks DSL.

Blackbox testing is based on the idea of agnosticism of internal structure of an object being tested. Sparrow6 takes a very simple approach allows one to create regexp based rules to analyze external scripts output.

Following are some examples.

Plain string search

Just create a task check file that includes a searched string

task.bash



!bash echo "Hello World"

task.check



Hello

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:



20:01:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/plain-string] Hello World [task check] stdout match <Hello> True

Regexp search

Sparrow6 allows to use Raku regular expressions to search in script output:

task.check



regexp: ^^ Hello \s+ World

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:



20:03:20 12/23/2019 [/home/melezhik/projects/blackbox-testing/regexp] Hello World [task check] stdout match <^^ Hello \s+ World> True

Sequential search

Sometime one need to test if a sequence of lines appears in a script output. begin: , end: notation is used to check sequences:

task.bash



!bash echo "ON" echo 1 echo 2 echo 3 echo "OFF"

task.check



begin: ON 1 2 3 OFF end

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:



20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] ON 20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 1 20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 2 20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 3 20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] OFF [task check] stdout match (s) <ON> True [task check] stdout match (s) <1> True [task check] stdout match (s) <2> True [task check] stdout match (s) <3> True [task check] stdout match (s) <OFF> True

If switch first two digits, the test will fail:

task.bash



!bash echo "ON" echo 2 echo 1 echo 3 echo "OFF"

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:



20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] ON 20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 2 20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 1 20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 3 20:37:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] OFF [task check] stdout match (s) <ON> True [task check] stdout match (s) <1> False [task check] stdout match (s) <2> False [task check] stdout match (s) <3> False [task check] stdout match (s) <OFF> False ================= TASK CHECK FAIL

Range search

If one doesn't care about lines order and need to check

that lines are included between certain lines, it's possible to reshape the previous example using range notation: between {'RexExp1'} {'RegExp2'} , where RexExp1 sets a regexp pattern for the very first string in the range, and the second one does the same for the very last string in the range, all other strings should appear within the range in no particular order.

task.check



between: {'ON'} {'OFF'} 1 2 3 end:

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:



20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] ON 20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 2 20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 1 20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 3 20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] OFF [task check] stdout match (r) <1> True [task check] stdout match (r) <2> True [task check] stdout match (r) <3> True

Dynamic search

Sparrow6 equips one with an ability to generate search criteria in run time, using a variety of programming languages ( Bash,Perl,Python,Ruby,Raku,Powershell)

Let's reshape the last example using generator: notation:

task.check



generator: <<CODE !perl6 say "between: \{'ON'\} \{'OFF'\}

",(1 ... 3).join("

"), "

end:"; CODE

Changing shebang we choose different language to implement dynamic search.

Let's do implementation on Bash.

task.check



generator: <<CODE !bash echo "between: {'ON'} {'OFF'}" for i in {1..3}; do echo $i; done echo 'end:' CODE

Captures

Captures allow to extract certain chunks from searched strings and process them.

Let's count a total sum for all numbers included between "ON" and "OFF" strings:

task.check



between: {'ON'} {'OFF'} regexp: (\d+) end: code: <<CODE !perl6 my $total; for captures()<> -> $c { $total+=$c[0] } update_state(%( cnt => $total )) CODE

Test run:

perl6 -MSparrow6::DSL -e 'my %state = task-run; say %state.perl'

Output:



23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] ON 23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 2 23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 1 23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 3 23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] OFF [task check] stdout match (r) <(\d+)> True {:cnt(6)}

Asserts

Asserts allow to set predicates - statements that return true of false and create related checks.

In previous example let's check if total sum of numbers is equal 6 :

task.check



between: {'ON'} {'OFF'} regexp: (\d+) end: generator: <<CODE !perl6 my $total; for captures()<> -> $c { $total+=$c[0] } say "assert: { $total == 6 }", " total sum is 6"; CODE

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:



19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] ON 19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 1 19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 2 19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 3 19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] OFF [task check] stdout match (r) <(\d+)> True [task check] <total sum is 6> True

Passing parameters

One can pass parameters handled in dynamic search constructions.

task.check



between: {'ON'} {'OFF'} regexp: (\d+) end: generator: <<CODE !perl6 my $amount = config()<amount>; my $total; for captures()<> -> $c { $total+=$c[0] } say "assert: { $total == $amount }", " total sum is amount"; CODE

Output:



perl6 -MSparrow6::DSL -e 'task-run("{$*CWD}",%( amount => 10 ))' 19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] ON 19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 1 19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 2 19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 3 19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] OFF [task check] stdout match (r) <(\d+)> True [task check] <total sum is 10> False ================= TASK CHECK FAIL

Streams

The last interesting feature to be mentioned here is called streams.

Steams allow one to iterate over blocks of data found during sequential or range search.

In the last example consider the case when we have more then one block of numbers between 'ON' and 'OFF' delimiters.

task.bash



echo "ON" echo 1 echo 2 echo 3 echo "OFF" echo "ON" echo 3 echo 3 echo 3 echo "OFF" echo "ON" echo 2 echo 2 echo 2 echo "OFF"

Let's write an iterator for all the blocks found:



between: {'ON'} {'OFF'} regexp: (\d+) end: code: <<CODE !perl6 my $stream-num = 1; for streams_array()<> -> $s { for $s<> -> $i { say "stream#{$stream-num}: {$i[0]}"; } $stream-num++; } CODE

Test run:

perl6 -MSparrow6::DSL -e 'task-run

Output:



19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 1 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2 19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF [task check] stdout match (r) <(\d+)> True [task check] stream#1: 1 [task check] stream#1: 2 [task check] stream#1: 3 [task check] stream#2: 3 [task check] stream#2: 3 [task check] stream#2: 3 [task check] stream#3: 2 [task check] stream#3: 2 [task check] stream#3: 2

Conclusion

Task Checks DSL is a simple, yet powerful tool allow one to write blackbox tests. The feature is a part of Sparrow6 core and provided out of the box.

Thank you for reading and Merry Christmas!