Unit testing Bash scripts

There is actually a shunit2, an xUnit based unit test framework for Bourne based shell scripts. I haven't used it myself, but it might be worth checking out.

Similar questions have been asked before:

• Unit Testing for Shell Scripts
• Test Anything Protocol in Shell Scripts

TAP-compliant Bash testing: Bash Automated Testing System

TAP, the Test Anything Protocol, is a simple text-based interface between testing modules in a test harness. TAP started life as part of the test harness for Perl but now has implementations in C, C++, Python, PHP, Perl, Java, JavaScript, and others.

bats-core

I created shellspec, because I wanted a easy-to-use and useful tool.

It written by a pure POSIX shell script. It has been tested with many shells more than shunit2. It has more powerful features than bats/bats-core.

For example, it supports nested block, easy to mock/stub, easy to skip/pending, parameterized tests, assertion line number, execute by line number, parallel execution, random execution, TAP/JUnit formatter, coverage and CI integration, profiler, etc.

See the demo on the project page.

• https://github.com/shellspec/shellspec
• https://shellspec.info/

Nikita Sobolev wrote an excellent blog post comparing a few different Bash test frameworks, Testing Bash applications. These are the frameworks listed in that post:

• assert.sh
• shunit2
• roundup
• Bats -> Bats-core

For the impatient: Nikita's conclusion was to use Bats, but it appears that Nikita missed the Bats-core project which appear to me to be the one to use going forward as the original Bats project has not been actively maintained since 2013.

I can't believe no one talked about OSHT! It's compatible with both TAP and JUnit, it's pure shell (that is, no other languages involved), it works standalone too, and it's simple and direct.

Testing looks like this (snippets taken from the project page):

#!/bin/bash
. osht.sh

# Optionally, indicate number of tests to safeguard against abnormal exits
PLAN 13

# Comparing stuff
IS $(whoami) != root
var="foobar"
IS "$var" =~ foo
ISNT "$var" == foo

# test(1)-based tests
OK -f /etc/passwd
NOK -w /etc/passwd

# Running stuff
# Check exit code
RUNS true
NRUNS false

# Check stdio/stdout/stderr
RUNS echo -e 'foo\nbar\nbaz'
GREP bar
OGREP bar
NEGREP . # verify empty

# diff output
DIFF <<EOF
foo
bar
baz
EOF

# TODO and SKIP
TODO RUNS false
SKIP test $(uname -s) == Darwin

A simple run:

$ bash test.sh
1..13
ok 1 - IS $(whoami) != root
ok 2 - IS "$var" =~ foo
ok 3 - ISNT "$var" == foo
ok 4 - OK -f /etc/passwd
ok 5 - NOK -w /etc/passwd
ok 6 - RUNS true
ok 7 - NRUNS false
ok 8 - RUNS echo -e 'foo\nbar\nbaz'
ok 9 - GREP bar
ok 10 - OGREP bar
ok 11 - NEGREP . # verify empty
ok 12 - DIFF <<EOF
not ok 13 - TODO RUNS false # TODO Test Know to fail

The last test shows as "not ok", but the exit code is 0 because it's a TODO. One can set verbose as well:

$ OSHT_VERBOSE=1 bash test.sh # Or -v
1..13
# dcsobral \!= root
ok 1 - IS $(whoami) != root
# foobar =\~ foo
ok 2 - IS "$var" =~ foo
# \! foobar == foo
ok 3 - ISNT "$var" == foo
# test -f /etc/passwd
ok 4 - OK -f /etc/passwd
# test \! -w /etc/passwd
ok 5 - NOK -w /etc/passwd
# RUNNING: true
# STATUS: 0
# STDIO <<EOM
# EOM
ok 6 - RUNS true
# RUNNING: false
# STATUS: 1
# STDIO <<EOM
# EOM
ok 7 - NRUNS false
# RUNNING: echo -e foo\\nbar\\nbaz
# STATUS: 0
# STDIO <<EOM
# foo
# bar
# baz
# EOM
ok 8 - RUNS echo -e 'foo\nbar\nbaz'
# grep -q bar
ok 9 - GREP bar
# grep -q bar
ok 10 - OGREP bar
# \! grep -q .
ok 11 - NEGREP . # verify empty
ok 12 - DIFF <<EOF
# RUNNING: false
# STATUS: 1
# STDIO <<EOM
# EOM
not ok 13 - TODO RUNS false # TODO Test Know to fail

Rename it to use a .t extension and put it in a t subdirectory, and you can use prove(1) (part of Perl) to run it:

$ prove
t/test.t .. ok
All tests successful.
Files=1, Tests=13,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.11 cusr  0.16 csys =  0.31 CPU)
Result: PASS

Set OSHT_JUNIT or pass -j to produce JUnit output. JUnit can also be combined with prove(1).

I have used this library both testing functions by sourcing their files and then running assertions with IS/OK and their negatives, and scripts by using RUN/NRUN. For me, this framework provides the most gain for the least overhead.

Epoxy is a Bash test framework I designed mainly for testing other software, but I use it to test Bash modules as well, including itself and Carton.

The main advantages are relatively low coding overhead, unlimited assertion nesting and flexible selection of assertions to verify.

I made a presentation comparing it to BeakerLib - a framework used by some at Red Hat.

Why do you say that it's "hard" to test Bash scripts?

What's wrong with test wrappers like the following?

 #!/bin/bash
 set -e
 errors=0
 results=$($script_under_test $args<<ENDTSTDATA
 # inputs
 # go
 # here
 #
 ENDTSTDATA
 )
 [ "$?" -ne 0 ] || {
     echo "Test returned error code $?" 2>&1
     let errors+=1
     }

 echo "$results" | grep -q $expected1 || {
      echo "Test Failed.  Expected $expected1"
      let errors+=1
 }
 # And so on, et cetera, ad infinitum, ad nauseum
 [ "$errors" -gt 0 ] && {
      echo "There were $errors errors found"
      exit 1
 }

Give assert.sh a try:

source "./assert.sh"

local expected actual
expected="Hello"
actual="World!"
assert_eq "$expected" "$actual" "not equivalent!"
# => x Hello == World :: not equivalent!

I can highly recommend https://bashunit.typeddevs.com/

It is a pretty fresh and simple testing library for bash scripts.

• Free and open source license
• Easy to install
• Well documented
• Active community and engaged ownership

I quite like shell2junit, a utility to generate JUnit-like output from Bash script tests. This is useful because the report generated can then be read by continuous integration systems, such as the JUnit plug-ins for Jenkins and Bamboo.

While shell2junit doesn't provide the comprehensive Bash scripting framework like shunit2, it does allow you have nice reporting of the test results.

Try bashtest. It’s simple way to test your scripts. For example, you have do-some-work.sh which changes some configuration files. For example, add a new line, PASSWORD = 'XXXXX', to configuration file /etc/my.cfg.

You write Bash commands line by line and then check output.

Install:

pip3 install bashtest

Create tests is a just writing bash commands.

File test-do-some-work.bashtest:

# Run the script
$ ./do-some-work.sh > /dev/null

# Testing that the line "PASSWORD = 'XXXXX'" is in the file /etc/my.cfg
$ grep -Fxq "PASSWORD = 'XXXXX'" /etc/my.cfg && echo "YES"
YES

Run tests:

bashtest *.bashtest

You can find some examples here and here.

Maybe this can be used, or contributed to:

https://thorsteinssonh.github.io/bash_test_tools/

It is intended to write results in the TAP protocol which I imagine is good for CI and is good for those that want shell environments. I imagine some things run in shell environments, so some might argue it should be tested in their shell environment.

I’ve tried a lot of the solutions presented here, but I found most of them too bulky and hard to use, so I built my own little testing framework: https://github.com/SnacOverflow/t-bash

It’s just one file in the repository that you can simply run directly, with a basic set of JUnit style asserts.

I’ve used it professionally in several internal projects and were able to make our Bash scripts super stable and regression resistant.

Create a test mytest.sh. It calls your script with specific inputs.

Create a text file expected/mytest.stdout. It contains the expected output for the test.

Commit them to version control (if any).

Run the test and redirect the output to another file:

mytest.sh > actual/mytest.stdout

Then just us a text diff tool to see where the results deviate. I think you can just do

diff expected/mytest.stdout actual/mytest.stdout

Take a look at Outthentic. It is simple, extensible by many languages (Perl, Python, Ruby, and Bash on choice) and cross platform (Linux and Windows) framework to test any command line applications.