Bash script to automatically test program output - C

If I get what you want; this might get you started:

A mix of bash + external tools like diff.

#!/bin/bash

# If number of arguments less then 1; print usage and exit
if [ $# -lt 1 ]; then
    printf "Usage: %s <application>\n" "$0" >&2
    exit 1
fi

bin="$1"           # The application (from command arg)
diff="diff -iad"   # Diff command, or what ever

# An array, do not have to declare it, but is supposedly faster
declare -a file_base=("file1" "file2" "file3")

# Loop the array
for file in "${file_base[@]}"; do
    # Padd file_base with suffixes
    file_in="$file.in"             # The in file
    file_out_val="$file.out"       # The out file to check against
    file_out_tst="$file.out.tst"   # The outfile from test application

    # Validate infile exists (do the same for out validate file)
    if [ ! -f "$file_in" ]; then
        printf "In file %s is missing\n" "$file_in"
        continue;
    fi
    if [ ! -f "$file_out_val" ]; then
        printf "Validation file %s is missing\n" "$file_out_val"
        continue;
    fi

    printf "Testing against %s\n" "$file_in"

    # Run application, redirect in file to app, and output to out file
    "./$bin" < "$file_in" > "$file_out_tst"

    # Execute diff
    $diff "$file_out_tst" "$file_out_val"


    # Check exit code from previous command (ie diff)
    # We need to add this to a variable else we can't print it
    # as it will be changed by the if [
    # Iff not 0 then the files differ (at least with diff)
    e_code=$?
    if [ $e_code != 0 ]; then
            printf "TEST FAIL : %d\n" "$e_code"
    else
            printf "TEST OK!\n"
    fi

    # Pause by prompt
    read -p "Enter a to abort, anything else to continue: " input_data
    # Iff input is "a" then abort
    [ "$input_data" == "a" ] && break

done

# Clean exit with status 0
exit 0

Edit.

Added exit code check; And a short walk trough:

This will in short do:

1. Check if argument is given (bin/application)
2. Use an array of "base names", loop this and generate real filenames.
   • I.e.: Having array ("file1" "file2") you get
     
     • In file: file1.in
     • Out file to validate against: file1.out
     • Out file: file1.out.tst
     • In file: file2.in
     • ...
3. Execute application and redirect in file to stdin for application by <, and redirect stdout from application to out file test by >.
4. Use a tool like i.e. diff to test if they are the same.
5. Check exit / return code from tool and print message (FAIL/OK)
6. Prompt for continuance.

Any and all of which off course can be modified, removed etc.

Some links:

• TLDP; Advanced Bash-Scripting Guide (can be a bit more readable with this)
  • Arrays
  • File test operators
  • Loops and branches
  • Exit-status
  • ...
• bash-array-tutorial
• TLDP; Bash-Beginners-Guide

Expect could be a perfect fit for this kind of problem:

Expect is a tool primarily for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial. Expect is also useful for testing these same applications.

First take a look at the Advanced Bash-Scripting Guide chapter on I/O redirection.

Then I have to ask Why use a bash script at all? Do it directly from your makefile.

For instance I have a generic makefile containing something like:

# type 'make test' to run a test.
# for example this runs your program with jackjill.txt as input
# and redirects the stdout to the file jackjill.out
test: $(program_NAME)
        ./$(program_NAME) < jackjill.txt > jackjill.out
        ./diff -q jackjill.out jackjill.expected

You can add as many tests as you want like this. You just diff the output file each time against a file containing your expected output.

Of course this is only relevant if you're actually using a makefile for building your program. :-)

Functions. Herestrings. Redirection. Process substitution. diff -q. test.

Expected outputs are a second kind of input.

For example, if you want to test a square function, you would have input like (0, 1, 2, -1, -2) and expected output as (0, 1, 4, 1, 4).

Then you would compare every result of input to the expected output and report errors for example.

You could work with arrays:

in=(0 1 2 -1 -2)
out=(0 1 4 2 4)

for i in $(seq 0 $((${#in[@]}-1)) )
do
   (( ${in[i]} * ${in[i]} - ${out[i]} )) && echo -n bad" " || echo -n fine" "
   echo $i ": " ${in[i]}"² ?= " ${out[i]}
done

fine 0 :  0² ?=  0
fine 1 :  1² ?=  1
fine 2 :  2² ?=  4
bad 3 :  -1² ?=  2
fine 4 :  -2² ?=  4

Of course you can read both arrays from a file.

Testing with (( ... )) can invoke arithmetic expressions, strings and files. Try

 help test 

for an overview.

Reading strings wordwise from a file:

for n in $(< f1); do echo $n "-" ; done

Read into an array:

arr=($(< file1)) 

Read file linewise:

for i in $(seq 1 $(cat file1 | wc -l ))
do
    line=$(sed -n ${i}p file1)
    echo $line"#"
done 

Testing against program output sounds like string comparison and capturing of program output n=$(cmd param1 param2):

asux:~/prompt > echo -e "foo\nbar\nbaz" 
foo
bar
baz
asux:~/prompt > echo -e "foo\nbar\nbaz" > file 
asux:~/prompt > for i in $(seq 1 3); do line=$(sed -n ${i}p file); test "$line" = "bar" && echo match || echo fail ; done 
fail
match
fail

Further usesful: Regular expression matching on Strings with =~ in [[ ... ]] brackets:

for i in $(seq 1 3)
do
   line=$(sed -n ${i}p file)
   echo -n $line
   if [[ "$line" =~ ba. ]]; then
     echo " "match
   else echo " "fail
   fi
done 
foo fail
bar match
baz match