Specsheet

Testing a Command-Line Program

Simple command-line programs can be tested using Specsheet, to make sure that they exit with the right status code, and produce the right output, under different combinations of command-line arguments and environment variables.

Testing command output

In this example, we’ll test the expr Unix utility, which evaluates mathematical expressions given on the command-line. Even though this program already exists, we can still write tests for it.

Two common things to test for are showing that the program succeeds when given valid input, and fails with the correct error when given invalid input. For expr, we shall write these two tests:

  1. If the user passes in a valid expression, such as 2 + 2, it should output the correct answer, 4, and return zero to indicate “Success”.
  2. If the uses passes in an invalid expression, such as 2 +, it should output nothing, print syntax error to standard error, and return two to indicate this fact.

These two pieces of behaviour can be written as these two checks in TOML:

[[cmd]]
shell = "expr 2 + 2"
status = 0
stdout = { string = "4" }
stderr = { empty = true }

[[cmd]]
shell = "expr 2 +"
status = 2
stdout = { empty = true }
stderr = { string = "syntax error" }

Once we’ve saved these two checks to a file, expr.toml, we can run them by passing the filename in on the command-line:

$ specsheet expr.toml

   expr.toml:
  Command ‘expr 2 + 2’ returns ‘0’ with stdout containing ‘4’ and empty stderr
  Command ‘expr 2 + ’ returns ‘2’ with empty stdout and stderr containing ‘syntax error’
   2/2 successful

Both checks passed. We can run this with the -s or --successes command-line option, showing the eight individual assertions that succeeded:

$ specsheet expr.toml --successes=expand

   expr.toml:
  Command ‘expr 2 + 2’ returns ‘0’ with stdout containing ‘4’ and empty stderr
    command was executed
    status code matches
    stdout matches string
    stderr is empty
  Command ‘expr 2 + ’ returns ‘2’ with empty stdout and stderr containing ‘syntax error’
    command was executed
    status code matches
    stdout is empty
    stderr matches string
   2/2 successful

Overriding the command

The example above tests whichever expr binary is found in the $PATH as Specsheet is being run. During development, or when running as part of a test pipeline, we will want to test not the installed version, but the development version.

The way to do this is to keep the input TOML file the same, but to use rewrites to change which binary is referred to by expr.

$ specsheet expr.toml -O cmd.target=./my-expr