Testing a command-line program
Simple command-line programs can be tested using Specsheet, to make sure that they exit with the right status code, and produce the right output, under different combinations of command-line arguments and environment variables.
Testing command output
In this example, we’ll test the expr
Unix utility, which evaluates mathematical expressions given on the command-line.
Even though this program already exists, we can still write tests for it.
Two common things to test for are showing that the program succeeds when given valid input, and fails with the correct error when given invalid input.
For expr
, we shall write these two tests:
- If the user passes in a valid expression, such as
2 + 2
, it should output the correct answer,4
, and return zero to indicate “Success”. - If the uses passes in an invalid expression, such as
2 +
, it should output nothing, print“ to standard error, and return two to indicate this fact.syntax error
”
These two pieces of behaviour can be written as these two checks in TOML:
[[cmd]] shell = "expr 2 + 2" status = 0 stdout = { string = "4" } stderr = { empty = true } [[cmd]] shell = "expr 2 +" status = 2 stdout = { empty = true } stderr = { string = "syntax error" }
Once we’ve saved these two checks to a file, expr.toml
, we can run them by passing the filename in on the command-line:
$ specsheet expr.toml expr.toml: ✔ Command ‘expr 2 + 2’ returns ‘0’ with stdout containing ‘4’ and empty stderr ✔ Command ‘expr 2 + ’ returns ‘2’ with empty stdout and stderr containing ‘syntax error’ 2/2 successful
Both checks passed.
We can run this with the -s
or --successes
command-line option, showing the eight individual assertions that succeeded:
$ specsheet expr.toml --successes=expand expr.toml: ✔ Command ‘expr 2 + 2’ returns ‘0’ with stdout containing ‘4’ and empty stderr ✔ command was executed ✔ status code matches ✔ stdout matches string ✔ stderr is empty ✔ Command ‘expr 2 + ’ returns ‘2’ with empty stdout and stderr containing ‘syntax error’ ✔ command was executed ✔ status code matches ✔ stdout is empty ✔ stderr matches string 2/2 successful
Overriding the command
The example above tests whichever expr
binary is found in the $PATH
as Specsheet is being run.
During development, or when running as part of a test pipeline, we will want to test not the installed version, but the development version.
The way to do this is to keep the input TOML file the same, but to use rewrites to change which binary is referred to by expr
.
$ specsheet expr.toml -O cmd.target=./my-expr