91 lines
3.4 KiB
Markdown
91 lines
3.4 KiB
Markdown
|
# cli-test
|
||
|
|
|
||
|
|
## What?
|
||
|
|
|
||
|
|
`cli-test` makes integration testing of command-line tools easier.
|
||
|
|
|
||
|
|
## Goals
|
||
|
|
|
||
|
|
* Readable syntax. Common cases should be concise, and pretty much anyone
|
||
|
|
should be able to read tests even if they've never seen this tool before.
|
||
|
|
|
||
|
|
* Minimal issues with quoting. The toybox tests -- being shell scripts --
|
||
|
|
quickly become a nightmare of quoting. Using a non ad hoc format (such as
|
||
|
|
JSON) would have introduced similar but different quoting issues. A custom
|
||
|
|
format, while annoying, side-steps this.
|
||
|
|
|
||
|
|
* Sensible defaults. We expect your exit status to be 0 unless you say
|
||
|
|
otherwise. We expect nothing on stderr unless you say otherwise. And so on.
|
||
|
|
|
||
|
|
* Convention over configuration. Related to sensible defaults, we don't let you
|
||
|
|
configure things that aren't absolutely necessary. So you can't keep your test
|
||
|
|
data anywhere except in the `files/` subdirectory of the directory containing
|
||
|
|
your test, for example.
|
||
|
|
|
||
|
|
## Non Goals
|
||
|
|
|
||
|
|
* Portability. Just being able to run on Linux (host and device) is sufficient
|
||
|
|
for our needs. macOS is probably easy enough if we ever need it, but Windows
|
||
|
|
probably doesn't make sense.
|
||
|
|
|
||
|
|
## Syntax
|
||
|
|
|
||
|
|
Any all-whitespace line, or line starting with `#` is ignored.
|
||
|
|
|
||
|
|
A test looks like this:
|
||
|
|
```
|
||
|
|
name: unzip -l
|
||
|
|
command: unzip -l $FILES/example.zip d1/d2/x.txt
|
||
|
|
after: [ ! -f d1/d2/x.txt ]
|
||
|
|
expected-stdout:
|
||
|
|
Archive: $FILES/example.zip
|
||
|
|
Length Date Time Name
|
||
|
|
--------- ---------- ----- ----
|
||
|
|
1024 2017-06-04 08:45 d1/d2/x.txt
|
||
|
|
--------- -------
|
||
|
|
1024 1 file
|
||
|
|
---
|
||
|
|
```
|
||
|
|
|
||
|
|
The `name:` line names the test, and is only for human consumption.
|
||
|
|
|
||
|
|
The `command:` line is the command to be run. Additional commands can be
|
||
|
|
supplied as zero or more `before:` lines (run before `command:`) and zero or
|
||
|
|
more `after:` lines (run after `command:`). These are useful for both
|
||
|
|
setup/teardown but also for testing post conditions (as in the example above).
|
||
|
|
|
||
|
|
Any `command:`, `before:`, or `after:` line is expected to exit with status 0.
|
||
|
|
Anything else is considered a test failure.
|
||
|
|
|
||
|
|
The `expected-stdout:` line is followed by zero or more tab-prefixed lines that
|
||
|
|
are otherwise the exact output expected from the command. (There's magic behind
|
||
|
|
the scenes to rewrite the test files directory to `$FILES` because otherwise any
|
||
|
|
path in the output would depend on the temporary directory used to run the test.)
|
||
|
|
|
||
|
|
There is currently no `expected-stderr:` line. Standard error is implicitly
|
||
|
|
expected to be empty, and any output will cause a test failure. (The support is
|
||
|
|
there, but not wired up because we haven't needed it yet.)
|
||
|
|
|
||
|
|
The fields can appear in any order, but every test must contain at least a
|
||
|
|
`name:` line and a `command:` line.
|
||
|
|
|
||
|
|
## Output
|
||
|
|
|
||
|
|
The output is intended to resemble gtest.
|
||
|
|
|
||
|
|
## Future Directions
|
||
|
|
|
||
|
|
* It's often useful to be able to *match* against stdout/stderr/a file rather
|
||
|
|
than give exact expected output. We might want to add explicit support for
|
||
|
|
this. In the meantime, it's possible to use an `after:` with `grep -q` if
|
||
|
|
you redirect in your `command:`.
|
||
|
|
|
||
|
|
* In addition to using a `before:` (which will fail a test), it can be useful
|
||
|
|
to be able to specify tests that would cause us to *skip* a test. An example
|
||
|
|
would be "am I running as root?".
|
||
|
|
|
||
|
|
* It might be useful to be able to make exit status assertions other than 0?
|
||
|
|
|
||
|
|
* There's currently no way (other than the `files/` directory) to share repeated
|
||
|
|
setup between tests.
|