regression test readme

Author: calccrypto

test/regression/README.md

@@ -0,0 +1,86 @@
+# Regression Tests
+
+This directory contains regression tests for GUFI.
+
+## Usage
+
+Each regression test consists of at least two files: a bash script and
+a corresponding .expected file. There might be additional files, such
+as Python scripts, associated with each test. The test scripts will
+not be installed. They will be copied to `test/regression` and
+modified to replace CMake variables.
+
+To run all tests, run `ctest` or `make test`. To run a single test,
+run the script file directly. The individual script files should pass
+when running from any writable directory that allows for extended
+attributes to be applied to filesystem entries. Note that some test
+scripts require `sudo`. All of tests should pass with or without
+`sudo`.
+
+The output of a test script will be printed to standard out as well as
+a corresponding .out file. The diff between the .expected file and
+.outfile will also be printed (the regression tests can be thought of
+as known answer tests). If a regression test's .expected needs to be
+updated, the .out file can be used to directly overwrite the .expected
+file with `cat` and a redirect, `cp`, or `mv`.
+
+## Common Files
+
+- `setup.sh` contains common functions, variables, setup code, cleanup code, etc.
+    - `replace`: replaces environment-specific text with common text,
+      such as the actual path to an executable with the name
+        - When adding a new test, add variables containing patterns to
+          replace and modify this function to do the replacement
+
+    - `run_sort`: runs the given command, printing the command as well
+      as the sorted output, passed into `replace`
+
+    - `run_no_sort`: runs the given command, printing the command as
+      well as the unsorted output, passed into `replace`
+
+    - `setup_cleanup`: function that is called on exit of script
+        - Wrap this with a test-specific cleanup function if the test
+          requires custom cleanup
+
+- `generatetree.sh` generates a fixed source tree with known properties
+    - This is the most commonly used script/tree
+    - This is generally automatically called through sourcing `setup.sh`
+
+- `rollup_tree.sh` generates a fixed tree with known properties for
+  testing rollups with
+    - Requires `sudo`
+
+- `gufi_tool.py` fakes entry points into Python tools in order to use
+  a test configuration file instead of the real one
+
+## Test Script Structure
+1. `#!/usr/bin/env bash`
+2. License
+3. `set -e`
+4. `. setup.sh 1`
+    - Pass in `0` if the source tree and index do not need to be generated
+5. Define variables
+6. If necessary, set up a cleanup exit function and call it
+    - Call `setup_cleanup` in this function
+7. Define the name where the output will be written to
+    - `OUTPUT="<name>.out"`
+8. Define functions
+9. Run tests in a subshell
+    - `tee` all output to `"${OUTPUT}"`
+10. `@DIFF@ @CMAKE_CURRENT_BINARY_DIR@/<name>.expected "${OUTPUT}"`
+11. `rm "${OUTPUT}"`
+    - This only happens if `@DIFF@` returned 0
+
+## Adding a New Test Script
+1. Create an empty test script with the name `<name>.sh.in` in this directory
+2. Create an empty .expected file in this directory
+3. Add the test script and expected file to git
+4. Add `<name>` to the appropriate location in `CMakeLists.txt`
+5. Add contents into the test script
+    - See [Test Script Structure](#test-script-structure)
+6. Rebuild to get the latest test script into the build directory
+    - Will be called `<name>.sh`
+7. If not satisfied with the tests or output, go to step 5
+8. Overwrite the source .expected file with `<name>.output`
+9. If available, run `make shellcheck` to find issues in the test script
+10. Commit and push