regression test readme

Author: calccrypto

test/regression/README.md

@@ -0,0 +1,78 @@
+# Regression Tests
+
+This directory contains regression tests for GUFI.
+
+## Usage
+
+Each regression test consists of at least two files: a bash script and
+a corresponding .expected file. There might be additional files, such
+as Python scripts, associated with each test. To run a single set of
+tests, run the script file from a directory that allows for extended
+attributes, and it should work. The output will be printed to standard
+out as well as a corresponding .out file. The diff between the
+.expected file and .outfile will also be printed. The regression tests
+can be viewed as known answer tests.
+
+If a regression test's .expected needs to be updated with the latest
+output from the test script, the .out file can be usd to directly
+overwrite the .expected file with `cp`, `mv`, or `cat` and a redirect.
+
+## Common Files
+
+- `setup.sh` contains common functions, variables, cleanup code, etc.
+    - `replace`: replaces environment-specific text with common text,
+      such as the actual path to an executable with the name
+        - When adding a new test, add variables containing patterns to
+          replace and modify this function to do the replacement
+
+    - `run_sort`: runs the given command, printing the command as well
+      as the sorted output, passed into `replace`
+
+    - `run_no_sort`: runs the given command, printing the command as
+      well as the unsorted output, passed into `replace`
+
+    - `setup_cleanup`: function that is called on exit of script
+        - Wrap this with a test-specific cleanup function if the test
+          requires custom cleanup
+
+- `generatetree.sh` generates a fixed tree with known properties
+    - This is the most commonly used script/tree
+
+- `rollup_tree.sh` generates a fixed tree with known properties for
+  testing rollups with
+    - Requires `sudo`
+
+- `gufi_tool.py` fakes entry points into Python tools in order to use
+  a test configuration file instead of the real one
+
+## Test Script Structure
+1. `#!/usr/bin/env bash`
+2. License
+3. `set -e`
+4. `. setup.sh 1`
+    - Pass in `0` if the source tree and index do not need to be generated
+5. Define variables
+6. If necessary, set up a cleanup exit function and call it
+    - Call `setup_cleanup` in this function
+7. Define the name where the output will be written to
+    - `OUTPUT="<name>.out"`
+8. Define functions
+9. Run tests in a subshell
+    - `tee` all output to `"${OUTPUT}"`
+10. `@DIFF@ @CMAKE_CURRENT_BINARY_DIR@/<name>.expected "${OUTPUT}"`
+11. `rm "${OUTPUT}"`
+    - This only happens if diff returned 0
+
+## Adding a New Test Script
+1. Create an empty test script with the name `<name>.sh.in` in this directory
+2. Create an empty .expected file in this directory
+3. Add the test script and expected file to git
+4. Add `<name>` to the appropriate location in `CMakeLists.txt`
+6. Add contents into the test script
+    - See [Test Script Structure](#test-script-structure)
+5. Rebuild to get the latest test script into the build directory
+    - Will be called `<name>.sh`
+7. Run the test script until satisfied with the output
+8. Overwrite the source .expected file with `<name>.output`
+9. If available, run `make shellcheck` to find issues in the test script
+10. Commit and push