|
| 1 | +# HOWTO: Writing script tests for a package |
| 2 | + |
| 3 | +Script testing is an advanced topic that assumes knowledge of [pipeline](./pipeline_testing.md) |
| 4 | +and [system](./system_testing.md) testing. |
| 5 | + |
| 6 | +Testing packages with script testing is only intended for testing cases that |
| 7 | +cannot be adequately covered by the pipeline and system testing tools such as |
| 8 | +testing failure paths and package upgrades. It can also be used for debugging |
| 9 | +integrations stack issues. |
| 10 | + |
| 11 | +## Introduction |
| 12 | + |
| 13 | +The script testing system is build on the Go testscript package with extensions |
| 14 | +provided to allow scripting of stack and integration operations such as |
| 15 | +bringing up a stack, installing packages and running agents. For example, using |
| 16 | +these commands it is possible to express a system test as described in the |
| 17 | +system testing [Conceptual Process](./system_testing.md#conceptual-process) section. |
| 18 | + |
| 19 | + |
| 20 | +## Expressing tests |
| 21 | + |
| 22 | +Tests are written as [txtar format](https://pkg.go.dev/golang.org/x/tools/txtar#hdr-Txtar_format) |
| 23 | +files in a data stream's \_dev/test/scripts directory. The logic for the test is |
| 24 | +written in the txtar file's initial comment section and any additional resource |
| 25 | +files are included in the txtar file's files sections. |
| 26 | + |
| 27 | +The standard commands and behaviors for testscript scripts are documented in |
| 28 | +the [testscript package documentation](https://pkg.go.dev/github.com/rogpeppe/go-internal/testscript). |
| 29 | + |
| 30 | + |
| 31 | +## Extension commands |
| 32 | + |
| 33 | +The test script command provides additional commands to aid in interacting with |
| 34 | +a stack, starting agents and services and validating results. |
| 35 | + |
| 36 | +- `sleep`: sleep for a duration (Go `time.Duration` parse syntax) |
| 37 | +- `date`: print the current time in RFC3339, optionally setting a variable with the value |
| 38 | +- `GET`: perform an HTTP GET request, emitting the response body to stdout |
| 39 | +- `POST`: perform an HTTP POST request, emitting the response body to stdout |
| 40 | +- `match_file`: perform a grep pattern match between a pattern file and a data file |
| 41 | + |
| 42 | +- stack commands: |
| 43 | + - `stack_up`: bring up a version of the Elastic stack |
| 44 | + - `use_stack`: use a running Elastic stack |
| 45 | + - `stack_down`: take down a started Elastic stack |
| 46 | + - `dump_logs`: dump the logs from the stack into a directory |
| 47 | + - `get_policy`: print the details for a policy |
| 48 | + |
| 49 | +- agent commands: |
| 50 | + - `install_agent`: install an Elastic Agent policy |
| 51 | + - `uninstall_agent`: remove an installed Elastic Agent policy |
| 52 | + |
| 53 | +- package commands: |
| 54 | + - `add_package`: add the current package's assets |
| 55 | + - `remove_package`: remove assets for the current package |
| 56 | + - `add_package_zip`: add assets from a Zip-packaged integration package |
| 57 | + - `remove_package_zip`: remove assets for Zip-packaged integration package |
| 58 | + - `upgrade_package_latest`: upgrade the current package or another named package to the latest version |
| 59 | + |
| 60 | +- data stream commands: |
| 61 | + - `add_data_stream`: add a data stream policy |
| 62 | + - `remove_data_stream`: remove a data stream policy |
| 63 | + - `get_docs`: get documents from a data stream |
| 64 | + |
| 65 | +- docker commands: |
| 66 | + - `docker_up`: start a docker service |
| 67 | + - `docker_down`: stop a started docker service and print the docker logs to stdout |
| 68 | + - `docker_signal`: send a signal to a running docker service |
| 69 | + - `docker_wait_exit`: wait for a docker service to exit |
| 70 | + |
| 71 | +- pipeline commands: |
| 72 | + - `install_pipelines`: install ingest pipelines from a path |
| 73 | + - `simulate`: run a pipeline test |
| 74 | + - `uninstall_pipelines`: remove installed ingest pipelines |
| 75 | + |
| 76 | + |
| 77 | +## Environment variables |
| 78 | + |
| 79 | +- `CONFIG_ROOT`: the `elastic-package` configuration root path |
| 80 | +- `CONFIG_PROFILES`: the `elastic-package` profiles configuration root path |
| 81 | +- `HOME`: the user's home directory path |
| 82 | +- `PKG`: the name of the running package |
| 83 | +- `PKG_ROOT`: the path to the root of the running package |
| 84 | +- `CURRENT_VERSION`: the current version of the package |
| 85 | +- `PREVIOUS_VERSION`: the previous version of the package |
| 86 | +- `DATA_STREAM`: the name of the data stream |
| 87 | +- `DATA_STREAM_ROOT`: the path to the root of the data stream |
| 88 | + |
| 89 | + |
| 90 | +## Conditions |
| 91 | + |
| 92 | +The testscript package allows conditions to be set that allow conditional |
| 93 | +execution of commands. The test script command adds a condition that reflects |
| 94 | +the state of the `--external-stack` flag. This allows tests to be written that |
| 95 | +conditionally use either an externally managed stack, or a stack that has been |
| 96 | +started by the test script. |
| 97 | + |
| 98 | + |
| 99 | +## Example |
| 100 | + |
| 101 | +As an example, a basic system test could be expressed as follows. |
| 102 | +``` |
| 103 | +# Only run the test if --external-stack=true. |
| 104 | +[!external_stack] skip 'Skipping external stack test.' |
| 105 | +# Only run the test if the jq executable is in $PATH. This is needed for a test below. |
| 106 | +[!exec:jq] skip 'Skipping test requiring absent jq command' |
| 107 | +
|
| 108 | +# Register running stack. |
| 109 | +use_stack -profile ${CONFIG_PROFILES}/default |
| 110 | +
|
| 111 | +# Install an agent. |
| 112 | +install_agent -profile ${CONFIG_PROFILES}/default NETWORK_NAME |
| 113 | +
|
| 114 | +# Bring up a docker container. |
| 115 | +# |
| 116 | +# The service is described in the test-hits/docker-compose.yml below with |
| 117 | +# its logs in test-hits/logs/generated.log. |
| 118 | +docker_up -profile ${CONFIG_PROFILES}/default -network ${NETWORK_NAME} test-hits |
| 119 | +
|
| 120 | +# Add the package resources. |
| 121 | +add_package -profile ${CONFIG_PROFILES}/default |
| 122 | +
|
| 123 | +# Add the data stream. |
| 124 | +# |
| 125 | +# The configuration for the test is described in test_config.yaml below. |
| 126 | +add_data_stream -profile ${CONFIG_PROFILES}/default test_config.yaml DATA_STREAM_NAME |
| 127 | +
|
| 128 | +# Start the service. |
| 129 | +docker_signal test-hits SIGHUP |
| 130 | +
|
| 131 | +# Wait for the service to exit. |
| 132 | +docker_wait_exit -timeout 5m test-hits |
| 133 | +
|
| 134 | +# Check that we can see our policy. |
| 135 | +get_policy -profile ${CONFIG_PROFILES}/default -timeout 1m ${DATA_STREAM_NAME} |
| 136 | +cp stdout got_policy.json |
| 137 | +exec jq '.name=="'${DATA_STREAM_NAME}'"' got_policy.json |
| 138 | +stdout true |
| 139 | +
|
| 140 | +# Take down the service and check logs for our message. |
| 141 | +docker_down test-hits |
| 142 | +! stderr . |
| 143 | +stdout '"total_lines":10' |
| 144 | +
|
| 145 | +# Get documents from the data stream. |
| 146 | +get_docs -profile ${CONFIG_PROFILES}/default -want 10 -timeout 5m ${DATA_STREAM_NAME} |
| 147 | +cp stdout got_docs.json |
| 148 | +
|
| 149 | +# Remove the data stream. |
| 150 | +remove_data_stream -profile ${CONFIG_PROFILES}/default ${DATA_STREAM_NAME} |
| 151 | +
|
| 152 | +# Uninstall the agent. |
| 153 | +uninstall_agent -profile ${CONFIG_PROFILES}/default -timeout 1m |
| 154 | +
|
| 155 | +# Remove the package resources. |
| 156 | +remove_package -profile ${CONFIG_PROFILES}/default |
| 157 | +
|
| 158 | +-- test-hits/docker-compose.yml -- |
| 159 | +version: '2.3' |
| 160 | +services: |
| 161 | + test-hits: |
| 162 | + image: docker.elastic.co/observability/stream:v0.20.0 |
| 163 | + volumes: |
| 164 | + - ./logs:/logs:ro |
| 165 | + command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9999 -p=tcp /logs/generated.log |
| 166 | +-- test-hits/logs/generated.log -- |
| 167 | +ntpd[1001]: kernel time sync enabled utl |
| 168 | +restorecond: : Reset file context quasiarc: liqua |
| 169 | +auditd[5699]: Audit daemon rotating log files |
| 170 | +anacron[5066]: Normal exit ehend |
| 171 | +restorecond: : Reset file context vol: luptat |
| 172 | +heartbeat: : <<eumiu.medium> Processing command: accept |
| 173 | +restorecond: : Reset file context nci: ofdeFin |
| 174 | +auditd[6668]: Audit daemon rotating log files |
| 175 | +anacron[1613]: Normal exit mvolu |
| 176 | +ntpd[2959]: ntpd gelit-r tatno |
| 177 | +-- test_config.yaml -- |
| 178 | +input: tcp |
| 179 | +vars: ~ |
| 180 | +data_stream: |
| 181 | + vars: |
| 182 | + tcp_host: 0.0.0.0 |
| 183 | + tcp_port: 9999 |
| 184 | +``` |
| 185 | + |
| 186 | +Other complete examples can be found in the [with_script test package](https://github.com/elastic/elastic-package/blob/main/test/packages/other/with_script/data_stream/first/_dev/test/scripts). |
| 187 | + |
| 188 | + |
| 189 | +## Running script tests |
| 190 | + |
| 191 | +The `elastic-package test script` command has the following sub-command-specific |
| 192 | +flags: |
| 193 | + |
| 194 | +- `--continue`: continue running the script if an error occurs |
| 195 | +- `--data-streams`: comma-separated data streams to test |
| 196 | +- `--external-stack`: use external stack for script tests (default true) |
| 197 | +- `--run`: run only tests matching the regular expression |
| 198 | +- `--scripts`: path to directory containing test scripts (advanced use only) |
| 199 | +- `--update`: update archive file if a cmp fails |
| 200 | +- `--verbose-scripts`: verbose script test output (show all script logging) |
| 201 | +- `--work`: print temporary work directory and do not remove when done |
| 202 | + |
| 203 | + |
| 204 | +## Limitations |
| 205 | + |
| 206 | +While the testscript package allows reference to paths outside the configuration |
| 207 | +root and the package's root, the backing `elastic-package` infrastructure does |
| 208 | +not, so it is advised that tests only refer to paths within the `$WORK` and |
| 209 | +`$PKG_ROOT` directories. |
0 commit comments