Skip to content

Commit a246c48

Browse files
dkcummingdkcumming
authored
UI test running (#66)
I added a couple of `Makefile` rules to pull ui tests from the [rust compiler ui test suite](https://github.com/rust-lang/rust). The logic from mir-semantics-compiletest that filters all the tests is now brought to stable-mir-json (Perhaps invalidating mir-semantics-compiletest). `remake-ui-tests` will take the filtered tests and run each one, if they pass they are added to `passing.tsv` and the source and `*.smir.json` is _optionally_ saved. If the tests fail the output is saved with the test name and error code being added to `failing.tsv`. Currently it seems that there are either panics or stack overflows as the errors. `test-ui` re-runs the passing ui tests with no ouput and only reports failing tests
1 parent 4a9ca85 commit a246c48

12 files changed

+6112
-1
lines changed

‎.github/workflows/test.yml

+34-1
Original file line numberDiff line numberDiff line change
@@ -66,4 +66,37 @@ jobs:
6666
6767
- name: 'Run smir integration tests'
6868
run: |
69-
make integration-test
69+
make integration-test
70+
71+
ui-tests:
72+
needs: code-quality
73+
name: "ui-tests"
74+
runs-on: [self-hosted, linux, normal]
75+
steps:
76+
- name: 'Check out code'
77+
uses: actions/checkout@v4
78+
with:
79+
# Check out pull request HEAD instead of merge commit.
80+
ref: ${{ github.event.pull_request.head.sha }}
81+
submodules: recursive
82+
83+
- name: 'Check out Rust repo'
84+
uses: actions/checkout@v4
85+
with:
86+
repository: rust-lang/rust
87+
ref: a2545fd6fc66b4323f555223a860c451885d1d2b # hash of Hardcoded Rust version
88+
path: rust
89+
fetch-depth: 1
90+
91+
- name: "Set up nightly Rust" # https://github.com/rust-lang/rustup/issues/3409
92+
uses: dtolnay/rust-toolchain@master
93+
with:
94+
toolchain: nightly-2024-11-29 # Hardcoded version, same as is in the build.rs
95+
96+
- name: 'Build stable-mir-json'
97+
run: | # Warning check should be redundant since code-quality runs first
98+
RUSTFLAGS='--deny warnings' cargo build -vv
99+
100+
- name: 'Run smir integration tests'
101+
run: |
102+
RUST_DIR_ROOT='rust' make test-ui

‎.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
11
target/
2+
tests/ui/failing/*
3+
tests/ui/passing/*

‎Makefile

+19
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,22 @@ format:
4848

4949
style-check: format
5050
cargo clippy
51+
52+
.PHONY: remake-ui-tests test-ui
53+
54+
remake-ui-tests:
55+
# Check if RUST_DIR_ROOT is set
56+
if [ -z "$$RUST_DIR_ROOT" ]; then \
57+
echo "Error: RUST_DIR_ROOT is not set. Please set it to the absolute path to rust compiler checkout."; \
58+
exit 1; \
59+
fi
60+
# This will run without saving source files. Run the script manually to do this.
61+
bash tests/ui/remake_ui_tests.sh "$$RUST_DIR_ROOT"
62+
63+
test-ui:
64+
# Check if RUST_DIR_ROOT is set
65+
if [ -z "$$RUST_DIR_ROOT" ]; then \
66+
echo "Error: RUST_DIR_ROOT is not set. Please set it to the absolute path to rust compiler checkout."; \
67+
exit 1; \
68+
fi
69+
bash tests/ui/run_ui_tests.sh "$$RUST_DIR_ROOT"

‎tests/ui/README.md

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Rust UI Tests
2+
3+
These tests are taken from The [Rust compiler UI test suite](https://github.com/rust-lang/rust/tests/ui/).
4+
Some tests here are not appropriate for us to test with yet, so we need to filter valid tests for
5+
the current state of Stable MIR JSON are generated. To generate and run the tests a checkout of the
6+
rust compiler is required.
7+
8+
## Usage
9+
To generate the tests
10+
11+
```bash
12+
cd tests/ui/
13+
RUST_TOP=`<PATH-TO-RUST>` ./collect_test_sources.sh > ui_sources.txt
14+
```
15+
16+
To remake the ui tests and filter into the passing and failing directories (optionally storing the output)
17+
18+
```bash
19+
cd tests/ui/
20+
./remake_ui_tests.sh <PATH-TO-RUST> [y|n]
21+
```
22+
23+
To run the passing tests again
24+
25+
```bash
26+
cd tests/ui/
27+
./run_ui_tests.sh <PATH-TO-RUST>
28+
```

‎tests/ui/collect_test_sources.sh

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
#!/bin/bash
2+
set -eu # fail on non-zero retcode or undefined variable
3+
print_usage() {
4+
printf "usage: $(basename $0) [-x] [-e] [-f] [-t] [-h]\n\n"
5+
printf "print rustc UI test sources which satisfy certain criteria; by default those that are:\n"
6+
printf "normal, passing, contain a non-empty main function, and compile in reasonable time\n\n"
7+
printf "options:\n"
8+
printf "x\tinclude fixed tests [may contain run-rustfix header]\n"
9+
printf "e\tinclude tests with empty/missing main function\n"
10+
printf "f\tinclude failing tests [may NOT contain run-pass header] (implies -t)\n"
11+
printf "t\tinclude tests that fail to generate MIR within 30s\n"
12+
printf "h\tprint this usage and exit\n\n"
13+
printf "if -t unset:\n"
14+
printf " set environment variable RUSTC_MIR_VERBOSE to see raw rustc invocation and CLI output\n"
15+
printf " set environment variable RUSTC_MIR_NOFILEOPTS to skip //@ compile-flags: ...\n\n"
16+
[ $# -ne 0 ] && echo "ERROR: $*"
17+
exit 1
18+
}
19+
20+
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
21+
22+
# setup time mir filter
23+
# NOTE: temp file is needed because rustc chokes when writing directly to /dev/null
24+
# NOTE: this filter does not work natively on mac due to no timeout command
25+
RUSTC_MIR=$SCRIPT_DIR/rustc_mir.sh
26+
TMPMIR=$SCRIPT_DIR/temp.mir
27+
RUSTOPT=('-C' 'overflow-checks=off')
28+
TIMEMIR=('-execdir' 'timeout' '30s' "$RUSTC_MIR" '{}' "$TMPMIR" "${RUSTOPT[@]}" ';')
29+
30+
# setup grep filters
31+
HAS_MATCH=$SCRIPT_DIR/has_match.sh
32+
RUSTFIX=('-execdir' "$HAS_MATCH" 'n' '//@[[:space:]]*run-rustfix' '{}' ';')
33+
RUNPASS=('-execdir' "$HAS_MATCH" 'y' '//@[[:space:]]*run-pass' '{}' ';')
34+
HASMAIN=('-execdir' "$HAS_MATCH" 'y' 'fn[[:space:]]\{1,\}main[[:space:]]*([[:space:]]*)' '{}' ';')
35+
EMPMAIN=('-execdir' "$HAS_MATCH" 'n' 'fn[[:space:]]\{1,\}main[[:space:]]*([[:space:]]*)[[:space:]]*{[[:space:]]*}' '{}' ';')
36+
37+
# parse opts
38+
while getopts 'xfet' opt; do
39+
case "${opt}" in
40+
x) RUSTFIX=() ;;
41+
e) HASMAIN=()
42+
EMPMAIN=() ;;
43+
f) RUNPASS=()
44+
TIMEMIR=() ;;
45+
t) TIMEMIR=() ;;
46+
?|h) print_usage "environment variable RUST_TOP must point to a valid rustc distribution" ;;
47+
esac
48+
done
49+
50+
# check test dir
51+
# NOTE: ${var:-default} expands to default if var unset
52+
[ -z "${RUST_TOP:-}" ] && print_usage "set RUST_TOP environment variable to your local Rust compiler source directory before running this script"
53+
RUST_TESTS=${RUST_TOP}/tests
54+
[ ! -d "${RUST_TESTS}" ] && print_usage "RUST_TOP environment variable does not appear to point to a local Rust compiler source directory"
55+
56+
# clean temp mir file whenever timing mir, ignore errors
57+
if [ -n "${TIMEMIR[*]}" ]; then
58+
trap "rm \"$TMPMIR\" &> /dev/null" SIGINT SIGTERM EXIT
59+
fi
60+
61+
# find test source files that pass filters
62+
( cd "${RUST_TOP}";
63+
find tests/ui \
64+
-name '*.rs' \
65+
"${RUSTFIX[@]}" \
66+
"${RUNPASS[@]}" \
67+
"${HASMAIN[@]}" \
68+
"${EMPMAIN[@]}" \
69+
"${TIMEMIR[@]}" \
70+
-print )

0 commit comments

Comments
 (0)