opa test --bench panics on failing test

[anderseknert]

Tested with both v0.70.0 and current main. While I think --bench is meant to work with successful tests only, it seems like the error handling is borked.

b.rego

package b

import rego.v1

x := true

test_x if {
    not x
}

> opa test b.rego --bench
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x10460281c]

goroutine 28 [running]:
testing.(*common).decorate(0x1400004fbe8?, {0x140003962d0, 0x29}, 0x1054439e0?)
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/testing.go:775 +0x9c
testing.(*common).logDepth(0x1400037fb88, {0x140003962d0, 0x29}, 0x3)
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/testing.go:1042 +0x150
testing.(*common).log(...)
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/testing.go:1011
testing.(*common).Fatal(0x1400037fb88, {0x1400004fee0?, 0x105670898?, 0x140003812d0?})
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/testing.go:1082 +0x50
github.com/open-policy-agent/opa/tester.(*Runner).runBenchmark.func1(0x1400037fb88)
	/Users/anderseknert/git/opa/opa/tester/runner.go:588 +0x36c
testing.(*B).runN(0x1400037fb88, 0x1)
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/benchmark.go:193 +0x128
testing.(*B).run1.func1()
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/benchmark.go:215 +0x4c
created by testing.(*B).run1 in goroutine 26
	/opt/homebrew/Cellar/go/1.23.3/libexec/src/testing/benchmark.go:208 +0x90

If I change the test to succeed, the command works as expected.

[ashutosh-narkar]

Looks like it's hitting this case.

[anderseknert]

Yes, but how does that turn into a panic / nil dereference? 🤔

[ashutosh-narkar]

I haven't checked the code but are there any other goroutines running?