Can nix-bisect be used to find evaluation errors?

[das-g]

Can nix-bisect be used to find the commit that introduced a specific evaluation error (as opposed to a build error)? If so, how?

E.g., on NixOS/nixpkgs@e0759a4, python2.pkgs.moto fails to evaluate with bcrypt-3.2.0 not supported for interpreter python2.7. Can I use nix-bisect to find the commit since when that's the case? If so, how? Would the call be

extra-bisect start e0759a49733dfc3aa225b8a7423c00da6e1ecb67 e0759a49733dfc3aa225b8a7423c00da6e1ecb67~5000
extra-bisect run \
	nix-build-status \
	--max-rebuild 100 \
	--failure-line 'bcrypt-3.2.0 not supported for interpreter python2.7' \
	python2.pkgs.moto

(assuming python2.pkgs.moto does evaluate and build for e0759a49733dfc3aa225b8a7423c00da6e1ecb67~5000, which it does) or would that look for a build error with that output?

[timokau]

You can use nix-build-status --on-instantiation-failure=bad. The CLI currently does not support matching the specific instantiation failure. You could use the nix-bisect python library for that:

from nix_bisect import nix
try:
	nix.instantiate("python2.pkgs.moto")
except nix.InstantiationFailiure as e:
	if "bcrypt" in repr(e):
		sys.exit(1)
	else:
		sys.exit(127)

(untested), and then use extra-bisect with that script. Better CLI-support for that would be nice though.

Yes, --failure-line would look for a build error (and by default would exit with a "skip" exit code on an instantiation failure).

[timokau]

Keep in mind that nix-bisect has some very rough edges right now. There is a weird bug in the extra-bisect bisection runner where sometimes the bisection would not terminate. I haven't had time to look into that yet. Consider it a proof of concept for now, it may or may not work.