Fix assertion against entering lossless mode #4662

Job	Run time
build-macos (Cargo build (x64), cargo-build, x86_64-apple-darwin)	4m 38s
build-windows (cargo-build, x86_64-pc-windows-msvc)	6m 12s
build-macos (Cargo test (x64), cargo-test, x86_64-apple-darwin)	7m 14s
build-unix (beta-build)	4m 17s
build-windows (cargo-test, x86_64-pc-windows-msvc)	6m 22s
rustfmt-clippy	3m 26s
build-macos (Cargo C-build (x64), cargo-c, x86_64-apple-darwin)	10m 25s
build-unix (1.70.0-tests)	6m 51s
build-windows (cargo-c, x86_64-pc-windows-gnu)	10m 38s
build-macos (Cargo build (Arm64), cargo-build, aarch64-apple-darwin)	3m 5s
build-unix (aom-tests)	9m 10s
build-macos (Cargo C-build (Arm64), cargo-c, aarch64-apple-darwin)	7m 55s
build-unix (dav1d-tests)	10m 45s
build-unix (no-asm-tests)	18m 46s
build-unix (grcov-codecov)	15m 43s
build-unix (bench)	4m 48s
build-unix (doc)	2m 41s
build-unix (cargo-c)	8m 19s
build-unix (check-no-default)	2m 33s
build-unix (check-extra-feats)	3m 36s
build-unix (check-unstable-feats)	2m 49s
build-unix (fuzz)	8m 14s
build-unix (dav1d-tests-arm64, stable)	11m 56s
	2h 50m 23s

Provide feedback