Test exit codes aren't checked on Windows #1792
Description
This is, I think, at least partly the same issue as #1050.
I recently had some test issues in a package that I was building and I realised that, as described on that previous issue, the failing tests were in fact being ignored on Windows and the test phase was successful regardless. As far as I can tell, since there are no additional checks performed/inserted by rattler-build, the overall test phase will only fail if the last (or single) test command fails so that the final exit code is non-zero. If any earlier command in the list fails, this will have no effect on the overall result.
On that previous issue, there is this reply:
Do you know if this is different from conda-build? We might miss some CLI arg when calling cmd.exe but apart from that idk what's different.
Windows has always been pretty weird with the if %exitcode% exit 1 stuff.
AFAICS, that is exactly the problem! If you look at the conda-build implementation, you can see that they are inserting exactly those commands after every test command in the generated batch file:
That particular construction is questionable - as was mentioned on the previous issue, you usually want IF ERRORLEVEL 1 rather than IF %ERRORLEVEL% NEQ 0 since otherwise something can set an ERRORLEVEL env var and break things. However, it seems that it was changed to work that way for a specific reason so possibly best to mimic it as-is. Or here's a comprehensive StackOverflow answer on the topic if you want to do slightly better at a cost of more complexity: https://stackoverflow.com/a/10936093