Multiple problems with stimela test system

[bennahugo]

Recent unchecked changes have resulted in multiple bugs in the test system

• stimela returns 0 when an exception is raised (logger catches but does not re-raise):

# Successfully built bfa89e6d7326
# Successfully tagged jenkins_cab/rfimasker:latest
...2020-04-14 22:29:20 STIMELA.test1 ERROR: Expecting any of types [(<class 'list'>, <class 'float'>)] for parameter 'bla3', but got 'str'
2020-04-14 22:29:20 STIMELA.test1 ERROR: job failed at 2020-04-14 22:29:20.035623 after 0:00:00.006004
2020-04-14 22:29:20 STIMELA.test1 ERROR: Traceback (most recent call last):
2020-04-14 22:29:20 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/recipe.py", line 610, in run
2020-04-14 22:29:20 STIMELA.test1 ERROR:     job.run_job()
2020-04-14 22:29:20 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/recipe.py", line 297, in run_job
2020-04-14 22:29:20 STIMELA.test1 ERROR:     self.job.parameter_file_name)
2020-04-14 22:29:20 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/cargo/cab/__init__.py", line 352, in update
2020-04-14 22:29:20 STIMELA.test1 ERROR:     param.validate(value)
2020-04-14 22:29:20 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/cargo/cab/__init__.py", line 84, in validate
2020-04-14 22:29:20 STIMELA.test1 ERROR:     self.dtype, self.name, type(value).__name__))
2020-04-14 22:29:20 STIMELA.test1 ERROR: stimela.exceptions.StimelaCabParameterError: Expecting any of types [(<class 'list'>, <class 'float'>)] for parameter 'bla3', but got 'str'
..2020-04-14 22:29:22 STIMELA.test1 ERROR: Parameter 'bla1', can only be either of ['a', 'b', 'c']
2020-04-14 22:29:22 STIMELA.test1 ERROR: job failed at 2020-04-14 22:29:22.679707 after 0:00:00.005612
2020-04-14 22:29:22 STIMELA.test1 ERROR: Traceback (most recent call last):
2020-04-14 22:29:22 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/recipe.py", line 610, in run
2020-04-14 22:29:22 STIMELA.test1 ERROR:     job.run_job()
2020-04-14 22:29:22 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/recipe.py", line 297, in run_job
2020-04-14 22:29:22 STIMELA.test1 ERROR:     self.job.parameter_file_name)
2020-04-14 22:29:22 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/cargo/cab/__init__.py", line 352, in update
2020-04-14 22:29:22 STIMELA.test1 ERROR:     param.validate(value)
2020-04-14 22:29:22 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/cargo/cab/__init__.py", line 68, in validate
2020-04-14 22:29:22 STIMELA.test1 ERROR:     self.name, self.choices))
2020-04-14 22:29:22 STIMELA.test1 ERROR: stimela.exceptions.StimelaCabParameterError: Parameter 'bla1', can only be either of ['a', 'b', 'c']
...2020-04-14 22:29:28 STIMELA.test1 ERROR: File 'testinput2.txt' for parameter 'bla2' could not be located at '/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/Stimela/stimela/tests/unit_tests/input/testinput2.txt'.
2020-04-14 22:29:28 STIMELA.test1 ERROR: job failed at 2020-04-14 22:29:28.244335 after 0:00:00.008241
2020-04-14 22:29:28 STIMELA.test1 ERROR: Traceback (most recent call last):
2020-04-14 22:29:28 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/recipe.py", line 610, in run
2020-04-14 22:29:28 STIMELA.test1 ERROR:     job.run_job()
2020-04-14 22:29:28 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/recipe.py", line 297, in run_job
2020-04-14 22:29:28 STIMELA.test1 ERROR:     self.job.parameter_file_name)
2020-04-14 22:29:28 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/cargo/cab/__init__.py", line 332, in update
2020-04-14 22:29:28 STIMELA.test1 ERROR:     _value, param.name, path))
2020-04-14 22:29:28 STIMELA.test1 ERROR: stimela.exceptions.StimelaCabParameterError: File 'testinput2.txt' for parameter 'bla2' could not be located at '/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/Stimela/stimela/tests/unit_tests/input/testinput2.txt'.
.2020-04-14 22:29:28 STIMELA.test1 ERROR: Parameter bla1 is required but has not been specified
2020-04-14 22:29:28 STIMELA.test1 ERROR: job failed at 2020-04-14 22:29:28.573429 after 0:00:00.002997
2020-04-14 22:29:28 STIMELA.test1 ERROR: Traceback (most recent call last):
2020-04-14 22:29:28 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/recipe.py", line 610, in run
2020-04-14 22:29:28 STIMELA.test1 ERROR:     job.run_job()
2020-04-14 22:29:28 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/recipe.py", line 297, in run_job
2020-04-14 22:29:28 STIMELA.test1 ERROR:     self.job.parameter_file_name)
2020-04-14 22:29:28 STIMELA.test1 ERROR:   File "/var/lib/jenkins/jobs/PR_stimela/workspace/13/projects/pyenv/lib/python3.6/site-packages/stimela/cargo/cab/__init__.py", line 262, in update
2020-04-14 22:29:28 STIMELA.test1 ERROR:     "Parameter {} is required but has not been specified".format(param0.name))
2020-04-14 22:29:28 STIMELA.test1 ERROR: stimela.exceptions.StimelaCabParameterError: Parameter bla1 is required but has not been specified

• with python 2 now dead the udocker tests of the kat7 acceptance test is no longer executed

[bennahugo]

@SpheMakh as discussed in the call right now

[bennahugo]

My bad on the last one, it does run - the output is no longer printed though:

<testcase classname="stimela-test-kat7.kat7_reduce" name="testEndToEndReduction" time="432.493"/>

[o-smirnov]

Is this output using the logging facility?

One difference is that Stimela no longer uses or initializes the root logger -- it makes a logger of its own. So if the testing code doesn't call logging.basicConfig(), it's possible that messages to the root logger are lost.

[bennahugo]

As discussed the issue is more to do with the fact that you are now handling info -> stdout and errors -> stderr. nosetests will print stderr even the test is making an assertion about an error condition. To clean up the build logs I introduced an override as discussed in #567