Error in st2-run-pack-tests when running with st2 3.9dev images

[NitinNahal]

SUMMARY

Testing st2-run-pack-tests with st2 3.9dev seems to be broken

STACKSTORM VERSION

Paste the output of st2 --version:

/opt/stackstorm# st2 --version
st2 3.9dev (49f61fc), on Python 3.10.12

OS, environment, install method

Docker method of installation with st2 3.9dev tagged images

Steps to reproduce the problem

I have used following command (which i have been using in st2 3.8 so far) to run the test case e.g. for a pack named PACK_NAME, the error seems to be related to pytest -

(PACK_NAME) /opt/stackstorm# st2-run-pack-tests -p packs/PACK_NAME/ -c -f TEST CASE

• First attempt : Tried running above command which includes "-c" for coverage - Fails with error related to

ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov=/opt/stackstorm/packs/PACK_NAME/img --cov=/opt/stackstorm/packs/PACK_NAME/.git --cov=/opt/stackstorm/packs/PACK_NAME/docs --cov=/opt/stackstorm/packs/PACK_NAME/flowcharts --cov=/opt/stackstorm/packs/PACK_NAME/rules --cov=/opt/stackstorm/packs/PACK_NAME/actions --cov=/opt/stackstorm/packs/PACK_NAME/lib --cov=/opt/stackstorm/packs/PACK_NAME/config
  inifile: None
  rootdir: /opt/stackstorm/packs/PACK_NAME

• Second attempt: Dropped "-c" to check if it runs without it - Fails with error that test case file is not located (which is located at the path of the pack)

Running tests...
=================================================================================== test session starts ===================================================================================
platform linux -- Python 3.10.12, pytest-7.0.1, pluggy-1.5.0 -- /opt/stackstorm/st2/bin/python
cachedir: .pytest_cache
rootdir: /opt/stackstorm/packs/PACK_NAME
collected 0 items                                                                                                                                                                         

================================================================================== no tests ran in 0.00s ==================================================================================
ERROR: file or directory not found: Test_file:TestCaseClass:testfunction

Expected Results

I expect test cases to run normally as mentioned in https://docs.stackstorm.com/development/pack_testing.html?#running-tests

Actual Results

Attempt 1 -

(PACK_NAME) /opt/stackstorm# st2-run-pack-tests -p packs/PACK_NAME/ -c -f TESTCASE
Running tests for pack: PACK_NAME
Creating virtualenv in /tmp/st2-pack-tests-virtualenvs/PACK_NAME...
created virtual environment CPython3.10.12.final.0-64 in 273ms
  creator CPython3Posix(dest=/tmp/st2-pack-tests-virtualenvs/PACK_NAME, clear=False, no_vcs_ignore=False, global=True)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==25.1.1, setuptools==80.9.0, wheel==0.45.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Activating virtualenv in /tmp/st2-pack-tests-virtualenvs/PACK_NAME...
Collecting pip==20.3.3
  Downloading pip-20.3.3-py2.py3-none-any.whl.metadata (4.3 kB)
Downloading pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 26.7 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 25.1.1
    Uninstalling pip-25.1.1:
      Successfully uninstalled pip-25.1.1
Successfully installed pip-20.3.3
Installing global pack test dependencies...
WARNING: You are using pip version 20.3.3; however, version 25.1.1 is available.
You should consider upgrading via the '/tmp/st2-pack-tests-virtualenvs/PACK_NAME/bin/python -m pip install --upgrade pip' command.
Installing pack-specific dependencies...
WARNING: You are using pip version 20.3.3; however, version 25.1.1 is available.
You should consider upgrading via the '/tmp/st2-pack-tests-virtualenvs/PACK_NAME/bin/python -m pip install --upgrade pip' command.
Running tests...
/opt/stackstorm/st2/lib/python3.10/site-packages/_pytest/config/__init__.py:325: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: helpconfig, Hook: pytest_cmdline_parse
UsageError: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov=/opt/stackstorm/packs/PACK_NAME/img --cov=/opt/stackstorm/packs/PACK_NAME/.git --cov=/opt/stackstorm/packs/PACK_NAME/docs --cov=/opt/stackstorm/packs/PACK_NAME/flowcharts --cov=/opt/stackstorm/packs/PACK_NAME/rules --cov=/opt/stackstorm/packs/PACK_NAME/actions --cov=/opt/stackstorm/packs/PACK_NAME/lib --cov=/opt/stackstorm/packs/PACK_NAME/config
  inifile: None
  rootdir: /opt/stackstorm/packs/PACK_NAME
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config = pluginmanager.hook.pytest_cmdline_parse(
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov=/opt/stackstorm/packs/PACK_NAME/img --cov=/opt/stackstorm/packs/PACK_NAME/.git --cov=/opt/stackstorm/packs/PACK_NAME/docs --cov=/opt/stackstorm/packs/PACK_NAME/flowcharts --cov=/opt/stackstorm/packs/PACK_NAME/rules --cov=/opt/stackstorm/packs/PACK_NAME/actions --cov=/opt/stackstorm/packs/PACK_NAME/lib --cov=/opt/stackstorm/packs/PACK_NAME/config
  inifile: None
  rootdir: /opt/stackstorm/packs/PACK_NAME

Attempt 2 -

/opt/stackstorm# st2-run-pack-tests -p /opt/stackstorm/packs/PACK_NAME/ -f Test_file:TestCaseClass:testfunction
Running tests for pack: PACK_NAME
Creating virtualenv in /tmp/st2-pack-tests-virtualenvs/PACK_NAME...
created virtual environment CPython3.10.12.final.0-64 in 356ms
  creator CPython3Posix(dest=/tmp/st2-pack-tests-virtualenvs/PACK_NAME, clear=False, no_vcs_ignore=False, global=True)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: ...
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Activating virtualenv in /tmp/st2-pack-tests-virtualenvs/PACK_NAME...
Collecting pip==20.3.3
  Using cached pip-20.3.3-py2.py3-none-any.whl.metadata (4.3 kB)
Using cached pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 25.1.1
    Uninstalling pip-25.1.1:
      Successfully uninstalled pip-25.1.1
Successfully installed pip-20.3.3
Installing global pack test dependencies...
Installing pack-specific dependencies...
WARNING: You are using pip version 20.3.3; however, version 25.1.1 is available.
You should consider upgrading via the '/tmp/st2-pack-tests-virtualenvs/PACK_NAME/bin/python -m pip install --upgrade pip' command.
Running tests...
=================================================================================== test session starts ===================================================================================
platform linux -- Python 3.10.12, pytest-7.0.1, pluggy-1.5.0 -- /opt/stackstorm/st2/bin/python
cachedir: .pytest_cache
rootdir: /opt/stackstorm/packs/PACK_NAME
collected 0 items                                                                                                                                                                         

================================================================================== no tests ran in 0.00s ==================================================================================
ERROR: file or directory not found: Test_file:TestCaseClass:testfunction

Thanks!

[NitinNahal]

Tested same thing in v3.8.1 , just to be sure, there it works fine.

[nzlosh]

@NitinNahal Any idea of a fix for this?

[NitinNahal]

@NitinNahal Any idea of a fix for this?

Currently i am not sure of what causes such an error. Can it be related to internal migration to pytest ?

[nzlosh]

There's a good chance it's related, but all the pytest work is was completed with tests passing. There might have been something overlooked in st2-run-pack-tests that needs fixing for pytest to work correctly.

[NitinNahal]

Hmmm, trying to gather the changes done to this script recently, i could just see the change was part of -

#6291

d356a75

[nzlosh]

That looks like that switch to pytest. Any idea what other changes might be required to get the script working again?

[NitinNahal]

Hello @nzlosh , Yes spent some time today to check, there seems to be multiple small issues with the script which seems to have been missed during migration. Also there might be a requirement to update the docs on how we run this script when a file has to be specified.

I will try to create a PR for the changes I found in this script, since now I am able to run the test as expected.
Also can you let me know how can we update something on docs for this section if I want - https://docs.stackstorm.com/development/pack_testing.html?#running-tests

[nzlosh]

@NitinNahal that's great news! Thank you for looking into that. You can update the documentation by opening a PR against the documentation repository https://github.com/StackStorm/st2docs

[NitinNahal]

Following two PRs for the same -

st2 : #6340
st2docs: StackStorm/st2docs#1200

I hope we can include them for 3.9 release.

[NitinNahal]

Hey @nzlosh can you help to review these PRs ? or if you could help to get it reviewed by some other TSC members
Also do we have any tentative timeline for when 3.9 release will be made officially ?