How to get a clean test run?

[mattbnz]

Is there any special trick to getting the tests passing cleanly?

I can see that the CI helpers seem to be running (at least a subset of) the tests cleanly under the coverage target listed at https://github.com/inventree/InvenTree/blob/master/.github/workflows/qc_checks.yaml#L175, but trying to run the same either locally or in a fresh/clean GitPod environment results in 17-20 failing test cases for me, and from a quick skim at the failures there's no obvious issue I can pick up...

Am I missing anythign about how the tests are meant to work?

Here's one example failure locally:

(env) inventree@argon:~/src/InvenTree$ python manage.py test InvenTree.tests.TestSettings 
Creating test database for alias 'default'...
System check identified no issues (1 silenced).
....F
======================================================================
FAIL: test_set_user_to_few (InvenTree.tests.TestSettings)
Test adding an admin user via env variables.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/inventree/env/lib/python3.10/site-packages/django/test/utils.py", line 387, in inner
    return func(*args, **kwargs)
  File "/home/inventree/src/InvenTree/InvenTree/tests.py", line 703, in test_set_user_to_few
    self.assertEqual(user_count(), 2)
AssertionError: 1 != 2

----------------------------------------------------------------------
Ran 5 tests in 3.919s

FAILED (failures=1)
Destroying test database for alias 'default'...

and same in a fresh GitPod environment:

🤙 This task ran as a workspace prebuild
🎉 Well done on saving 2 minutes

start_server done
Waiting for database...
Database connection sucessful!
Watching for file changes with StatReloader
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (1 silenced).
August 10, 2022 - 02:01:09
Django version 3.2.15, using settings 'InvenTree.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

^C(venv) gitpod /workspace/InvenTree (master) $ cd InvenTree/
(venv) gitpod /workspace/InvenTree/InvenTree (master) $ python manage.py test InvenTree.tests.TestSettings
Creating test database for alias 'default'...
InvenTree.tests.TestSettings 
System check identified no issues (1 silenced).
InvenTree configuration file 'config.yaml' not found - creating default file
Created config file /workspace/InvenTree/InvenTree/my_special_conf.yaml
.Plugin configuration file does not exist - creating default file
...F
======================================================================
FAIL: test_set_user_to_few (InvenTree.tests.TestSettings)
Test adding an admin user via env variables.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/InvenTree/venv/lib/python3.8/site-packages/django/test/utils.py", line 387, in inner
    return func(*args, **kwargs)
  File "/workspace/InvenTree/InvenTree/InvenTree/tests.py", line 703, in test_set_user_to_few
    self.assertEqual(user_count(), 2)
AssertionError: 1 != 2

----------------------------------------------------------------------
Ran 5 tests in 7.810s

FAILED (failures=1)
Destroying test database for alias 'default'...
(venv) gitpod /workspace/InvenTree/InvenTree (master) $

It really feels like this test (and the others) are just broken - but then https://github.com/inventree/InvenTree/runs/7741210093?check_suite_focus=true seems to be an example of them passing fine in QC...

So I must be missing something, but what?

[SchrodingersGat]

@mattbnz you should be able to run the tests locally without fault. all tests get run on every merge of code, and we get notified of any failure due to regression or bug or whatever.

• Is it just the user_count test error you are seeing, or others?
• What is your "local" setup?
• Have you made any changes at all to the code?

My dev setup is under docker on windows 10. Not sure what @matmair runs. We both regularly run tests locally and I have not seen such an issue.

[matmair]

I am running debian buster, debian bullseye and MacOs 12.5.

[mattbnz]

I've included the list of tests I'm seeing failing below.

I'm on Ubuntu 22.04, running Inventree as a local user (separate from my normal account) in a virtualenv, basicaly following the Bare Metal setup at https://inventree.readthedocs.io/en/latest/start/install/, but then also seeing the same behaviour in GitPod which I was using to try and rule out any issues in my local setup.

No changes, I'm doing a fresh clone from github, adding a config file, and then running invoke update, invoke superuser and then invoke coverage.

Failing tests

ERROR: test_api_list (common.tests.PluginSettingsApiTest)
ERROR: test_plugin_list (common.tests.PluginSettingsApiTest)
ERROR: test_valid_plugin_slug (common.tests.PluginSettingsApiTest)
ERROR: test_disabled (plugin.base.integration.test_mixins.PanelMixinTests)
ERROR: test_enabled (plugin.base.integration.test_mixins.PanelMixinTests)
ERROR: test_api (plugin.base.label.test_label_mixin.LabelMixinTests)
ERROR: test_printing_endpoints (plugin.base.label.test_label_mixin.LabelMixinTests)
ERROR: test_printing_process (plugin.base.label.test_label_mixin.LabelMixinTests)
ERROR: test_email (plugin.builtin.integration.test_core_notifications.CoreNotificationTestTests)
ERROR: test_run_event (plugin.samples.event.test_event_sample.EventPluginSampleTests)
ERROR: test_run_locator (plugin.samples.locate.test_locate_sample.SampleLocatePlugintests)
ERROR: test_admin_action (plugin.test_api.PluginDetailAPITest)
ERROR: test_model (plugin.test_api.PluginDetailAPITest)
ERROR: test_plugin_install (plugin.test_api.PluginDetailAPITest)
FAIL: test_new_so_notification (order.test_sales_order.SalesOrderTest)
FAIL: test_overdue_notification (order.test_sales_order.SalesOrderTest)
FAIL: test_new_po_notification (order.tests.OrderTest)
FAIL: test_overdue_notification (order.tests.OrderTest)
FAIL: test_plugins_enabled (part.test_part.TemplateTagTest)
FAIL: test_function (plugin.base.integration.test_mixins.AppMixinTest)
FAIL: test_installed (plugin.base.integration.test_mixins.PanelMixinTests)
FAIL: test_installed (plugin.base.label.test_label_mixin.LabelMixinTests)
FAIL: test_function (plugin.builtin.action.test_simpleactionplugin.SimpleActionPluginTests)
FAIL: test_errors (plugin.builtin.barcodes.test_inventree_barcode.TestInvenTreeBarcode)
FAIL: test_return (plugin.samples.integration.test_api_caller.SampleApiCallerPluginTests)
FAIL: test_view (plugin.samples.integration.test_sample.SampleIntegrationPluginTests)
FAIL: test_calling (plugin.samples.integration.test_scheduled_task.ExampleScheduledTaskPluginTests)
FAIL: test_function (plugin.samples.integration.test_scheduled_task.ExampleScheduledTaskPluginTests)
FAIL: test_tag_safe_url (plugin.test_plugin.PluginTagTests)
FAIL: test_settings_page (InvenTree.test_views.ViewTests)
FAIL: test_helpers_cfg_file (InvenTree.tests.TestSettings)
FAIL: test_set_user_to_few (InvenTree.tests.TestSettings)

I will dig further on my end. Thanks for confirming the expectation is they should all pass.

[SchrodingersGat]

What db engine are you using? We run the test suite against postgresql, mysql and sqlite

[matmair]

@mattbnz with what command are you starting your tests? From the list of failing tests I can tell you that it is a problem with plugins beeing turned off/not working.
Normally the plugin settings are overritten when the system detects that tests are running - depending on what wrapper you use that might not work.

[matmair]

I just saw that the command is included above. Maybe try to run the tests from the root of the repo (/workspace/InvenTree) as all our runners do it that way. So the command would be:
python InvenTree/manage.py test InvenTree.tests.TestSettings

[mattbnz]

I'm using a sqlite DB, config snippet:

  # --- Example Configuration - sqlite3 ---
  ENGINE: sqlite3
  NAME: '/home/inventree/database.sqlite3'

Running from the repo root doesn't seem to change anything.

(env) inventree@argon:~/src$ python InvenTree/manage.py test InvenTree.tests.TestSettings
Creating test database for alias 'default'...
System check identified no issues (1 silenced).
F...F
======================================================================
FAIL: test_helpers_cfg_file (InvenTree.tests.TestSettings)
Test get_config_file.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/inventree/src/InvenTree/InvenTree/tests.py", line 752, in test_helpers_cfg_file
    self.assertIn('inventree/my_special_conf.yaml', str(config.get_config_file()).lower())
AssertionError: 'inventree/my_special_conf.yaml' not found in '/home/inventree/src/my_special_conf.yaml'

======================================================================
FAIL: test_set_user_to_few (InvenTree.tests.TestSettings)
Test adding an admin user via env variables.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/inventree/env/lib/python3.10/site-packages/django/test/utils.py", line 387, in inner
    return func(*args, **kwargs)
  File "/home/inventree/src/InvenTree/InvenTree/tests.py", line 703, in test_set_user_to_few
    self.assertEqual(user_count(), 2)
AssertionError: 1 != 2

----------------------------------------------------------------------
Ran 5 tests in 3.823s

FAILED (failures=2)
Destroying test database for alias 'default'...
(env) inventree@argon:~/src$ 

I've got some time scheduled tomorrow morning to dig into this - thanks for the hint/pointer about the plugins.