tests: refactor folder structure to split on the type of test

[eivindj-nordic]

Place unit tests in tests/unit/ folder, other tests in tests/ folder.

[github-actions]

You can find the documentation preview for this PR here.

[fundakol]

The testcase.yaml file may include multiple harnesses, not limited to just pytest. For instance, tests/pytest/subsys/kmu/hello_for_kmu/testcase.yaml contains both pytest and console harnesses. Therefore, moving these tests to the pytest folder is not sensible, as we run all these tests using twister, not with pytest. Also there is no such split in zephyr-rtos and nrf-sdk.

[eivindj-nordic]

The testcase.yaml file may include multiple harnesses, not limited to just pytest. For instance, tests/pytest/subsys/kmu/hello_for_kmu/testcase.yaml contains both pytest and console harnesses. Therefore, moving these tests to the pytest folder is not sensible, as we run all these tests using twister, not with pytest. Also there is no such split in zephyr-rtos and nrf-sdk.

You are correct there is no such split in zephyr, sdk-nrf etc. The reason we want to split it is that it gives an easier overview of the tests we have, e.g. what modules we have unit tests for, and what we have system tests (ztest, pytest ...) for. Also, there might be modules where we have both unit tests and system tests, and it is not very practical putting them in the same location. Of course the testcase.yaml file will have twister pick up the correct tests regardless. Twister can be used to run all the tests.

I'm open for other suggestions on how to structure it, though I don't think keeping it as it is is ideal.

[PerMac]

I agree with @fundakol on this. To me it is more useful and functional to have tests grouped by the relevant area than by a specific harness (framework) they use. If I am looking for bluetooth related stuff I would prefer to have them all in mutual directory than scattered across multiple places and trying to figure out what is the difference between ztest/console/pytest.
I think when we are talking about tests coverage we first think about the component/area and how well with what king of tests we have it covered, rather then "where we have unit test". In other words, first comes "what?" and only then "how". It is also easier from maintenance perspective where we work in multiple groups responsible for multiple areas. We don't have any group dedicated to unit tests and another to system ones.
Also as pointed by @fundakol harness in twister is not the best indicator of test type. One can have both unit level tests in ztests and system.

[lemrey]

Would it be fine to split just tests/unity and tests/ztest?

[eivindj-nordic]

Would it be fine to split just tests/unity and tests/ztest?

I don't think unity and ztest would be the correct terms then, though perhaps test/unit and test/system or so?
@MechanicV1 How would that work for your integration? How would it work leaving it as it is?

[MechanicV1]

Splitting it to unit/ and system/ seems cleaner !

[eivindj-nordic]

Split the tests into unit and system instead of based on the test harness. That should allow for tests using multiple harnesses, while keeping a clear distinction of what is unit tested and what is system tested.

[nordicjm]

-1 on system subfolder, have it /tests/subsys/X

[eivindj-nordic]

Other suggestions are welcome. Point is that we want to have one folder with unit tests and one folder with system tests. There will be more system tests to follow, not only in subsys.
Corrected codeowners file alignment.

[nordicjm]

the unit tests go in /tests/unit the other tests go in /tests just as they do in ncs and zephyr

[carlescufi]

Agreed, I would suggest we follow the tests/unit/ and tests/ to match NCS and Zephyr

[eivindj-nordic]

Ok, I was not aware of that pattern in NCS. On a side note, it seems that some cleanup is required in sdk-nrf (search CONFIG_UNITY=y). The tests are not within that folder.

Will update the PR.