Integrate pytest-subtests

[nicoddemus]

This PR copies the files from pytest-subtests and fixes a number of integration issues, code simplification, --lf/--sw support, etc.

Why?

Pros

• subtests is a standard unittest feature, so it makes sense for pytest to support it as well.
• Provides a simple alternative to parametrization.
• Adds the ability to generate new test cases at runtime during the test execution, which is not possible with parametrization.
• While it can exist as an external plugin, it requires many hacks, and better report integration is not easily achievable without core integration (off the top of my head: issues with terminal reporting, last failed and stepwise support, among others).

Cons

• Adds another maintenance burden to the core.

---

This PR 👍 votes and no 👎 votes, so we proceeded with the integration.

Fixes #1367, Fixes pytest-dev/pytest-subtests#71

[Pierre-Sassoulas]

I agree that this feature should be in pytest core because it's a feature in unittest and pytest should aim to be a drop in replacement for unittest (plus the original issue have 40 👍 and no 👎 at time of writing, overwhelming popular support in my book).

[RonnyPfannschmidt]

I recall we need to fix marking the owning test case as failed if one subtest fails

[RonnyPfannschmidt]

But yeah i want to see this in

[webknjaz]

Sounds reasonable

[nicoddemus]

Ready for an initial review folks.

[bluetech]

Nice to see this happening.

I ran out of time for the review for today, so didn't really get to the implementation parts, but already have some comments so submitting a partial review.

[RonnyPfannschmidt]

we might want to bikeshed a api around sub-section, subtests and parameter loops a little - not necessarily for doing right now - but for setting up a roadmap

[nicoddemus]

I was hoping to get this feature into 9.0, if possible.

[RonnyPfannschmidt]

we absolutely want this in 9.0

we shouldnt change the api to ensure compat with existing users
we shoul dplan for mkaing some things nicer

we should try to ensure lastfailed does rerun tests with failed subtests before releasing 9.0

[nicoddemus]

we should try to ensure lastfailed does rerun tests with failed subtests before releasing 9.0

I will add a test, good call.

[bluetech]

Unfortunately ran out of time to do a full review again, but left some comments.

I also noticed two things while I was testing it:

---

If some or all subtests fail, the parent test is still PASSED, is this intentional?

---

For some reason the errors are shown as "Captured log call", seems wrong as there are not log calls in my test.

---

Regarding the name SUBPASS etc., I saw that regular pass is PASSED so I wonder it shouldn't be SUBPASSED etc? (I can check the code later).

[nicoddemus]

Now the top-level test will fail with a message in case it contains failed subtests but otherwise does not contain failed assertions on its own:

__________________________________ test_foo ___________________________________
Contains 2 failed subtests

[nicoddemus]

Merged! Thanks @bluetech for the thorough review. 👍