Task: Review Lucene.Net.TestFramework for Missing Features, Incomplete Features and Inconsistencies

[NightOwl888]

Is there an existing issue for this?

• I have searched the existing issues

Task description

While it is presumed that we have enough of the test framework ported for the 4.8.0 release, we don't currently have an inventory of the features that are not ported or only partially working. For example, there are properties like LuceneTestCase.SuiteFailureMarker that are public but currently do not function that should either be made internal or functional before the release. There are also "after suite" cleanup features that are not ported (#898) that seem like they may be useful or even critical for fixing issues like #271.

For informational purposes, here are the open issues for the test framework at the time of this writing:

Known Test Framework Incomplete Features

• Port RandomizedTesting Test Runner #264
• Make assertions into a 1st class feature in the test framework #306
• Task: Finish [SuppressTempFileChecks] attribute functionality #898

Known Test Framework Bugs/Inconsistencies

• [Slow] attribute runs the code in BeforeClass() and AfterClass() even when the tests:slow property is set to false #739
• Fix test name reporting when test is in a base class #923
• Change failed test message to use lucene.testsettings.json instead of .runsettings #999
• Upgrade NUnit to 4.x #1001
• Task: Rename BeforeClass() to OneTimeSetUp() and AfterClass() to OneTimeTearDown() to match NUnit Conventions #1016

---

This task is to analyze the test framework to determine which features exist that we don't have. It will take some analysis to work out where a feature begins and ends, but once that is determined, we should use it to generate a list of missing features. From that list, we should then open an issue for each feature (assuming there isn't already an issue for it) so we can prioritize it and assign it a milestone.

We should also be on the lookout for features that are incomplete or inconsistent and open issues for those, as well.

If there are any notes that come from this analysis that could be helpful for us to decide on how difficult it is to port a feature or how valuable it is, please add them to the corresponding issue.

NOTE: The main places to look are in LuceneTestCase and the missing classes in the Lucene.Net.Util namespace (i.e. "TestRule" classes).

NOTE: Some of the randomized-testing runner classes were ported to Lucene.Net.TestFramework (at least at one point), so if there are types that are from that library, they can be excluded from the features list because they are covered by #264. We have a substitute for the randomized-testing runner in the Support/Util folder and don't intend to port the entire library for the 4.8.0 release. It is a considerable amount of extra work to make those features general enough to work outside of the context of the test framework. Although, we may be missing tests from randomized-testing that would help to ensure our substitute is working as expected that could be considered "features".