Unable to start test session when using XCTestObservationCenter (Xcode 13)

[cyberowl]

New Issue Checklist

• Updated fastlane-plugin-test_center to the latest version
• I read the README.md
• I reviewed the example(s) for the action(s) I am using
• I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
• I have reviewed the Discussions forum to see my question has already been addressed.

If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any
gesture: https://github.com/sponsors/lyndsey-ferguson. 😍

Steps to reproduce

1. Declare class that implements XCTestObservation protocol (see XCTObservationCenter)

@objc
final class SimpleObserver: NSObject {
    override init() {
        super.init()
        XCTestObservationCenter.shared.addTestObserver(self)
    }
}

extension SimpleObserver: XCTestObservation {
    func testBundleWillStart(_ testBundle: Bundle) {}
    func testBundleDidFinish(_ testBundle: Bundle) {}
}

2. Edit Info.plist: add new key/value

<key>NSPrincipalClass</key>
<string>SimpleObserver</string>

3. Run multi_scan
4. Catch exception
   *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '+[SampleObserver _identifierForSelectorString:]: unrecognized selector sent to class 0x109e98320'

Why? Because xctest_list finds symbols (testBundleWillStart / testBundleDidFinish / ...) and returns that names as test identifiers

DEBUG [2021-09-22 09:27:39.63]: Getting tests from xctestrun file at '/Users/USER/Library/Developer/Xcode/DerivedData/PROJECT/Build/Products/TESTPLAN_iphonesimulator15.0-x86_64.xctestrun'
DEBUG [2021-09-22 09:27:40.41]: Found the following tests: SimpleObserver/testBundleWillStart
SimpleObserver/testBundleDidFinish

[cyberowl]

Workaround for .xctestplan users: manually add incorrect test case's identifiers in skippedTests array

[lyndsey-ferguson]

Thanks for the workaround. multi_scan uses a very simple mechanism to find the tests....to test: it looks at the xctest binary and looks for methods named test....

For people who do not use xctestplan, here's a workaround with similar idea

    ignored_tests = ["Incorrect identifier", "Incorrect identifier 2"]
    options_override_block = lambda do |options|
      options[:only_testing] = options[:only_testing].reject { |t| ignored_tests.include? t }
      options
    end

    multi_scan(
      override_scan_options_block: options_override_block
    )

[tachtevrenidis]

having the same issue (as of Xcode 13) but for me the test identified doesn't seem to correspond to a class in my bundle

UITests/CMARBcpftYdMZHyBDBozWUKr/testCondition

[mokagio]

For what is worth, and without taking away for the great value this plugin has been delivering to retry tests from the terminal and CI, Xcode 13 introduced a Test Repetition option in the the Test Plan, which, as far as I can tell, can be used to obtain the same result as multi_scan with vanilla xcodebuild test or Fastlane's scan (run_test).

There are four modes. Here's my thoughts on when to use them:

• None. Do not repeat the tests. This is the default.
• Until Failure. Runs the tests until one fails, up to three times. Useful to discover flaky tests.
• Retry on Failure. Will retry the tests that failed up to three times. Helpful to work around flaky tests that you haven't had the time to fix yet.
• Up Until Maximum Repetition. Will run the tests three times (the number doesn't seem configurable yet) and report the success ratio. Useful to gauge the stability of your test suite.

We've enabled Retry on Failure in the WordPress UI tests, going back from multi_scan to scan, and so far it's been working well.

There is a noteworthy limitation in Xcode 13 that might still sway you towards using multi_scan. The number of repetitions is locked to 3. If for some reason you want to try more times, then multi_scan remains the only solution I know of.

[lyndsey-ferguson]

Thanks @mokagio , this is an important update which will reduce the appeal of this plugin. I think the other reason people may still want to use the plugin is the ability to parallelize tests in near-equal sized batches of tests per test runner. Xcode's parallelize tests option would parallelize based on Test Suite which wasn't as ideal as the number of tests per Test Suite can vary widely.

[lyndsey-ferguson]

I want to apologize that I have not responded to this issue. A family health issue is requiring me to narrow my focus on the essentials and I don't have the time I need to focus on the plugin more than a minute. If you can get some other contributors to this project to help with this, I would consider merging a resulting PR.

[mokagio]

Best of wishes for everything to resolve for the best as soon as possible @lyndsey-ferguson . Virtual hugs.