This repository was archived by the owner on Jan 23, 2023. It is now read-only.
This repository was archived by the owner on Jan 23, 2023. It is now read-only.
multi_scan rerunning all tests, not just failed tests. #388
Description
New Issue Checklist
- Updated
fastlane-plugin-test_centerto the latest version - I read the README.md
- I reviewed the example(s) for the action(s) I am using
- I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
- I have reviewed the Discussions forum to see my question has already been addressed.
If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any
gesture: https://github.com/sponsors/lyndsey-ferguson. 😍
Issue Description
We've been running multi_scan for several years now with good results, but we've noticed a problem in the last few days. When multi_scan makes the second and third pass, it's not recognizing the tests that were successful. Instead it reruns all the tests. We currently have 1100 tests in our test suite. Typically maybe 2 or 3 might fail during a run, but instead of just rerunning those few tests, now it is rerunning all 1100 tests.
We've noticed this is a switchover from Xcode 13 to 14, specifically 14.1 RC2. Could there be an issue with Xcode 14?
Here's the fastfile entry for multiscan:
multi_scan(
devices: ["My Mac"],
scheme: ADHOC_SCHEME,
clean: true,
code_coverage: true,
skip_slack: true,
#output_types: "xcresult",
catalyst_platform: "macos",
fail_build: true,
try_count: 3,
#batch_count: 10,
retry_test_runner_failures: true,
derived_data_path: derived_data_path,
output_directory: output_directory,
buildlog_path: "#{output_directory}/logs",
xcargs: "-allowProvisioningUpdates",
)
Complete output when running fastlane, including the stack trace and command used
Fastlane output too big.
Environment
### Stack | Key | Value | | --------------------------- | ---------------------------------------------------------- | | OS | 13.0 | | Ruby | 3.1.2 | | Bundler? | false | | Git | git version 2.37.1 (Apple Git-137.1) | | Installation Source | /opt/homebrew/Cellar/fastlane/2.210.1/libexec/bin/fastlane | | Host | macOS 13.0 (22A380) | | Ruby Lib Dir | /opt/homebrew/Cellar/ruby/3.1.2_1/lib | | OpenSSL Version | OpenSSL 1.1.1q 5 Jul 2022 | | Is contained | false | | Is homebrew | true | | Is installed via Fabric.app | false | | Xcode Path | /Applications/Xcode.app/Contents/Developer/ | | Xcode Version | 14.1 | | Swift Version | 5.7.1 | ### System Locale | Variable | Value | | | -------- | ----------- | - | | LANG | en_US.UTF-8 | ✅ | | LC_ALL | | | | LANGUAGE | | | ### fastlane files:**No Appfile found** ### fastlane gems | Gem | Version | Update-Status | | -------- | ------- | ------------- | | fastlane | 2.210.1 | ✅ Up-To-Date | ### Loaded fastlane plugins: | Plugin | Version | Update-Status | | --------------------------- | ------- | ------------- | | fastlane-plugin-s3_actions | 0.1.3 | ✅ Up-To-Date | | fastlane-plugin-appcenter | 2.0.0 | ✅ Up-To-Date | | fastlane-plugin-test_center | 3.19.1 | ✅ Up-To-Date | | fastlane-plugin-json | 1.1.0 | ✅ Up-To-Date |`./fastlane/Fastfile`
```ruby fastlane_version "2.170.0" xcversion(version: "~> 14.1") import "Config" platform :mac do desc "Runs Mac tests" lane :test do output_directory = "#{cwd}/build/test/mac" derived_data_path = "#{output_directory}/derived_data" clear_derived_data(derived_data_path: derived_data_path) multi_scan( devices: ["My Mac"], scheme: ADHOC_SCHEME, clean: true, code_coverage: true, skip_slack: true, #output_types: "xcresult", catalyst_platform: "macos", fail_build: true, try_count: 3, #batch_count: 10, retry_test_runner_failures: true, derived_data_path: derived_data_path, output_directory: output_directory, buildlog_path: "#{output_directory}/logs", xcargs: "-allowProvisioningUpdates", ) end end # More information about multiple platforms in fastlane: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Platforms.md # All available actions: https://docs.fastlane.tools/actions # fastlane reports which actions are used # No personal data is recorded. Learn more at https://github.com/fastlane/enhancer ```*generated on:* **2022-10-27**Loaded gems
| Gem | Version | | ------------------------------- | ------------ | | error_highlight | 0.3.0 | | did_you_mean | 1.6.1 | | atomos | 0.1.3 | | rexml | 3.2.5 | | CFPropertyList | 3.0.5 | | claide | 1.1.0 | | colored2 | 3.1.2 | | nanaimo | 0.3.0 | | xcodeproj | 1.22.0 | | rouge | 2.0.7 | | xcpretty | 0.3.0 | | terminal-notifier | 2.0.0 | | unicode-display_width | 1.8.0 | | terminal-table | 1.8.0 | | plist | 3.6.0 | | public_suffix | 5.0.0 | | addressable | 2.8.1 | | multipart-post | 2.0.0 | | word_wrap | 1.0.0 | | optparse | 0.1.1 | | tty-screen | 0.8.1 | | tty-cursor | 0.7.1 | | tty-spinner | 0.9.3 | | artifactory | 3.0.15 | | babosa | 1.0.4 | | colored | 1.2 | | highline | 2.0.3 | | commander | 4.6.0 | | excon | 0.93.1 | | faraday-em_http | 1.0.0 | | faraday-em_synchrony | 1.0.0 | | faraday-excon | 1.1.0 | | faraday-httpclient | 1.0.1 | | faraday-multipart | 1.0.4 | | faraday-net_http | 1.0.1 | | faraday-net_http_persistent | 1.2.0 | | faraday-patron | 1.0.0 | | faraday-rack | 1.0.0 | | faraday-retry | 1.0.3 | | ruby2_keywords | 0.0.5 | | faraday | 1.10.2 | | unf_ext | 0.0.8.2 | | unf | 0.1.4 | | domain_name | 0.5.20190701 | | http-cookie | 1.0.5 | | faraday-cookie_jar | 0.0.7 | | faraday_middleware | 1.2.0 | | fastimage | 2.2.6 | | gh_inspector | 1.1.3 | | json | 2.6.1 | | mini_magick | 4.11.0 | | naturally | 2.2.1 | | rubyzip | 2.3.2 | | security | 0.1.3 | | xcpretty-travis-formatter | 1.0.1 | | dotenv | 2.8.1 | | bundler | 2.3.11 | | simctl | 1.6.8 | | jwt | 2.5.0 | | uber | 0.1.0 | | declarative | 0.0.20 | | trailblazer-option | 0.1.2 | | representable | 3.2.0 | | retriable | 3.1.2 | | mini_mime | 1.1.2 | | memoist | 0.16.2 | | multi_json | 1.15.0 | | os | 1.1.4 | | signet | 0.17.0 | | googleauth | 1.3.0 | | httpclient | 2.8.3 | | webrick | 1.7.0 | | google-apis-core | 0.9.1 | | google-apis-playcustomapp_v1 | 0.12.0 | | google-apis-androidpublisher_v3 | 0.29.0 | | google-cloud-env | 1.6.0 | | google-cloud-errors | 1.3.0 | | google-cloud-core | 1.6.0 | | google-apis-iamcredentials_v1 | 0.15.0 | | google-apis-storage_v1 | 0.19.0 | | rake | 13.0.6 | | digest-crc | 0.6.4 | | google-cloud-storage | 1.43.0 | | emoji_regex | 3.2.3 | | jmespath | 1.6.1 | | aws-partitions | 1.649.0 | | aws-eventstream | 1.2.0 | | aws-sigv4 | 1.5.2 | | aws-sdk-core | 3.164.0 | | aws-sdk-kms | 1.58.0 | | aws-sdk-s3 | 1.116.0 | | set | 1.0.2 | | forwardable | 1.3.2 | | logger | 1.5.0 | | pathname | 0.2.0 | | shellwords | 0.1.0 | | cgi | 0.3.1 | | date | 3.2.2 | | timeout | 0.2.0 | | stringio | 3.0.1 | | securerandom | 0.1.1 | | uri | 0.11.0 | | openssl | 3.0.0 | | digest | 3.1.0 | | io-nonblock | 0.1.0 | | ipaddr | 1.2.4 | | io-wait | 0.2.1 | | zlib | 2.1.1 | | resolv | 0.2.1 | | time | 0.2.0 | | open-uri | 0.2.0 | | mutex_m | 0.1.1 | | net-protocol | 0.1.2 | | net-http | 0.2.0 | | ostruct | 0.5.2 | | english | 0.7.1 | | erb | 2.2.3 | | strscan | 3.0.1 | | abbrev | 0.1.0 | | io-console | 0.5.11 | | tempfile | 0.1.2 | | delegate | 0.2.0 | | fileutils | 1.6.0 | | tmpdir | 0.1.2 | | base64 | 0.1.1 | | singleton | 0.1.1 | | open3 | 0.1.1 | | nkf | 0.1.1 | | prettyprint | 0.1.1 | | pp | 0.3.0 | | find | 0.1.1 | | yaml | 0.2.0 | | psych | 4.0.3 | | proxies | 0.2.3 | | s3 | 0.3.29 | | fastlane-plugin-s3_actions | 0.1.3 | | fastlane-plugin-appcenter | 2.0.0 | | csv | 3.2.2 | | xctest_list | 1.2.1 | | colorize | 0.8.1 | | fastlane-plugin-test_center | 3.19.1 | | fastlane-plugin-json | 1.1.0 |