macOS upgrade causes some build breakages

[meteorcloudy]

We recently updated macOS to 15.3.1 and Xcode 16.1 for all macOS VMs on Bazel CI.

https://buildkite.com/bazel/rules-apple-darwin/builds/10138#01958280-cdc9-4e72-85bb-7399fcdcc056

(01:03:28) ERROR: /Users/buildkite/builds/bk-macos-arm64-nn86/bazel/rules-apple-darwin/test/starlark_tests/targets_under_test/ios/BUILD:129:16: AssetCatalogCompile test/starlark_tests/targets_under_test/ios/app-intermediates/bundle_library_ios.bundle/xcassets failed: (Exit 1): xctoolrunner failed: error executing AssetCatalogCompile command (from target //test/starlark_tests/targets_under_test/ios:app)
  (cd /private/var/tmp/_bazel_buildkite/3ebd711cd99f106e0bfcf0a4dddc286c/execroot/_main && \
  exec env - \
    APPLE_SDK_PLATFORM=iPhoneSimulator \
    APPLE_SDK_VERSION_OVERRIDE=18.2 \
    PATH=/Users/buildkite/Library/Caches/bazelisk/downloads/sha256/ac72ad67f7a8c6b18bf605d8602425182b417de4369715bf89146daf62f7ae48/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/bin \
    XCODE_VERSION_OVERRIDE=16.2.0.16C5032a \
  bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/tools/xctoolrunner/xctoolrunner actool --compile '[ABSOLUTE]bazel-out/ios_sim_arm64-fastbuild-ios-sim_arm64-min12.0-applebin_ios-ST-26d4f6b9029b/bin/test/starlark_tests/targets_under_test/ios/app-intermediates/bundle_library_ios.bundle/xcassets' --platform iphonesimulator --minimum-deployment-target 12.0 --compress-pngs --target-device iphone '[ABSOLUTE]test/starlark_tests/resources/assets.xcassets')
# Configuration: 8931e66e98dfc14da7a42f9369975f695461c8159360912aca94c6ca1a76bc52
# Execution platform: @@platforms//host:host
/Users/buildkite/builds/bk-macos-arm64-nn86/bazel/rules-apple-darwin/test/starlark_tests/resources/assets.xcassets: error: No simulator runtime version from [<DVTBuildVersion 21F79>, <DVTBuildVersion 22D8075>] available to use with iphonesimulator SDK version <DVTBuildVersion 22C146>

[meteorcloudy]

@fweikert @brentleyjones

[fweikert]

The tests request Xcode 15.4, which is no longer installed on our MacOS workers. That's why CI bumps the version ("Fixed Xcode version: 15.4 -> 16.2..."), which might cause the problem. @brentleyjones is there something we need to do here, or is it a fix in rules_apple?

[brentleyjones]

Even if using Xcode 16.2, we have the simulator runtime issue. @aaronsky has some experience with that and can give more context.

[aaronsky]

For the period we're using Xcode 16.2 on the CI nodes, xcodebuild -downloadPlatform iOS will install the iOS 18.3.1 simulator runtime by default, rather than the originally distributed iOS 18.2. The rules expect by default that the SDK version (18.2) match the simulator runtime version (now 18.3, represented sometimes as 18.3.1). The best thing to do in the short term would be to add a line like this somewhere in the VM/image setup:

# ...
# somewhere after running `xcodebuild -runFirstLaunch` and `xcodebuild -downloadPlatform iOS`

xcode_short_version="$(xcodebuild -version | head -n1 | cut -d' ' -f2 | cut -d. -f1,2)"
if [ "$xcode_short_version" = "16.2" ]; then
    xcodebuild -downloadPlatform iOS -buildVersion 18.2
fi

# ...
# xcodebuild -checkFirstLaunchStatus

This was the best workaround I could find in my own environment to make Bazel, simulator_creator.py, and xcodebuild happy. I wouldn't recommend running this on every CI job, as -downloadPlatform is very slow and needs some time to finish mounting the simulator runtime disk image after it's been installed.

[meteorcloudy]

@aaronsky Where is this code? Can we fix the code to not make the assumption?

[aaronsky]

The code where this all breaks down is in these spots:

• https://github.com/bazelbuild/rules_apple/blob/master/apple/testing/default_runner/ios_xctestrun_runner.bzl#L53
• https://github.com/bazelbuild/rules_apple/blob/master/apple/testing/default_runner/simulator_creator.py#L85
• (and to a lesser extent) https://github.com/bazelbuild/rules_apple/blob/master/apple/testing/default_runner/ios_test_runner.bzl#L45

When I examined this a couple weeks ago I couldn't figure out another sensible default that could be used to replace the assumption without requiring a new field or some new functionality in xcode_version. Making os_version required on the test runner would break building across different Xcode versions.

[meteorcloudy]

I'm mostly worried that the next time we upgrade macOS this will happen again, is there a long term solution for this you have in mind?

[aaronsky]

That's a reasonable concern, and no, I don't have a long-term plan in the event Apple drops a new runtime on us out of the blue again. I agree it needs a more robust solution beyond this workaround.

[meteorcloudy]

We added

xcode_short_version="$(xcodebuild -version | head -n1 | cut -d' ' -f2 | cut -d. -f1,2)"
if [ "$xcode_short_version" = "16.2" ]; then
    xcodebuild -downloadPlatform iOS -buildVersion 18.2
fi

to our setup script and updated the VMs, can you please verify if it works now?

[meteorcloudy]

Looks like rules_apple is still red: https://buildkite.com/bazel/rules-apple-darwin/builds/10157

@aaronsky Can you try to fix this from rules_apple side? I don't know what else we could do on the infra side and building and deploying a new VIM image is not very trivial.

[aaronsky]

There seems to still be a mismatch related to the installed simulator runtimes, but this time with the visionOS runtime. Which, as far as I'm aware, haven't received an update recently. I don't recognize runtime 22N895, but 21O5565d is the SDK that shipped alongside Xcode 15.4, and 22N799 is the SDK in Xcode 16.2.

[meteorcloudy]

@aaronsky I believe this has to be fixed from rules_apple side, so I'm closing this now.