Add Arm option to building packages on Jenkins #38990
Open
+160
−71
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jclarkeSTFC
added
macOS Only
jclarkeSTFC
force-pushed
the
arm_build_package_from_branch
branch
8 times, most recently
from
bfa1ea0 to
f5e7c26
Compare
|
pre-commit.ci run |
jclarkeSTFC
force-pushed
the
arm_build_package_from_branch
branch
2 times, most recently
from
25ac9c7 to
755f7a3
Compare
|
This is blocked because the version of I've also made a PR conda-forge/staged-recipes#29358 to add |
jclarkeSTFC
force-pushed
the
arm_build_package_from_branch
branch
from
1893306 to
d4b27e7
Compare
jclarkeSTFC
force-pushed
the
arm_build_package_from_branch
branch
from
b8d2332 to
332c204
Compare
The osx-arm64 platform can only run on Arm macOS machines, and it will compile native Arm code instead of cross-compiling for x86_64. The platform is used for specifying the cmake preset, which is why I added the osx-arm64-ci preset.
We need the option to be able to specify the platform so Arm Macs can either build for osx-64 or osx-arm64. In conda-buildscript we take the platform from the cmake preset, which should then work with existing CI. To activate Arm we then just use the osx-arm64-ci preset.
Allow for passing in the target platform, e.g. linux-64 to the setup_mamba method. The `clean` parameter is now no longer optional to simplify matters. Where setup_mamba was called without the clean parameter it now specifies `false` (the old default).
Using the old SDK results in a linker error: ld: warning: ignoring file /opt/MacOSX10.10.sdk/usr/lib/libSystem.tbd, missing required architecture arm64 in file /opt/MacOSX10.10.sdk/usr/ lib/libSystem.tbd (2 slices) macOS 10.10 was released in 2014. The first version of macOS with Arm support was 11 (Big Sur), released in 2020.
The archive_env_logs script needs to know the platform in order to extract the log files.
We need the target platform in the script that creates the dmg for macOS, since it could be either osx-64 or osx-arm64. Previously it was always osx-64 so could be hard-coded. I also added the platform name to the dmg file name to avoid a clash when making both the osx-64 and osx-arm64 standalones.
When archiving the environments, we get the platform from the conda environment, so we need it to be the correct platform. This only currently applies when building for osx-64 an a osx-arm64 machine.
Version 1.0.0b15 of quickbayes does not have an ARM build, so the pip install will fail in that case. Version 1.0.1b0 is a noarch package, so works on ARM, but it has a pin on numpy<2, so if we pip install that then numpy in the standalone will be downgraded from 2.* to 1.*. Hence we do not install quickbayes into the macOS ARM standalone package.
jclarkeSTFC
force-pushed
the
arm_build_package_from_branch
branch
from
332c204 to
9e2f83f
Compare
|
Package build all platforms: |
|
Build and test, then package build, all platforms: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the option to build a native package for Arm Macs. It will be listed as a new option called
osx-arm64, with the existing option remaining asosx-64.The
osx-arm64platform can only run on Arm macOS machines, and it will compile native Arm code instead of cross-compiling for x86_64. The platform is used for specifying the cmake preset, which is why I added theosx-arm64-cipreset.Needed for #38973
Fixes #38995
In various places the desired platform is passed as an argument. We cannot simply use the native platform, because any Arm build machine could be either targeting
osx-64orosx-arm64, so we need to know what was selected.I've simplified some code in
mamba-utils, and deleted the bit that was forcingosx-64on Macs.When archiving the environment logs we get the platform from the configuration of the current conda environment. This is because that script is called from the build script in the conda recipe, which doesn't know about options specified in Jenkins, so it seemed like the best way of getting that information.
The
jemalloclibrary is pinned to version5.2.0for Linux, but5.2.1is the first version with an Arm build. Luckily we don't needjemallocon macOS anyway.Version
1.0.0b15ofquickbayesdoes not have an ARM build, so thepip installwhen creating the standalone will fail in that case. Version1.0.1b0is anoarchpackage, so works on ARM, but it has a pin onnumpy<2, so if wepip installthatthen
numpyin the standalone will be downgraded from 2.* to 1.*. Hence we do not installquickbayesinto the macOS ARM standalone package.To test:
Core_Team_test_pipelinewith theosx-arm64option and theosx-64option.Reviewer
Please comment on the points listed below (full description).
Your comments will be used as part of the gatekeeper process, so please comment clearly on what you have checked during your review. If changes are made to the PR during the review process then your final comment will be the most important for gatekeepers. In this comment you should make it clear why any earlier review is still valid, or confirm that all requested changes have been addressed.
Code Review
Functional Tests
Does everything look good? Mark the review as Approve. A member of
@mantidproject/gatekeeperswill take care of it.Gatekeeper
If you need to request changes to a PR then please add a comment and set the review status to "Request changes". This will stop the PR from showing up in the list for other gatekeepers.