Skip to content

Commit 8495f97

Browse files
DilumAluthgeDilumAluthge
authored
Merge branch 'main' into ib/prioritize_release
2 parents 2ebaf8d + e684c24 commit 8495f97

20 files changed

+114
-50
lines changed

‎.buildkite/hooks/post-checkout

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@ if [[ "${DISABLE_UPSTREAM_CACHE}" != "true" ]]; then
2222
mkdir -p "$(dirname "${UPSTREAM_CACHE}")"
2323
git clone --mirror "${UPSTREAM_URL}" "${UPSTREAM_CACHE}"
2424
else
25+
echo "Prune cache for ${UPSTREAM_URL}"
26+
rm -f "${UPSTREAM_CACHE}/gc.log"
27+
git -C "${UPSTREAM_CACHE}" prune
2528
echo "Update cache for ${UPSTREAM_URL}"
2629
git -C "${UPSTREAM_CACHE}" fetch
2730
fi
@@ -35,7 +38,16 @@ rm -rf ..?* .[!.]* *
3538
UPSTREAM_GITREF="$(buildkite-agent meta-data get --default "origin/${UPSTREAM_BRANCH}" BUILDKITE_JULIA_VERSION)"
3639

3740
if [[ "${DISABLE_UPSTREAM_CACHE}" != "true" ]]; then
38-
git clone --dissociate --reference "${UPSTREAM_CACHE}" "${UPSTREAM_URL}" "./"
41+
# For some unknown reason, the `--dissociate` flag (which causes a `git repack -a -d`
42+
# to be run after the clone is set) causes a spurious error on macOS:
43+
# fatal: unable to parse commit 1a2b3c3d
44+
# warning: Clone succeeded, but checkout failed.
45+
# You can inspect what was checked out with 'git status'
46+
# and retry with 'git restore --source=HEAD :/'
47+
# This occurs when cloning manually as well, however the `git reset --hard`
48+
# below tends to work. So we just ignore the error with `|| true` here,
49+
# and leave it to `git reset` to fail if there is actually a problem.
50+
git clone --dissociate --reference "${UPSTREAM_CACHE}" "${UPSTREAM_URL}" "./" || true
3951
else
4052
git clone "${UPSTREAM_URL}" "./"
4153
fi

‎devdocs/macos-developer-id.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
## How to see if the certificate is expired
2+
3+
The main symptom will be that the upload job will be failing. The unlock keychain step prints the status of the certificate and it will print something like (CSSMERR_TP_CERT_EXPIRED) if the certificate is expired.
4+
5+
## Update expired Apple Developer ID
6+
7+
To replace the certificate you will need first a MacOS machine and to emit a new certificate.
8+
9+
1. To replace an expired Apple certificate, clone julia-buildkite repo and clone https://github.com/staticfloat/cryptic-buildkite-plugin in its root.
10+
2. You can decrypt by running `make decrypt`.
11+
3. Get the macos_codesigning.keychain file in the `secrets` directory and add it to your local keychains with the Keychain Access app.
12+
4. From that app delete the old certificate and add the new one (it's a `.cer` file).
13+
5. Test the certificate by right clicking and running both the general evaluation and the codesigning one.
14+
6. Update the identity in `MACOS_CODESIGN_IDENTITY` (You can find the identity by doing `security find-identity -p codesigning $(PATH_TO_KEYCHAIN)/macos_codesigning.keychain` ).
15+
7. You can also test it by running the codesign.sh script in this repo with `./utilities/macos/codesign.sh --keychain ./secrets/macos_codesigning.keychain --identity $(NEW_IDENTITY) ./test` with some executable.
16+
8. Afterward reencrypt the keychain by running `./cryptic-buildkite-plugin/bin/encrypt_file --private-key=$(INSERT_PRIVATE_KEY) --repo-key=$(INSERT_REPO_KEY) ./secrets/macos_codesigning.keychain`
17+
9. Finally sign the repo with `make sign_treehashes`
18+
19+
The `security` cli app is also useful for debugging and managing keychains. You can find more information about it by running `man security` in the terminal.

‎pipelines/main/launch_signed_jobs.yml.signature

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1 @@
1-
Salted__��,�`jsl@�dڇ��^��A{� ��8%�)��Y
2-
t�2��t�`�}B�ȮriD�ą�* U�����2S��\�
3-
�aBSe��H�ss!
1+
Salted__m{KRF�{7y��-u�Ʉ�6����a�OU)4U�[����@3R �d�+��?D���ɴPU�� �GϦ%��9��!+��F�=��

‎pipelines/main/launch_unsigned_jobs.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,11 @@ steps:
105105
bash .buildkite/utilities/arches_pipeline_upload.sh \
106106
.buildkite/pipelines/main/platforms/test_macos.arches \
107107
.buildkite/pipelines/main/platforms/test_macos.yml
108+
# Launch FreeBSD test jobs
109+
GROUP="Test" \
110+
bash .buildkite/utilities/arches_pipeline_upload.sh \
111+
.buildkite/pipelines/main/platforms/test_freebsd.arches \
112+
.buildkite/pipelines/main/platforms/test_freebsd.yml
108113
# Launch windows test jobs
109114
GROUP="Test" \
110115
bash .buildkite/utilities/arches_pipeline_upload.sh \

‎pipelines/main/launch_upload_jobs.yml.signature

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
Salted__�xR���R�7餖�=����>S�Y`�j�Wr'�&��*VM�D����u-b��ɖ?+yϭ�#��ɑcZP��p�[4vE�u��
1+
Salted__�>a���{䮆s�T�U�>0Q+��V�/�^���� ��Ɉe�<t���H�x¨*�3:�`��A:��W�� N����Lq�a�E>

‎pipelines/main/platforms/build_windows.arches

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# OS TRIPLET ARCH DOCKER_ARCH MAKE_FLAGS TIMEOUT DOCKER_TAG
2-
windows x86_64-w64-mingw32 x86_64 x86_64 VERBOSE=1 . v7.2
3-
windows i686-w64-mingw32 x86_64 i686 VERBOSE=1 . v7.2
2+
windows x86_64-w64-mingw32 x86_64 x86_64 VERBOSE=1 . v7.10
3+
windows i686-w64-mingw32 x86_64 i686 VERBOSE=1 . v7.10
44

55
# These special lines allow us to embed default values for the columns above.
66
# Any column without a default mapping here will simply substitute a `.` to the empty string

‎pipelines/main/platforms/test_freebsd.arches

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
# OS TRIPLET ARCH TIMEOUT
2+
freebsd x86_64-unknown-freebsd x86_64 .
23

34
# These special lines allow us to embed default values for the columns above.
45
# Any column without a default mapping here will simply substitute a `.` to the empty string

‎pipelines/main/platforms/test_freebsd.soft_fail.arches

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
# OS TRIPLET ARCH TIMEOUT
2-
freebsd x86_64-unknown-freebsd x86_64 .
32

43
# These special lines allow us to embed default values for the columns above.
54
# Any column without a default mapping here will simply substitute a `.` to the empty string

‎pipelines/main/platforms/upload_windows.arches

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# OS TRIPLET DOCKER_TAG TIMEOUT
2-
windows x86_64-w64-mingw32 v5.44 .
3-
windows i686-w64-mingw32 v5.44 .
2+
windows x86_64-w64-mingw32 v7.10 .
3+
windows i686-w64-mingw32 v7.10 .
44

55
# These special lines allow us to embed default values for the columns above.
66
# Any column without a default mapping here will simply substitute a `.` to the empty string

‎pipelines/main/platforms/upload_windows.yml

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,9 @@ steps:
2121
variables:
2222
- AWS_ACCESS_KEY_ID="U2FsdGVkX184v87+NPs3j9r/JoIuOrYt4/Z4wnRdklnY17NP8C8AMZvWYLJfT9t1"
2323
- AWS_SECRET_ACCESS_KEY="U2FsdGVkX1+qptnxR/Mo5jZdH8OQfflRPiQBEhjgZIiTpn8KNCJYh/Cb8xxaUWazlcM9ceOlo0InDubL+J8zdg=="
24-
- WINDOWS_CODESIGN_PASSWORD="U2FsdGVkX1+wiqniliFf7YWn1f/Y2rppITsOMvLGgK8n+GYWXWJH+POojLhEwU75"
24+
- AZURE_TENANT_ID="U2FsdGVkX1+c2HbeSTq1c0fEep5riO3+PSQCiovdRT5akVpyy5hC311cXUCUB+A7ivIezAdJ7oz3dt19t60iuw=="
25+
- AZURE_CLIENT_ID="U2FsdGVkX195QeT0ywNdXSa2ctMb9DFXyFVo0zG0/GtZUw95PTycqSZ+qEwXZbRZhGxXjGMVzcP2XZ0f/OuNnQ=="
26+
- AZURE_CLIENT_SECRET="U2FsdGVkX1/905nz82Iy5toukRXEO8QS2etwfFCocofgnpBB62wJRDrWClPCNEX6ICo0jQzgoSbwDr2+C1ETSA=="
2527
files:
2628
- .buildkite/secrets/tarball_signing.gpg
2729
- .buildkite/secrets/windows_codesigning.pfx
@@ -42,9 +44,11 @@ steps:
4244
# Have to include this for `buildkite-agent` to work:
4345
- "BUILDKITE_AGENT_ACCESS_TOKEN"
4446
# Have to include these for codesigning and uploading
45-
- "WINDOWS_CODESIGN_PASSWORD"
4647
- "AWS_ACCESS_KEY_ID"
4748
- "AWS_SECRET_ACCESS_KEY"
49+
- "AZURE_TENANT_ID"
50+
- "AZURE_CLIENT_ID"
51+
- "AZURE_CLIENT_SECRET"
4852
# These control where we upload, important when running on julia-buildkite
4953
- "S3_BUCKET"
5054
- "S3_BUCKET_PREFIX"

‎pipelines/scheduled/coverage/coverage.yml.signature

0 Bytes
Binary file not shown.

‎pipelines/scheduled/launch_signed_jobs.yml.signature

0 Bytes
Binary file not shown.

‎pipelines/scheduled/launch_upload_jobs.yml.signature

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
1-
Salted__*b��]��#�At����t�k�ă��ڟ��W�EE�j�E�/YV�Yl"�T`�����&8�]�s�YG�I@ ^]�0���E�ys
1+
Salted__��C;7X�G��D����B
2+
�� �r{�bP�Ш�|�N ����c�ɰ�ۑ
3+
����"!�0CC7�����g�J(�T��v���

‎secrets/macos_codesigning.keychain.encrypted

7.89 KB
Binary file not shown.

‎utilities/build_julia.sh

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,19 @@ MFLAGS+=( "-j${JULIA_CPU_THREADS}")
3838

3939
# Add a few default flags to our make flags:
4040
MFLAGS+=( "VERBOSE=1" )
41-
MFLAGS+=( "TAGGED_RELEASE_BANNER=Official https://julialang.org/ release" )
41+
# Taken from https://stackoverflow.com/a/4024263
42+
verlte() {
43+
printf '%s\n' "$1" "$2" | sort -C -V
44+
}
45+
verlt() {
46+
! verlte "$2" "$1"
47+
}
48+
if verlt "1.12" "$(cat VERSION)"; then
49+
MFLAGS+=( "TAGGED_RELEASE_BANNER=Official https://julialang.org release" )
50+
else
51+
# Keep trailing slash for compatability. The slash was removed in 1.12 with https://github.com/JuliaLang/julia/pull/53978
52+
MFLAGS+=( "TAGGED_RELEASE_BANNER=Official https://julialang.org/ release" )
53+
fi
4254
MFLAGS+=( "JULIA_CPU_TARGET=${JULIA_CPU_TARGET}" )
4355

4456
# Finish off with any extra make flags from the `.arches` file

‎utilities/test_julia.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ if [[ "${OS}" == "macos" ]]; then
3333
echo "--- [mac] Codesigning"
3434
.buildkite/utilities/macos/codesign.sh "${JULIA_INSTALL_DIR}"
3535
echo "--- [mac] Update checksums for stdlib cachefiles after codesigning"
36-
${JULIA_INSTALL_DIR}/bin/julia .buildkite/utilities/update_stdlib_pkgimage_checksums.jl
36+
JULIA_DEBUG=all ${JULIA_INSTALL_DIR}/bin/julia .buildkite/utilities/update_stdlib_pkgimage_checksums.jl
3737
fi
3838

3939

‎utilities/update_stdlib_pkgimage_checksums.jl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,4 +99,4 @@ for dir in readdir(stdlib_cache_dir, join = true)
9999
end
100100

101101
# check stdlibs caches are valid
102-
Base.isprecompiled(Base.PkgId(Base.UUID("44cfe95a-1eb2-52ea-b672-e2afdf69b78f"), "Pkg")) || error()
102+
Base.isprecompiled(Base.PkgId(Base.UUID("8bb1440f-4735-579b-a4ab-409b98df4dab"), "DelimitedFiles")) || error()

‎utilities/upload_julia.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ if [[ "${BUILDKITE_PULL_REQUEST}" == "false" ]]; then
2424

2525
# This _must_ be an absolute path
2626
KEYCHAIN_PATH="$(pwd)/.buildkite/secrets/macos_codesigning.keychain"
27-
MACOS_CODESIGN_IDENTITY="2053E9292809B66582CA9F042B470C0929340362"
27+
MACOS_CODESIGN_IDENTITY="E7CEA0DEF3BD5B83E9C50D9318845219097F43A0"
2828

2929
# Add the keychain to the list of keychains to search, then unlock it
3030
security -v list-keychains -s -d user "${KEYCHAIN_PATH}"
@@ -81,15 +81,15 @@ if [[ "${BUILDKITE_PULL_REQUEST}" == "false" ]]; then
8181
/F"${UPLOAD_FILENAME}" \
8282
/O"$(cygpath -w "$(pwd)")" \
8383
/Dsign=true \
84-
/Smysigntool="bash.exe '${codesign_script}' --certificate='${certificate}' \$f" \
84+
/Smysigntool="bash.exe '${codesign_script}' \$f" \
8585
"$(cygpath -w "${iss_file}")"
8686

8787
# Add the `.exe` to our upload targets
8888
UPLOAD_EXTENSIONS+=( "exe" )
8989

9090
# Next, directly codesign every executable file in the install dir
9191
echo "--- [windows] Codesign everything in the install directory"
92-
"${codesign_script}" --certificate="${certificate}" "${JULIA_INSTALL_DIR}"
92+
"${codesign_script}" "${JULIA_INSTALL_DIR}"
9393

9494
echo "--- [windows] Update checksums for stdlib cachefiles"
9595
${JULIA_INSTALL_DIR}/bin/julia .buildkite/utilities/update_stdlib_pkgimage_checksums.jl

‎utilities/windows/codesign.sh

Lines changed: 27 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,18 @@
33

44
set -euo pipefail
55

6+
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
7+
8+
DLIB_DEFAULT_PATH='C:\Program Files\TrustedSigning\bin\x64\Azure.CodeSigning.Dlib.dll'
9+
DLIB_PATH="${DLIB_DEFAULT_PATH}"
10+
METADATA_JSON_PATH="$(cygpath -w ${SCRIPT_DIR}/codesign_metadata.json)"
11+
612
usage() {
7-
echo "Usage: $0 --certificate=<path> --password=<password> <target>"
13+
echo "Usage: $0 [--dlib-path=<path>] <target>"
814
echo
915
echo "Parameter descriptions:"
1016
echo
11-
echo " key: A '.pfx' file that contains the codesigning certificate"
12-
echo
13-
echo " password: The password to unlock the given '.pfx' file."
17+
echo " dlib-path: The path to the Trusted Signing .dlib file (defaults to ${DLIB_DEFAULT_PATH})."
1418
echo
1519
echo " target: A file or directory to codesign (must come last!)"
1620
}
@@ -26,22 +30,13 @@ fi
2630

2731
while [ "$#" -gt 1 ]; do
2832
case "${1}" in
29-
--certificate)
30-
CERT_PATH="$2"
33+
--dlib-path)
34+
DLIB_PATH="$2"
3135
shift
3236
shift
3337
;;
34-
--certificate=*)
35-
CERT_PATH="${1#*=}"
36-
shift
37-
;;
38-
--password)
39-
CERT_PASSWORD="$2"
40-
shift
41-
shift
42-
;;
43-
--password=*)
44-
CERT_PASSWORD="${1#*=}"
38+
--dlib-path=*)
39+
DLIB_PATH="${1#*=}"
4540
shift
4641
;;
4742
*)
@@ -51,32 +46,33 @@ while [ "$#" -gt 1 ]; do
5146
;;
5247
esac
5348
done
49+
DLIB_PATH="$(cygpath -w "${DLIB_PATH}")"
5450

55-
# We tend to receive this via an environment variable on CI, so as to
56-
# not print it out when `make` is run in verbose mode
57-
CERT_PASSWORD="${CERT_PASSWORD:-${WINDOWS_CODESIGN_PASSWORD}}"
51+
if [[ -z "${AZURE_TENANT_ID:-}" ]] ||
52+
[[ -z "${AZURE_CLIENT_ID:-}" ]] ||
53+
[[ -z "${AZURE_CLIENT_SECRET:-}" ]]; then
54+
echo "ERROR: Missing AZURE_* secret variables!" >&2
55+
exit 1
56+
fi
5857

59-
if [[ ! -f "${CERT_PATH}" ]]; then
60-
echo "ERROR: Certificate path '${CERT_PATH}' does not exist!" >&2
58+
if [[ ! -f "${DLIB_PATH}" ]]; then
59+
echo "ERROR: No Trusted Signing dlib found at '${DLIB_PATH}'" >&2
6160
exit 1
6261
fi
63-
CERT_PATH="$(cygpath -w "$(abspath "${CERT_PATH}")")"
62+
6463

6564
# We will try to codesign, using multiple timestamping servers in case one is down
6665
SERVERS=(
67-
"http://timestamp.digicert.com/?alg=sha1"
68-
"http://timestamp.globalsign.com/scripts/timstamp.dll"
69-
"http://timestamp.comodoca.com/authenticode"
66+
"http://timestamp.acs.microsoft.com"
67+
"http://timestamp.digicert.com"
7068
"http://tsa.starfieldtech.com"
7169
)
7270
NUM_RETRIES=3
7371

7472
function do_codesign() {
7573
for retry in $(seq 1 ${NUM_RETRIES}); do
7674
for SERVER in ${SERVERS[@]}; do
77-
# Note that we're using SHA1 signing here, because that's what our certificate supports.
78-
# In the future, we may be able to upgrade to SHA256.
79-
if MSYS2_ARG_CONV_EXCL='*' signtool sign /debug /fd certHash /f "${CERT_PATH}" /p "${CERT_PASSWORD}" /t "${SERVER}" "$1"; then
75+
if MSYS2_ARG_CONV_EXCL='*' signtool sign /q /fd SHA256 /tr "${SERVER}" /td SHA256 /dlib "${DLIB_PATH}" /dmdf "${METADATA_JSON_PATH}" "$1"; then
8076
return 0
8177
fi
8278
done
@@ -89,7 +85,7 @@ function do_codesign() {
8985
# This codesign script only works on files
9086
if [ -f "${1}" ]; then
9187
# If we're codesigning a single file, directly invoke codesign on that file
92-
echo "Codesigning file ${1} with certificate ${CERT_PATH}"
88+
echo "Codesigning file ${1}"
9389
do_codesign "${1}"
9490
elif [ -d "${1}" ]; then
9591
# Create a fifo to communicate from `find` to `while`
@@ -105,7 +101,7 @@ elif [ -d "${1}" ]; then
105101
# This while loop reads in from the fifo, and invokes `do_codesign`,
106102
# but it does so in a background task, so that the codesigning can
107103
# happen in parallel. This speeds things up by a few seconds.
108-
echo "Codesigning dir ${1} with certificate ${CERT_PATH}"
104+
echo "Codesigning dir ${1}"
109105
NUM_CODESIGNS=0
110106
while IFS= read -r -d '' exe_file; do
111107
do_codesign "${exe_file}" &

‎utilities/windows/codesign_metadata.json

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"Endpoint": "https://eus.codesigning.azure.net/",
3+
"CodeSigningAccountName": "juliahubwincertsaccount",
4+
"CertificateProfileName": "JuliaHubWinCert",
5+
"ExcludeCredentials": [
6+
"WorkloadIdentityCredential",
7+
"ManagedIdentityCredential",
8+
"SharedTokenCacheCredential",
9+
"VisualStudioCredential",
10+
"VisualStudioCodeCredential",
11+
"AzureCliCredential",
12+
"AzurePowerShellCredential",
13+
"AzureDeveloperCliCredential",
14+
"InteractiveBrowserCredential"
15+
]
16+
}

0 commit comments

Comments
 (0)