Allow test failures for build uploads when not on release branch or tags #420

ericphanson · 2025-01-24T21:46:49Z

I propose this change because currently if tests fail, the build does not upload. I think some of the point of using nightly/pr-builds is to test things manually, and potentially to debug failing tests and issues like that. Therefore, I don't think uploads should be gated on tests passing.

DilumAluthge · 2025-01-24T23:00:34Z

We do want to gate it for releases. Can we make the behavior different for nightlies vs releases?

This reverts commit 9c8fd9e.

ericphanson · 2025-01-24T23:34:22Z

ah I see. I am having trouble figuring out how to do that. I think we want the conditional (build.branch =~ /^release-/) || (build.tag =~ /^v/) or similar somewhere, but I can't figure out how to get a conditional dependency, or a conditional allowed-failure... 🤔

ericphanson · 2025-01-25T00:01:54Z

I gave it a try, but I don't know if it will work -- in particular, I don't know if ENV interpolation works like this for allow_failures

staticfloat · 2025-01-29T15:55:44Z

If the concern is that you want to be able to download a build of Julia that is failing tests so you can inspect it, you can download it from the buildkite interface in the Artifacts tab. I don't think there's much use uploading these to S3, is there?

IanButterworth · 2025-01-29T16:25:47Z

Sometimes there are PRs that don't yet have all the tests figured out, but would be good for people to try out. The --trim PR was one of those.

giordano · 2025-01-29T16:39:26Z

I don't think there's much use uploading these to S3, is there?

My guess is that this would be useful for juliaup? I'm not sure where it downloads the tarball from, but if it's S3 that'd be a discriminant here

ericphanson · 2025-01-29T23:43:52Z

Yeah, my gripe here is I wanted to use the Pkg apps feature before JuliaLang/julia#57187 was merged, but I couldn't use the juliaup PR feature for it since the macos CI had failed. It was confusing initially as well bc juliaup reports it can't be installed, but I saw the PR is there and didn't understand that uploads were gated on CI passing.

staticfloat · 2025-01-30T15:27:41Z

Fair enough, the juliaup feature makes this much more useful.

DilumAluthge · 2025-01-30T15:30:31Z

I don't think this is ready to merge yet? We need to make it so that release branches and tags DO gate on tests, but nightlies and PRs do NOT gate on tests.

It looks like this PR attempts to implement that, but something about the implementation is causing CI to fail.

DilumAluthge · 2025-01-30T15:32:26Z

Also needs treehash re-signing.

DilumAluthge · 2025-01-30T15:38:54Z

fatal: pipeline interpolation of "upload_linux.yml" failed: interpolating remaining fields: $ALLOW_TEST_FAILURES: not set

ericphanson · 2025-01-30T21:49:24Z

looks like I missed a codepath for the allowed-to-fail upload jobs, should be covered now

DilumAluthge · 2025-01-30T22:10:37Z

Before we re-sign the pipelines, could you add comments that explain the purpose of the logic (i.e. that we want to gate on the tests in release branches and tags, but we don't want to gate on the tests in master/nightlies/prs).

ericphanson · 2025-02-01T16:59:17Z

added

Copilot

Pull Request Overview

This PR modifies the CI/CD pipeline to allow test failures during build uploads for non-release branches and non-version tags, enabling nightly and PR builds to be uploaded to S3 even when tests fail.

Updates upload pipeline steps to conditionally allow test failures based on branch/tag type
Adds logic to set ALLOW_TEST_FAILURES environment variable based on whether the build is on a release branch or version tag
Modifies all platform upload configurations (Windows, macOS, Linux, FreeBSD) to respect the new test failure policy

Reviewed Changes

Copilot reviewed 8 out of 10 changed files in this pull request and generated 2 comments.

File	Description
pipelines/main/launch_upload_jobs.yml	Adds conditional logic to allow test failures for non-release builds
pipelines/main/platforms/upload_*.yml	Updates dependency configuration to allow test step failures when permitted
*.signature files	Updated cryptographic signatures for modified pipeline files

pipelines/main/launch_upload_jobs.yml

IanButterworth · 2025-07-31T19:33:31Z

@ericphanson @DilumAluthge lets merge this, if you agree it's still in good shape?

DilumAluthge

Two requests:

Rename variable name to make it more clear that this is specifically just affecting whether or not we upload - it doesn't affect anything else.
Ideally, I'd like to have a small GitHub Actions CI job that runs a Bash script to just test that our Bash syntax for pattern matching is correct, and that it gives the expected output when the branch is release-something or the tag is vSOMETHING.

Once those two are done, this is good to merge from my POV.

DilumAluthge · 2025-07-31T19:48:28Z

pipelines/main/launch_upload_jobs.yml

+          # allow test-failures for non-release/tags so that nightly & PR builds
+          # can be uploaded to s3 even if tests are not passing
+          # (allowing for usage with juliaup)
+          export ALLOW_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] || [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" || echo "true")


I haven't actually verified myself that this pattern matching is correct, so ideally someone would just do some quick tests in Bash locally before we merge.

Or alternatively, we can have a quick GitHub Actions CI job to check miscellaneous things such as this. E.g. just a quick Bash-based CI job that puts in various values of BUILDKITE_BRANCH and BUILDKITE_TAG, and tests that the output ALLOW_TEST_FAILURES is correct.

The latter is probably easier, because we can do it in CI instead of relying on people to manually test things locally.

DilumAluthge · 2025-07-31T19:48:47Z

pipelines/main/launch_upload_jobs.yml

+          # allow test-failures for non-release/tags so that nightly & PR builds
+          # can be uploaded to s3 even if tests are not passing
+          # (allowing for usage with juliaup)
+          export ALLOW_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] || [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" || echo "true")


Same as above - would be nice to quickly test in CI (in a small GitHub Actions CI job).

DilumAluthge · 2025-07-31T19:52:04Z

pipelines/main/launch_upload_jobs.yml

+          # allow test-failures for non-release/tags so that nightly & PR builds
+          # can be uploaded to s3 even if tests are not passing
+          # (allowing for usage with juliaup)
+          export ALLOW_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] || [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" || echo "true")


Also, can we rename this something else? In Buildkite, "allow fail" (or "soft fail") means something different - it means failures show up as a red warning sign (not a red X), and failures don't block Test from being green.

Here, we have a different meaning, so can we make the variable name more unambiguous?

Suggested change

export ALLOW_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] || [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" || echo "true")

export UPLOAD_EVEN_IF_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] || [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" || echo "true")

DilumAluthge · 2025-07-31T19:52:09Z

pipelines/main/platforms/upload_freebsd.yml

          # Wait for the tester to finish
-          - "test_${TRIPLET?}"
+          - step: "test_${TRIPLET?}"
+            allow_failure: "${ALLOW_TEST_FAILURES?}"


Suggested change

allow_failure: "${ALLOW_TEST_FAILURES?}"

allow_failure: "${UPLOAD_EVEN_IF_TEST_FAILURES?}"

DilumAluthge · 2025-07-31T19:52:13Z

pipelines/main/platforms/upload_linux.yml

          # Wait for the tester to finish
-          - "test_${TRIPLET?}"
+          - step: "test_${TRIPLET?}"
+            allow_failure: "${ALLOW_TEST_FAILURES?}"


Suggested change

allow_failure: "${ALLOW_TEST_FAILURES?}"

allow_failure: "${UPLOAD_EVEN_IF_TEST_FAILURES?}"

DilumAluthge · 2025-07-31T19:52:18Z

pipelines/main/platforms/upload_macos.yml

          # Wait for the tester to finish
-          - "test_${TRIPLET?}"
+          - step: "test_${TRIPLET?}"
+            allow_failure: "${ALLOW_TEST_FAILURES?}"


Suggested change

allow_failure: "${ALLOW_TEST_FAILURES?}"

allow_failure: "${UPLOAD_EVEN_IF_TEST_FAILURES?}"

DilumAluthge · 2025-07-31T19:52:22Z

pipelines/main/platforms/upload_windows.yml

          # Wait for the tester to finish
-          - "test_${TRIPLET?}"
+          - step: "test_${TRIPLET?}"
+            allow_failure: "${ALLOW_TEST_FAILURES?}"


Suggested change

allow_failure: "${ALLOW_TEST_FAILURES?}"

allow_failure: "${UPLOAD_EVEN_IF_TEST_FAILURES?}"

Upload after build without waiting for tests

9c8fd9e

IanButterworth requested review from DilumAluthge and staticfloat January 24, 2025 21:47

Revert "Upload after build without waiting for tests"

92a6532

This reverts commit 9c8fd9e.

try to do conditional allowed failures

04bfe0c

ericphanson changed the title ~~Upload after build without waiting for tests~~ Allow test failures for build uploads when not on release branch or tags Jan 25, 2025

staticfloat approved these changes Jan 30, 2025

View reviewed changes

DilumAluthge added the needs treehash re-signing label Jan 30, 2025

sign treehashes

020a601

DilumAluthge removed the needs treehash re-signing label Jan 30, 2025

add ALLOW_TEST_FAILURES in second place

1497415

DilumAluthge added the needs treehash re-signing label Jan 30, 2025

add comments

6c9d34f

IanButterworth added 2 commits July 31, 2025 15:27

Merge branch 'main' into pr/420

7996b54

sign treehashes

1ed1a42

IanButterworth requested a review from Copilot July 31, 2025 19:28

Copilot AI reviewed Jul 31, 2025

View reviewed changes

pipelines/main/launch_upload_jobs.yml Show resolved Hide resolved

pipelines/main/launch_upload_jobs.yml Show resolved Hide resolved

IanButterworth removed the needs treehash re-signing label Jul 31, 2025

DilumAluthge reviewed Jul 31, 2025

View reviewed changes

	export ALLOW_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] \|\| [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" \|\| echo "true")
	export UPLOAD_EVEN_IF_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] \|\| [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" \|\| echo "true")

	allow_failure: "${ALLOW_TEST_FAILURES?}"
	allow_failure: "${UPLOAD_EVEN_IF_TEST_FAILURES?}"

Allow test failures for build uploads when not on release branch or tags #420

Are you sure you want to change the base?

Allow test failures for build uploads when not on release branch or tags #420

Uh oh!

Conversation

ericphanson commented Jan 24, 2025

Uh oh!

DilumAluthge commented Jan 24, 2025

Uh oh!

ericphanson commented Jan 24, 2025

Uh oh!

ericphanson commented Jan 25, 2025

Uh oh!

staticfloat commented Jan 29, 2025

Uh oh!

IanButterworth commented Jan 29, 2025

Uh oh!

giordano commented Jan 29, 2025

Uh oh!

ericphanson commented Jan 29, 2025

Uh oh!

staticfloat commented Jan 30, 2025

Uh oh!

DilumAluthge commented Jan 30, 2025

Uh oh!

DilumAluthge commented Jan 30, 2025

Uh oh!

DilumAluthge commented Jan 30, 2025

Uh oh!

ericphanson commented Jan 30, 2025

Uh oh!

DilumAluthge commented Jan 30, 2025

Uh oh!

ericphanson commented Feb 1, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

IanButterworth commented Jul 31, 2025

Uh oh!

DilumAluthge left a comment

Choose a reason for hiding this comment

Uh oh!

DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants