Skip to content

GH-46075: [Release][CI] Fix binary verification #46076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Apr 15, 2025
Merged

GH-46075: [Release][CI] Fix binary verification #46076

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a6df2b6
fix regex filter for asset tuples
assignUser Apr 9, 2025
41dd3f8
format
assignUser Apr 9, 2025
5cb0007
allow testing rc workflow
assignUser Apr 9, 2025
0833ac0
add missing --repository
assignUser Apr 9, 2025
2c7558a
add missing --tag
assignUser Apr 9, 2025
6b9465f
fix windows wheel verification
assignUser Apr 9, 2025
10eb84c
fix wheels_dir
assignUser Apr 9, 2025
0b9d0f0
fix windows package type
assignUser Apr 9, 2025
d4ce91e
debug windows wheel
assignUser Apr 9, 2025
f14dcd4
add token to avoid rate limit
assignUser Apr 9, 2025
5778a55
debug
assignUser Apr 10, 2025
30f7649
revert - edit verify script to fix deb/yum pkg location
assignUser Apr 14, 2025
8312ad3
more debug
assignUser Apr 15, 2025
2f40933
use current checkout for testing deps instead of a main clone
assignUser Apr 15, 2025
a93a4e8
fix test_binary
assignUser Apr 15, 2025
aba5d14
fix windows wheel script paths
assignUser Apr 15, 2025
6566355
remove debug code
assignUser Apr 15, 2025
9187ac2
use absolute path to wheel
assignUser Apr 15, 2025
7c5b414
Revert "revert - edit verify script to fix deb/yum pkg location"
assignUser Apr 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 32 additions & 12 deletions .github/workflows/verify_rc.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,24 @@ on:
push:
tags:
- "*-rc*"
pull_request:
paths:
- ".github/workflows/verify_rc.yml"
workflow_dispatch:
inputs:
rc_tag:
description: "Tag of the rc to verify"
type: string
required: true


permissions:
contents: read

env:
TEST_DEFAULT: "0"
VERBOSE: "1"
RC_TAG: "${{ inputs.rc_tag || github.event_name == 'pull_request' && 'apache-arrow-20.0.0-rc0' || github.ref_name }}"

jobs:
apt:
Expand All @@ -46,9 +57,9 @@ jobs:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Run
run: |
package_id=${GITHUB_REF_NAME%-rc*}
package_id=${RC_TAG%-rc*}
version=${package_id#apache-arrow-}
rc=${GITHUB_REF_NAME#*-rc}
rc=${RC_TAG#*-rc}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about creating a job that detects target information something like the following?

diff --git a/.github/workflows/verify_rc.yml b/.github/workflows/verify_rc.yml
index fe46ae6f23..cc1eb6e6bf 100644
--- a/.github/workflows/verify_rc.yml
+++ b/.github/workflows/verify_rc.yml
@@ -30,7 +30,37 @@ env:
   VERBOSE: "1"
 
 jobs:
+  target:
+    runs-on: ubuntu-latest
+    timeout: 5
+    outputs:
+      package_id: ${{ steps.detect.outputs.package_id }}
+      version: ${{ steps.detect.outputs.version }}
+      rc: ${{ steps.detect.outputs.rc }}
+    steps:
+      - name: Detect
+        id: detect
+        run: |
+          case "${GITHUB_EVENT_NAME}" in
+            workflow_dispatch)
+              tag="${{ input.rc_tag }}"
+              ;;
+            pull_request)
+              tag="$(detect the latest RC tag)"
+              ;;
+            *)
+              tag="${GITHUB_REF_NAME}"
+              ;;
+          esac
+          package_id=${tag%-rc*}
+          version=${package_id#apache-arrow-}
+          rc=${tag#*-rc}
+          echo "package_id=${package_id}" >> "${GITHUB_OUTPUT}"
+          echo "version=${version}" >> "${GITHUB_OUTPUT}"
+          echo "rc=${rc}" >> "${GITHUB_OUTPUT}"
+
   apt:
+    needs: target
     name: APT
     runs-on: ${{ matrix.runs-on }}
     timeout-minutes: 30
@@ -41,15 +71,15 @@ jobs:
           - ubuntu-latest
           - ubuntu-24.04-arm
     env:
+      PACKAGE_ID: ${{ needs.target.outputs.package_id }}
+      RC: ${{ needs.target.outputs.rc }}
       TEST_APT: "1"
+      VERSION: ${{ needs.target.outputs.version }}
     steps:
       - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       - name: Run
         run: |
-          package_id=${GITHUB_REF_NAME%-rc*}
-          version=${package_id#apache-arrow-}
-          rc=${GITHUB_REF_NAME#*-rc}
-          dev/release/verify-release-candidate.sh ${version} ${rc}
+          dev/release/verify-release-candidate.sh ${VERSION} ${RC}
 
   binary:
     name: Binary

dev/release/verify-release-candidate.sh ${version} ${rc}

binary:
Expand All @@ -61,9 +72,9 @@ jobs:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Run
run: |
package_id=${GITHUB_REF_NAME%-rc*}
package_id=${RC_TAG%-rc*}
version=${package_id#apache-arrow-}
rc=${GITHUB_REF_NAME#*-rc}
rc=${RC_TAG#*-rc}
dev/release/verify-release-candidate.sh ${version} ${rc}

wheels-linux:
Expand All @@ -89,9 +100,9 @@ jobs:
run: python -m pip install -e dev/archery[docker]
- name: Prepare
run: |
package_id=${GITHUB_REF_NAME%-rc*}
package_id=${RC_TAG%-rc*}
echo "VERSION=${package_id#apache-arrow-}" >> ${GITHUB_ENV}
echo "RC=${GITHUB_REF_NAME#*-rc}" >> ${GITHUB_ENV}
echo "RC=${RC_TAG#*-rc}" >> ${GITHUB_ENV}
distro=${{ matrix.distro }}
if [ "${distro}" = "conda" ]; then
echo "SERVICE=${distro}-verify-rc" >> ${GITHUB_ENV}
Expand All @@ -102,13 +113,16 @@ jobs:
echo "$(echo ${os} | tr a-z A-Z)=${version}" >> ${GITHUB_ENV}
fi
- name: Run
env:
GH_TOKEN: ${{ github.token }}
run: |
archery docker run \
-e TEST_DEFAULT="${TEST_DEFAULT}" \
-e TEST_WHEELS="${TEST_WHEELS}" \
-e VERBOSE="${VERBOSE}" \
-e VERIFY_RC="${RC}" \
-e VERIFY_VERSION="${VERSION}" \
-e GH_TOKEN="$GH_TOKEN" \
${SERVICE}

wheels-macos:
Expand All @@ -126,10 +140,12 @@ jobs:
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Run
env:
GH_TOKEN: ${{ github.token }}
run: |
package_id=${GITHUB_REF_NAME%-rc*}
package_id=${RC_TAG%-rc*}
version=${package_id#apache-arrow-}
rc=${GITHUB_REF_NAME#*-rc}
rc=${RC_TAG#*-rc}
dev/release/verify-release-candidate.sh ${version} ${rc}

wheels-windows:
Expand All @@ -141,12 +157,14 @@ jobs:
TEST_WHEELS: "1"
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
submodules: recursive
- name: Prepare
shell: bash
run: |
package_id=${GITHUB_REF_NAME%-rc*}
package_id=${RC_TAG%-rc*}
echo "VERSION=${package_id#apache-arrow-}" >> ${GITHUB_ENV}
echo "RC=${GITHUB_REF_NAME#*-rc}" >> ${GITHUB_ENV}
echo "RC=${RC_TAG#*-rc}" >> ${GITHUB_ENV}
- uses: conda-incubator/setup-miniconda@505e6394dae86d6a5c7fbb6e3fb8938e3e863830 # v3.1.1
- name: Install System Dependencies
run: |
Expand All @@ -156,6 +174,8 @@ jobs:
shell: bash
run: ci/scripts/download_tz_database.sh
- name: Run verification
env:
GH_TOKEN: ${{ github.token }}
shell: cmd
run: |
dev/release/verify-release-candidate-wheels.bat %VERSION% %RC%
Expand All @@ -176,7 +196,7 @@ jobs:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Run
run: |
package_id=${GITHUB_REF_NAME%-rc*}
package_id=${RC_TAG%-rc*}
version=${package_id#apache-arrow-}
rc=${GITHUB_REF_NAME#*-rc}
rc=${RC_TAG#*-rc}
dev/release/verify-release-candidate.sh ${version} ${rc}
Loading
Loading