GH-21: Add release CI (#122)

Fix GH-21 This is based on https://github.com/apache/arrow-julia/tree/main/dev/release and https://github.com/apache/arrow-adbc/tree/main/dev/release . Workflow: Cut a RC: 1. Update `PkgVersion` in `arrow/doc.go` 2. Run `dev/release/release_rc.sh` 1. `dev/release/release_rc.sh` pushes `v${version}-rc${rc}` tag 2. `.github/workflows/rc.yml` creates `apache-arrow-go-${version}.tar.gz{,.sha256,.sha512}` 3. `.github/workflows/rc.yml` uploads `apache-arrow-go-${version}.tar.gz{,.sha256,.sha512}` to GitHub Releases 4. `dev/release/release_rc.sh` downloads `apache-arrow-go-${version}.tar.gz` from GitHub Releases 5. `dev/release/release_rc.sh` signs `apache-arrow-go-${version}.tar.gz` as `apache-arrow-go-${version}.tar.gz.asc` 6. `dev/release/release_rc.sh` uploads `apache-arrow-go-${version}.tar.gz.asc` to GitHub Releases 3. Start a vote (GitHub Actions instead of https://dist.apache.org/repos/dist/dev/arrow/ is used like ADBC.) Verify a RC: 1. Run `dev/release/verify_rc.sh` Release an approved RC: 1. Run `dev/release/release.sh` 1. `dev/release/release.sh` pushes `v${version}` tag that refers that same commit ID as `v${version}-rc${rc}` 2. `dev/release/release.sh` downloads `apache-arrow-go-${version}.tar.gz{,.asc,.sha256,.sha512}` from GitHub Actions 3. `dev/release/release.sh` uploads `apache-arrow-go-${version}.tar.gz{,.asc,.sha256,.sha512}` to https://dist.apache.org/repos/dist/release/arrow 4. `dev/release/release.sh` removes old releases from https://dist.apache.org/repos/dist/release/arrow 2. Add this release to ASF's report database: https://reporter.apache.org/addrelease.html?arrow Follow-up tasks: * Add support for running integration test in the verification script * Add support for releasing a release note * Add support for creating "v${version}"'s GitHub Releases * We can copy "v${version}-rc${rc}"'s GitHub Releases and adjust it for the official release --------- Co-authored-by: Raúl Cumplido <[email protected]>
apache · Sep 17, 2024 · c6d372a · c6d372a
1 parent 98559bb
commit c6d372a
Show file tree

Hide file tree

Showing 10 changed files with 726 additions and 18 deletions.
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
@@ -18,8 +18,11 @@
 name: Lint
 on:
   push:
-    branches-ignore:
-      - 'dependabot/**'
+    branches:
+      - '**'
+      - '!dependabot/**'
+    tags:
+      - '*'
   pull_request:
 concurrency:
   group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}

diff --git a/.github/workflows/rc.yml b/.github/workflows/rc.yml
@@ -0,0 +1,135 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: RC
+on:
+  push:
+    branches:
+      - '**'
+      - '!dependabot/**'
+    tags:
+      - '*-rc*'
+  pull_request:
+concurrency:
+  group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+  cancel-in-progress: true
+permissions:
+  contents: read
+jobs:
+  archive:
+    name: Archive
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Checkout
+        uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
+        with:
+          submodules: recursive
+      - name: Prepare for tag
+        if: github.ref_type == 'tag'
+        run: |
+          version=${GITHUB_REF_NAME%-rc*}
+          version=${version#v}
+          rc=${GITHUB_REF_NAME#*-rc}
+          echo "VERSION=${version}" >> ${GITHUB_ENV}
+          echo "RC=${rc}" >> ${GITHUB_ENV}
+      - name: Prepare for branch
+        if: github.ref_type == 'branch'
+        run: |
+          version=$(grep -o '^const PkgVersion = ".*"' "arrow/doc.go" |
+            sed \
+              -e 's/^const PkgVersion = "//' \
+              -e 's/"$//')
+          rc=$(date +%Y%m%d)
+          echo "VERSION=${version}" >> ${GITHUB_ENV}
+          echo "RC=${rc}" >> ${GITHUB_ENV}
+      - name: Archive
+        run: |
+          id="apache-arrow-go-${VERSION}"
+          tar_gz="${id}.tar.gz"
+          echo "TAR_GZ=${tar_gz}" >> ${GITHUB_ENV}
+          git archive HEAD --prefix "${id}/" --output "${tar_gz}"
+          sha256sum "${tar_gz}" > "${tar_gz}.sha256"
+          sha512sum "${tar_gz}" > "${tar_gz}.sha512"
+      - name: Audit
+        run: |
+          dev/release/run_rat.sh "${TAR_GZ}"
+      - uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
+        with:
+          name: archive
+          path: |
+            apache-arrow-go-*
+  verify:
+    name: Verify
+    needs:
+      - archive
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os:
+          - macos-latest
+          - ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
+        with:
+          submodules: recursive
+      - uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
+        with:
+          name: archive
+      - name: Verify
+        run: |
+          tar_gz=$(echo apache-arrow-go-*.tar.gz)
+          version=${tar_gz#apache-arrow-go-}
+          version=${version%.tar.gz}
+          # rc isn't used with VERIFY_DOWNLOAD=0
+          if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+            rc="${GITHUB_REF_NAME#*-rc}"
+          else
+            rc=$(date +%Y%m%d)
+          fi
+          VERIFY_DEFAULT=0 dev/release/verify_rc.sh "${version}" "${rc}"
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+  upload:
+    name: Upload
+    if: github.ref_type == 'tag'
+    needs:
+      - verify
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - name: Checkout
+        uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
+        with:
+          submodules: recursive
+      - uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
+        with:
+          name: archive
+      - name: Upload
+        run: |
+          # TODO: Add support for release note
+          gh release create ${GITHUB_REF_NAME} \
+            --prerelease \
+            --title "Apache Arrow Go ${GITHUB_REF_NAME}" \
+            --verify-tag \
+            apache-arrow-go-*.tar.gz \
+            apache-arrow-go-*.tar.gz.sha*
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -18,8 +18,11 @@
 name: Test
 on:
   push:
-    branches-ignore:
-      - 'dependabot/**'
+    branches:
+      - '**'
+      - '!dependabot/**'
+    tags:
+      - '*'
   pull_request:
 concurrency:
   group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
@@ -202,15 +205,12 @@ jobs:
           go-version: ${{ matrix.go }}
           cache: true
           cache-dependency-path: go.sum
-      - name: Install bash via Homebrew
-        run: |
-          brew install bash
       - name: Build
         run: |
-          $(brew --prefix)/bin/bash ci/scripts/build.sh $(pwd)
+          ci/scripts/build.sh $(pwd)
       - name: Test
         run: |
-          $(brew --prefix)/bin/bash ci/scripts/test.sh $(pwd)
+          ci/scripts/test.sh $(pwd)
   macos-cgo:
     name: AMD64 macOS 12 Go ${{ matrix.go }} - CGO
     runs-on: macos-12
@@ -235,18 +235,16 @@ jobs:
           cache: true
           cache-dependency-path: go.sum
       - name: Brew Install Arrow and pkg-config
-        shell: bash
-        run: brew install apache-arrow pkg-config bash
-      - name: Add To pkg config path
-        shell: bash
+        run: brew install apache-arrow pkg-config
+      - name: Setup PKG_CONFIG_PATH
         run: |
           echo "PKG_CONFIG_PATH=$(brew --prefix openssl@3)/lib/pkgconfig:$PKG_CONFIG_PATH" >> $GITHUB_ENV
       - name: Build
         run: |
-          $(brew --prefix)/bin/bash ci/scripts/build.sh $(pwd)
+          ci/scripts/build.sh $(pwd)
       - name: Test
         run: |
-          $(brew --prefix)/bin/bash ci/scripts/test.sh $(pwd)
+          ci/scripts/test.sh $(pwd)
   windows:
     name: AMD64 Windows 2019 Go ${{ matrix.go }}
     runs-on: windows-2019

diff --git a/.gitignore b/.gitignore
@@ -16,6 +16,8 @@
 # under the License.
 
 .vscode
+/apache-arrow-go-*.tar.gz
+/apache-arrow-go-*.tar.gz.asc
 /apache-arrow-go.tar.gz
 /dev/release/apache-rat-*.jar
 /dev/release/filtered_rat.txt

diff --git a/arrow/doc.go b/arrow/doc.go
@@ -34,7 +34,7 @@ To build with tinygo include the noasm build tag.
 */
 package arrow
 
-const PkgVersion = "18.0.0-SNAPSHOT"
+const PkgVersion = "18.0.0"
 
 //go:generate go run _tools/tmpl/main.go -i -data=numeric.tmpldata type_traits_numeric.gen.go.tmpl type_traits_numeric.gen_test.go.tmpl array/numeric.gen.go.tmpl array/numericbuilder.gen.go.tmpl array/bufferbuilder_numeric.gen.go.tmpl
 //go:generate go run _tools/tmpl/main.go -i -data=datatype_numeric.gen.go.tmpldata datatype_numeric.gen.go.tmpl tensor/numeric.gen.go.tmpl tensor/numeric.gen_test.go.tmpl

diff --git a/ci/scripts/build.sh b/ci/scripts/build.sh
@@ -28,7 +28,7 @@ pushd "${source_dir}/arrow"
 
 : "${ARROW_GO_TESTCGO:=}"
 
-go_install_arrow_options=()
+go_install_arrow_options=(-v)
 if [[ -n "${ARROW_GO_TESTCGO}" ]]; then
   if [[ "${MSYSTEM:-}" = "MINGW64" ]]; then
     export PATH=${MINGW_PREFIX}/bin:${PATH}
@@ -38,7 +38,7 @@ if [[ -n "${ARROW_GO_TESTCGO}" ]]; then
   go_install_arrow_options+=("-tags" "assert,test,ccalloc")
 fi
 
-go install "${go_install_arrow_options[@]}" -v ./...
+go install "${go_install_arrow_options[@]}" ./...
 
 popd
 

diff --git a/dev/release/README.md b/dev/release/README.md
@@ -0,0 +1,116 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Release
+
+## Overview
+
+  1. Test the revision to be released
+  2. Increment version number in `arrow/doc.go`
+  3. Prepare RC and vote (detailed later)
+  4. Publish (detailed later)
+
+### Prepare RC and vote
+
+Run `dev/release/release_rc.sh` on a working copy of `[email protected]:apache/arrow-go` not your fork:
+
+```console
+$ git clone [email protected]:apache/arrow-go.git
+$ dev/release/release_rc.sh ${RC}
+(Send a vote email to [email protected].
+ You can use a draft shown by release_rc.sh for the email.)
+```
+
+Here is an example to release RC1:
+
+```console
+$ GH_TOKEN=${YOUR_GITHUB_TOKEN} dev/release/release_rc.sh 1
+```
+
+The argument of `release_rc.sh` is the RC number. If RC1 has a problem, we'll increment the RC number such as RC2, RC3 and so on.
+
+Requirements to run `release_rc.sh`:
+
+  * You must be an Apache Arrow committer or PMC member
+  * You must prepare your PGP key for signing
+
+If you don't have a PGP key, https://infra.apache.org/release-signing.html#generate may be helpful.
+
+Your PGP key must be registered to the followings:
+
+  * https://dist.apache.org/repos/dist/dev/arrow/KEYS
+  * https://dist.apache.org/repos/dist/release/arrow/KEYS
+
+See the header comment of them how to add a PGP key.
+
+Apache arrow committers can update them by Subversion client with their ASF account. e.g.:
+
+```console
+$ svn co https://dist.apache.org/repos/dist/dev/arrow
+$ cd arrow
+$ editor KEYS
+$ svn ci KEYS
+```
+
+### Publish
+
+We need to do the followings to publish a new release:
+
+  * Publish to apache.org
+
+Run `dev/release/release.sh` to publish to apache.org:
+
+```console
+$ GH_TOKEN=${YOUR_GITHUB_TOKEN} dev/release/release.sh ${VERSION} ${RC}
+```
+
+Here is an example to release 18.0.0 RC1:
+
+```console
+$ GH_TOKEN=${YOUR_GITHUB_TOKEN} dev/release/release.sh 18.0.0 1
+```
+
+Add the release to ASF's report database via [Apache Committee Report Helper](https://reporter.apache.org/addrelease.html?arrow).
+
+### Verify
+
+We have a script to verify a RC.
+
+You must install the following commands to use the script:
+
+  * `curl`
+  * `gpg`
+  * `shasum` or `sha256sum`/`sha512sum`
+  * `tar`
+
+You don't need to install Go. If Go doesn't exist in system, the latest Go is automatically installed only for verification.
+
+To verify a RC, run the following command line:
+
+```console
+$ dev/release/verify_rc.sh ${VERSION} ${RC}
+```
+
+Here is an example to verify the release 18.0.0 RC1:
+
+```console
+$ dev/release/verify_rc.sh 18.0.0 1
+```
+
+If the verification is successful, the message `RC looks good!` is shown.