Unified CI Release - 2023-02-08 (#466)

* Add option to skip `32-bit` go test (#412)

Introduce an option to configure `go-test` to allow completely skipping
`32-bit` tests.

Fixes #388

* Run at most 1 dispatch job per ref (#414)

* fix: check if git tag returns any results (#415)

* make go generate print the commands it executs (#440)

* use pull_request_target event for release-check workflow (#295)

* Revert "include cross-package coverage in codecov"

* Revert "Revert "include cross-package coverage in codecov""

* Make automerge a reusable workflow (#260)

* move automerge from template to workflows

* make automerge reusable and use it from new automerge template

* pass parent job name to reusable automerge

* check github actions yamls (#272)

* check github actions yamls

* make yaml linter happy about go-test

* mention VS Code YAML extension in the readme

* add info about other YAML checking extensions

* make yaml checker more generic

* use validate-yaml-schema action from mainline (#277)

* upgrade lewagon/wait-on-check-action to v1.1.1 (#278)

* always add a version.json file if it doesn't exist (#281)

* fix go-test runner string

* check if tag already exists in release-check (#287)

* allow specifying custom PATH for 386 arch (#289)

* use pull_request_target event for release-check workflow

* add comment on missing version.json

* chore: revert release checker path trigger change

* chore: add footnote when non-docs files are modified with the release

* fix: prev version calculation

* feat: allow configuring custom go-test runners (#443)

* feat: allow configuring custom go-test runners

* docs: update readme to include info on configuration variables

* feat: allow skipping go-test on certain OSes (#455)

* feat: standarise JSON config reading

* feat: allow skipping go-test on certain OSes

* fix: go-test conditional

* chore: show config after extracting it

* chore: udpate actions and go modules (#458)

* fix: source read-config from next for now

* simplify Go version upgrade procedure (#280)

* chore: simplify Go version upgrade procedure

* chore: add default for the go-version input of release-check

* Update .github/actions/copy-workflow-go/action.yml

* Update configs/README.md

Co-authored-by: Laurent Senta <[email protected]>

---------

Co-authored-by: Laurent Senta <[email protected]>

* Go through all the workflows and clean them up ahead of the next major release (#462)

* chore: clean up deprecated set-output

* chore: do not use substitution inside run

* chore: do not use substitution in if

* chore: skip env var brakets where possible

* fix: env var substitution

* fix: double toJSON

* Update templates/.github/workflows/js-test-and-release.yml

* feat: create gh releases in release-check/releaser workflows (#456)

* feat: create gh releases in release-check/releaser workflows

* fix: fill expr in release check workflow

* fix: add missing gh token in release check

* fix: add missing prev version env var in release check workflow

* fix: release check in release check

* chore: clean up obsolete step from releaser

* fix: step outputs in release workflows

* fix: labels in releaser

* fix: action gh release

* update go version to 1.20.x (#463)

* update go version to 1.20.x

* fix: go 1.20 upgrade

* Revert "fix: go 1.20 upgrade"

This reverts commit ceb72ef.

* clean up where source ref was set to next (#464)

* perform self-review before final release

---------

Co-authored-by: Masih H. Derkani <[email protected]>
Co-authored-by: Marten Seemann <[email protected]>
Co-authored-by: Laurent Senta <[email protected]>

Author: 4 people

.github/actions/copy-workflow-go/action.yml

@@ -4,23 +4,34 @@ description: Copy workflow steps specific to go
 runs:
   using: "composite"
   steps:
+    # GitHub Actions Expressions do not support last item access/array length retrieval
+    - id: go
+      run: echo "version=$(jq -r '.[-1]' <<< '${{ toJSON(matrix.cfg.go.versions) }}')" >> $GITHUB_OUTPUT
+      shell: bash
     - uses: actions/setup-go@v3
       with:
         # This should be the same Go version we use in the go-check workflow.
         # go mod tidy, go vet, staticcheck and gofmt might behave differently depending on the version.
-        go-version: "1.19.x"
+        go-version: ${{ steps.go.outputs.version }}
     - name: bump go.mod go version if needed
       uses: protocol/[email protected]
+      env:
+        VERSION: ${{ matrix.cfg.go.versions[0] }}
       with:
         working-directory: ${{ env.TARGET_REPO_DIR }}
         run: |
           # We want our modules to support two Go versions at a time.
-          # As of August 2022, Go 1.19 is the latest stable.
           # go.mod's Go version declares the language version being used.
           # As such, it has to be the minimum of all Go versions supported.
-          # Bump this every six months, as new Go versions come out.
-          TARGET_VERSION=1.18
-          PREVIOUS_TARGET_VERSION=1.17
+          TARGET_VERSION="$VERSION"
+          TARGET_VERSION="${TARGET_VERSION%.x}"
+          TARGET_MAJOR_VERSION="${TARGET_VERSION%.[0-9]*}"
+          TARGET_MINOR_VERSION="${TARGET_VERSION#[0-9]*.}"
+          # Assumptions:
+          # - all versions are targetted incrementally
+          # - no versions are skipped
+          # - patch version is never pinned explicitly
+          PREVIOUS_TARGET_VERSION="$TARGET_MAJOR_VERSION.$(($TARGET_MINOR_VERSION-1))"
 
           # Note that the "<" comparison doesn't understand semver,
           # but it should be good enough for the foreseeable future.
@@ -50,18 +61,18 @@ runs:
             # As of Go 1.19 io/ioutil is deprecated
             # We automate its upgrade here because it is quite a widely used package
             while read file; do
-              sed -i 's/ioutil.NopCloser/io.NopCloser/' "${file}";
-              sed -i 's/ioutil.ReadAll/io.ReadAll/' "${file}";
+              sed -i 's/ioutil.NopCloser/io.NopCloser/' "$file";
+              sed -i 's/ioutil.ReadAll/io.ReadAll/' "$file";
               # ReadDir replacement might require manual intervention (https://pkg.go.dev/io/ioutil#ReadDir)
-              sed -i 's/ioutil.ReadDir/os.ReadDir/' "${file}";
-              sed -i 's/ioutil.ReadFile/os.ReadFile/' "${file}";
-              sed -i 's/ioutil.TempDir/os.MkdirTemp/' "${file}";
-              sed -i 's/ioutil.TempFile/os.CreateTemp/' "${file}";
-              sed -i 's/ioutil.WriteFile/os.WriteFile/' "${file}";
-              sed -i 's/ioutil.Discard/io.Discard/' "${file}";
+              sed -i 's/ioutil.ReadDir/os.ReadDir/' "$file";
+              sed -i 's/ioutil.ReadFile/os.ReadFile/' "$file";
+              sed -i 's/ioutil.TempDir/os.MkdirTemp/' "$file";
+              sed -i 's/ioutil.TempFile/os.CreateTemp/' "$file";
+              sed -i 's/ioutil.WriteFile/os.WriteFile/' "$file";
+              sed -i 's/ioutil.Discard/io.Discard/' "$file";
             done <<< "$(find . -type f -name '*.go')"
 
-            go install golang.org/x/tools/cmd/goimports@v0.1.12
+            go install golang.org/x/tools/cmd/goimports@v0.5.0
             goimports -w .
 
             git add .
@@ -70,7 +81,7 @@ runs:
             fi
           fi
     - name: go mod tidy (on initial workflow deployment)
-      if: ${{ env.INITIAL_WORKFLOW_DEPLOYMENT == 1 }}
+      if: env.INITIAL_WORKFLOW_DEPLOYMENT == 1
       uses: protocol/[email protected]
       with:
         working-directory: ${{ env.TARGET_REPO_DIR }}
@@ -81,7 +92,7 @@ runs:
             git commit -m "run go mod tidy"
           fi
     - name: gofmt -s (on initial workflow deployment and on new Go version)
-      if: ${{ env.INITIAL_WORKFLOW_DEPLOYMENT == 1 || env.GO_VERSION_BUMP == 1}}
+      if: env.INITIAL_WORKFLOW_DEPLOYMENT == 1 || env.GO_VERSION_BUMP == 1
       working-directory: ${{ env.TARGET_REPO_DIR }}
       shell: bash
       run: |

.github/actions/read-config/action.yml

@@ -0,0 +1,22 @@
+name: read config
+description: Reads workflow config
+
+outputs:
+  json:
+    description: JSON config
+    value: ${{ steps.config.outputs.json }}
+
+runs:
+  using: "composite"
+  steps:
+    - id: config
+      run: |
+        eof="EOF$RANDOM"
+        path="${GITHUB_WORKFLOW_REF%.yml@*}"
+        path="./${path#*/*/}-config.json"
+        printf "json<<$eof\n%s\n$eof" "$(cat "$path" || echo '{}')" >> $GITHUB_OUTPUT
+      shell: bash
+    - env:
+        CONFIG: ${{ steps.config.outputs.json }}
+      run: echo "$CONFIG"
+      shell: bash

.github/workflows/add-label-by-query.yml

@@ -22,7 +22,7 @@ jobs:
     steps:
       - name: Add label by query
         env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GITHUB_TOKEN: ${{ github.token }}
           QUERY: ${{ github.event.inputs.query }}
           LABEL: ${{ github.event.inputs.label }}
           DRY_RUN: ${{ github.event.inputs.dry-run }}

.github/workflows/automerge.yml

@@ -21,17 +21,20 @@ jobs:
         fetch-depth: 0
     - name: Check if we should automerge
       id: should-automerge
+      env:
+        BASE_REF: ${{ github.event.pull_request.base.ref }}
+        HEAD_SHA: ${{ github.event.pull_request.head.sha }}
       run: |
-        for commit in $(git rev-list --first-parent origin/${{ github.event.pull_request.base.ref }}..${{ github.event.pull_request.head.sha }}); do
+        for commit in $(git rev-list --first-parent origin/$BASE_REF..$HEAD_SHA); do
           committer=$(git show --format=$'%ce' -s $commit)
           echo "Committer: $committer"
           if [[ "$committer" != "[email protected]" ]]; then
             echo "Commit $commit wasn't committed by web3-bot, but by $committer."
-            echo "::set-output name=status::false"
+            echo "status=false" >> $GITHUB_OUTPUT
             exit
           fi
         done
-        echo "::set-output name=status::true"
+        echo "status=true" >> $GITHUB_OUTPUT
   automerge:
     needs: automerge-check
     runs-on: ubuntu-latest
@@ -40,16 +43,16 @@ jobs:
     if: github.event.pull_request.user.login == 'web3-bot' && needs.automerge-check.outputs.status == 'true'
     steps:
     - name: Wait on tests
-      uses: lewagon/wait-on-check-action@752bfae19aef55dab12a00bc36d48acc46b77e9d # v1.1.1
+      uses: lewagon/wait-on-check-action@3a563271c3f8d1611ed7352809303617ee7e54ac # v1.2.0
       with:
         ref: ${{ github.event.pull_request.head.sha }}
-        repo-token: ${{ secrets.GITHUB_TOKEN }}
+        repo-token: ${{ github.token }}
         wait-interval: 10
         running-workflow-name: '${{ inputs.job }} / ${{ github.job }}' # the name of the check for this job
     - name: Merge PR
-      uses: pascalgn/automerge-action@741c311a47881be9625932b0a0de1b0937aab1ae # v0.13.1
+      uses: pascalgn/automerge-action@eb68b061739cb9d81564f8e812d0b3c45f0fb09a # v0.15.5
       env:
-        GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
+        GITHUB_TOKEN: "${{ github.token }}"
         MERGE_LABELS: ""
         MERGE_METHOD: "squash"
         MERGE_DELETE_BRANCH: true

.github/workflows/check-3rd-party.yml

@@ -15,16 +15,18 @@ jobs:
       - id: set-matrix
         run: |
           TARGETS=$(find . -type f -name "*.yml" | sed "s|^\./||" | grep -v workflow-templates/header.yml | jq -R -s -c 'split("\n")[:-1]')
-          echo "::set-output name=targets::$TARGETS"
+          echo "targets=$TARGETS" >> $GITHUB_OUTPUT
   check:
     needs: [ matrix ]
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
-        file: ${{ fromJson(needs.matrix.outputs.targets) }}
+        file: ${{ fromJSON(needs.matrix.outputs.targets) }}
     name: ${{ matrix.file }}
     steps:
       - uses: actions/checkout@v3
       - name: Run check
-        run: .github/workflows/check-3rd-party.sh ${{ matrix.file }}
+        env:
+          FILE: ${{ matrix.file }}
+        run: .github/workflows/check-3rd-party.sh $FILE

.github/workflows/check-yaml.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
-      - uses: nwisbeta/validate-yaml-schema@c3734e647d2a3beb98b9132330067e900fdbd1a2 # v2.0.0
+      - uses: pl-strflt/validate-yaml-schema@21116b89ed801672cb7354717c0d049974f265b0
         with:
           yamlSchemasJson: |
             {

.github/workflows/copy-workflow.yml

@@ -15,7 +15,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        cfg: ${{ fromJson(github.event.inputs.targets) }}
+        cfg: ${{ fromJSON(github.event.inputs.targets) }}
       max-parallel: 10
     env:
       TARGET_REPO_DIR: "target-repo"
@@ -53,40 +53,40 @@ jobs:
     - name: determine files to add
       # By setting the environment variable, it's possible to programmatically add / modify this list.
       # See https://github.com/protocol/.github/blob/38135c75e47839623bf9b2748275d8c6167a8fa8/.github/workflows/copy-workflow.yml#L163-L168 for an example, how we used to make use of this.
-      run: |
-        files=${{ toJson(toJson(matrix.cfg.files)) }}
-        extra_files=${{ toJson(toJson(matrix.cfg.extra_files)) }}
-        files=$(echo -e "$files" "$extra_files" | jq -nc '[inputs] | add')
-        echo "FILES=$files" >> $GITHUB_ENV
+      env:
+        FILES: |
+          ${{ toJSON(matrix.cfg.files) }}
+          ${{ toJSON(matrix.cfg.extra_files) }}
+      run: echo "FILES=$(jq -nc '[inputs] | add' <<< "$FILES")" >> $GITHUB_ENV
     - name: is initial workflow deployment
       # INITIAL_WORKFLOW_DEPLOYMENT=1 iff none of the files in the target repository exist yet
       run: |
         initial_workflow_deployment=1
-        for f in $(jq -r '.[]' <<< ${{ toJson(env.FILES) }}); do
+        for f in $(jq -r '.[]' <<< "$FILES"); do
           if [[ -f $TARGET_REPO_DIR/$f ]]; then
             initial_workflow_deployment=0
             break
           fi
         done
         echo "INITIAL_WORKFLOW_DEPLOYMENT=$initial_workflow_deployment" >> $GITHUB_ENV
     - name: remove Travis (on initial workflow deployment)
-      if: ${{ env.INITIAL_WORKFLOW_DEPLOYMENT == 1 }}
+      if: env.INITIAL_WORKFLOW_DEPLOYMENT == 1
       working-directory: ${{ env.TARGET_REPO_DIR }}
       run: |
         if [[ -f .travis.yml ]]; then
           git rm .travis.yml
           git commit -m "disable Travis"
         fi
     - name: remove CircleCI (on initial workflow deployment)
-      if: ${{ env.INITIAL_WORKFLOW_DEPLOYMENT == 1 }}
+      if: env.INITIAL_WORKFLOW_DEPLOYMENT == 1
       working-directory: ${{ env.TARGET_REPO_DIR }}
       run: |
         if [[ -d .circleci ]]; then
           git rm -r .circleci
           git commit -m "disable CircleCI"
         fi
     - name: remove gx (on initial workflow deployment)
-      if: ${{ env.INITIAL_WORKFLOW_DEPLOYMENT == 1 }}
+      if: env.INITIAL_WORKFLOW_DEPLOYMENT == 1
       working-directory: ${{ env.TARGET_REPO_DIR }}
       run: |
         if [[ -d .gx ]]; then
@@ -111,7 +111,7 @@ jobs:
           }
         TEMPLATE_ENGINE: s#\$\{\{\{\s*(.*?)\s*\}\}\}#`jq -cjn 'env.CONTEXT | fromjson.$1'`#ge
       run: |
-        for f in $(jq -r '.[]' <<< ${{ toJson(env.FILES) }}); do
+        for f in $(jq -r '.[]' <<< "$FILES"); do
           echo -e "\nProcessing $f."
           # add DO NOT EDIT header
           tmp=$(mktemp)
@@ -144,7 +144,7 @@ jobs:
       working-directory: ${{ env.TARGET_REPO_DIR }}
       run: echo "NEEDS_UPDATE=$(git rev-list HEAD...origin/$(git rev-parse --abbrev-ref HEAD) --ignore-submodules --count 2> /dev/null || echo 1)" >> $GITHUB_ENV
     - name: Force push web3-bot/sync branch
-      if: ${{ env.NEEDS_UPDATE != 0 }}
+      if: env.NEEDS_UPDATE != 0
       working-directory: ${{ env.TARGET_REPO_DIR }}
       run: |
         git checkout -B web3-bot/sync

.github/workflows/create-prs.yml

@@ -28,16 +28,16 @@ jobs:
           done
           failed=()
           for target in ${targets[@]}; do
-            echo "Processing ${target}"
-            base="$(gh api "/repos/${target}" --jq '.default_branch')"
+            echo "Processing $target"
+            base="$(gh api "/repos/$target" --jq '.default_branch')"
             # checks if a PR needs to be created
-            if [[ "$(gh api -X GET "/repos/${target}/compare/${base}...${{ env.PR_BRANCH }}" --jq '.status')" == 'ahead' ]]; then
-              if [[ "$(gh api -X GET "/repos/${target}/pulls" -f head="$(echo "${target}" | cut -d/ -f1):${{ env.PR_BRANCH }}" -f base="$base" --jq 'length')" != '0' ]] ; then
+            if [[ "$(gh api -X GET "/repos/$target/compare/$base...$PR_BRANCH" --jq '.status')" == 'ahead' ]]; then
+              if [[ "$(gh api -X GET "/repos/$target/pulls" -f head="$(echo "$target" | cut -d/ -f1):$PR_BRANCH" -f base="$base" --jq 'length')" != '0' ]] ; then
                 echo "The PR already exists. Skipping."
                 continue
               fi
             else
-              echo "The branch does not exist or has diverged from ${base}. Skipping."
+              echo "The branch does not exist or has diverged from $base. Skipping."
               continue
             fi
             # tries to create a PR in target
@@ -47,7 +47,7 @@ jobs:
             pr_create_cooldown_in_seconds=1
             # max cumulative sleep time - 68.25 minutes
             while true; do
-              if result="$(gh api "/repos/$target/pulls" -f title="${{ env.PR_TITLE }}" -f head="${{ env.PR_BRANCH }}" -f base="$base")"; then
+              if result="$(gh api "/repos/$target/pulls" -f title="$PR_TITLE" -f head="$PR_BRANCH" -f base="$base")"; then
                 echo "Successfully created a PR for '$target' ($pr_create_attempt/$pr_create_max_attempts)"
                 echo "Sleeping for $pr_create_cooldown_in_seconds seconds before creating a next one"
                 sleep $pr_create_cooldown_in_seconds

.github/workflows/dispatch.yml

@@ -9,6 +9,10 @@ on:
     branches: [ master, testing ]
   workflow_dispatch:
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 env:
   # Number of repositories in a batch.
   # Matrix jobs within a copy workflow run are run in parallel.
@@ -48,8 +52,8 @@ jobs:
             targets+=($(jq -c ".repositories[] | $defaults + ." $config))
             echo "::endgroup::"
           done
-          batches=$(jq -sc '[. | _nwise(${{ env.MAX_REPOS_PER_WORKFLOW }})] | to_entries' <<< "${targets[@]}")
-          echo "::set-output name=batches::$batches"
+          batches=$(jq -sc "[. | _nwise($MAX_REPOS_PER_WORKFLOW)] | to_entries" <<< "${targets[@]}")
+          echo "batches=$batches" >> $GITHUB_OUTPUT
   dispatch:
     needs: [ matrix ]
     name: Dispatch copy workflow(batch ${{ matrix.cfg.key }})
@@ -69,7 +73,7 @@ jobs:
         # The triggered copy workflow runs use that final array as their matrix.
         # Since max-parallel here is 1, we'll end up with at most max-parallel from copy workflow + 1 parallel jobs.
         # 20 is the upper limit on parallel jobs on a free plan.
-        cfg: ${{ fromJson(needs.matrix.outputs.batches) }}
+        cfg: ${{ fromJSON(needs.matrix.outputs.batches) }}
       max-parallel: 1
     env:
       GITHUB_TOKEN: ${{ secrets.WEB3BOT_GITHUB_TOKEN }}
@@ -78,31 +82,36 @@ jobs:
     steps:
       - id: dispatch
         name: Dispatch copy workflow
+        env:
+          TARGETS: ${{ toJSON(matrix.cfg.value) }}
         run: |
           start_date="$(date +%s)"
-          # 2 toJson calls are needed to turn the array of targets into a string
-          echo '{"targets":${{ toJson(toJson(matrix.cfg.value)) }}}' | jq -c '.' | gh workflow run "${{ env.WORKFLOW_YML }}" --ref "${{ github.ref }}" --repo "${{ env.WORKFLOW_REPO }}" --json
-          echo "::set-output name=start_date::$start_date"
+          gh workflow run "$WORKFLOW_YML" --ref "$GITHUB_REF" --repo "$WORKFLOW_REPO" --field "targets=$TARGETS"
+          echo "start_date=$start_date" >> $GITHUB_OUTPUT
       - id: run
         name: Wait for copy workflow run to start
+        env:
+          START_DATE: ${{ steps.dispatch.outputs.start_date }}
         run: |
           # checks every 3 seconds until the most recent copy workflow run's created_at is later than this job's start_date
           while sleep 3; do
-            run="$(gh api "/repos/${{ env.WORKFLOW_REPO }}/actions/workflows/${{ env.WORKFLOW_YML }}/runs?per_page=1" --jq '.workflow_runs[0]')"
+            run="$(gh api "/repos/$WORKFLOW_REPO/actions/workflows/$WORKFLOW_YML/runs?per_page=1" --jq '.workflow_runs[0]')"
             # nothing to check if no copy workflow run was returned
             if [[ ! -z "$run" ]]; then
               run_start_date="$(date --date="$(jq -r '.created_at' <<< "$run")" +%s)"
-              if [[ "$run_start_date" > "${{ steps.dispatch.outputs.start_date }}" ]]; then
-                echo "::set-output name=id::$(jq -r '.id' <<< "$run")"
+              if [[ "$run_start_date" > "$START_DATE" ]]; then
+                echo "id=$(jq -r '.id' <<< "$run")" >> $GITHUB_OUTPUT
                 break
               fi
             fi
           done
       - name: Wait for copy workflow run to complete
+        env:
+          RUN_ID: ${{ steps.run.outputs.id }}
         run: |
           # delays checking copy workflow's run status to save on GH API requests
-          sleep ${{ env.WORKFLOW_COMPLETION_CHECK_DELAY }}
+          sleep $WORKFLOW_COMPLETION_CHECK_DELAY
 
           # checks every 3 seconds until the copy workflow run's status is completed
           # redirects the stdout to /dev/null because it is very chatty
-          gh run watch "${{ steps.run.outputs.id }}" --repo "${{ env.WORKFLOW_REPO }}" > /dev/null
+          gh run watch "$RUN_ID" --repo "$WORKFLOW_REPO" > /dev/null