Skip to content
This repository was archived by the owner on Aug 29, 2023. It is now read-only.

Commit 0ca4a50

Browse files
galarghgalargh
authored
Prepare copy-workflow for setting up unified CI for JS repos (#239)
* move config*.json files to configs directory * rename configs/*.json * make checks aware of all the configs/*.json * turn configs into objects with repositories key * make files part of the config * move language specific copy-workflow setup to actions * make dispatch workflow support more than one config json * create fewer batches if possible * set up copy-workflow actions properly * extract defaults from config separately * bring back double toJson * use copy-workflow actions from same branch * use compact json representation for storing FILES * fix files and local actions * run actions from template repo * add missing shell property to actions * add configs README * set deploy_versioning=true for go repositories * address review comments * restore hardcoded template-repo reference * add section on config testing * fix batches creation * fix command that produces batches * fix needs update logic
1 parent a2d8ec3 commit 0ca4a50

File tree

15 files changed

+417
-308
lines changed

15 files changed

+417
-308
lines changed

‎.githooks/pre-commit‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,14 @@
11
#!/bin/bash
22

3-
tmp=$(mktemp)
4-
git show :config.json > $tmp
5-
status=0
6-
if ! output=$(.github/workflows/check-config.sh $tmp); then
7-
echo "$output"
8-
status=1
9-
fi
10-
rm $tmp
3+
for config in configs/*.json; do
4+
tmp=$(mktemp)
5+
git show :$config > $tmp
6+
status=0
7+
if ! output=$(.github/workflows/check-config.sh $tmp $config); then
8+
echo "$output"
9+
status=1
10+
fi
11+
rm $tmp
12+
done
1113

1214
exit $status

‎.github/actions/copy-workflow-go/action.yml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
name: copy workflow go
2+
description: Copy workflow steps specific to go
3+
4+
runs:
5+
using: "composite"
6+
steps:
7+
- uses: actions/setup-go@v2
8+
with:
9+
# This should be the same Go version we use in the go-check workflow.
10+
# go mod tidy, go vet, staticcheck and gofmt might behave differently depending on the version.
11+
go-version: "1.17.x"
12+
- name: bump go.mod go version if needed
13+
uses: protocol/[email protected]
14+
with:
15+
working-directory: ${{ env.TARGET_REPO_DIR }}
16+
run: |
17+
# We want our modules to support two Go versions at a time.
18+
# As of August 2021, Go 1.17 is the latest stable.
19+
# go.mod's Go version declares the language version being used.
20+
# As such, it has to be the minimum of all Go versions supported.
21+
# Bump this every six months, as new Go versions come out.
22+
TARGET_VERSION=1.16
23+
24+
# Note that the "<" comparison doesn't understand semver,
25+
# but it should be good enough for the foreseeable future.
26+
CURRENT_VERSION=$(go list -m -f {{.GoVersion}})
27+
28+
if [[ $CURRENT_VERSION < $TARGET_VERSION ]]; then
29+
echo "GO_VERSION_BUMP=1" >> $GITHUB_ENV
30+
31+
# Update the version in go.mod. This alone ensures there's a diff.
32+
go mod edit -go $TARGET_VERSION
33+
34+
# In the future, "go fix" may make changes to Go code,
35+
# such as to adapt to language changes or API deprecations.
36+
# This is largely a no-op as of Go 1.17, and that's fine.
37+
go fix ./...
38+
git add .
39+
40+
# We don't tidy, because the next step does that.
41+
# Separate commits also help with reviews.
42+
git commit -m "bump go.mod to Go $TARGET_VERSION and run go fix"
43+
fi
44+
- name: go mod tidy (on initial workflow deployment and on new Go version)
45+
if: ${{ env.INITIAL_WORKFLOW_DEPLOYMENT == 1 || env.GO_VERSION_BUMP == 1}}
46+
uses: protocol/[email protected]
47+
with:
48+
working-directory: ${{ env.TARGET_REPO_DIR }}
49+
run: |
50+
go mod tidy
51+
if ! git diff --quiet; then
52+
git add .
53+
git commit -m "run go mod tidy"
54+
fi
55+
- name: gofmt -s (on initial workflow deployment and on new Go version)
56+
if: ${{ env.INITIAL_WORKFLOW_DEPLOYMENT == 1 || env.GO_VERSION_BUMP == 1}}
57+
working-directory: ${{ env.TARGET_REPO_DIR }}
58+
shell: bash
59+
run: |
60+
gofmt -s -w .
61+
if ! git diff --quiet; then
62+
git add .
63+
git commit -m "run gofmt -s"
64+
fi

‎.github/actions/copy-workflow-versioning/action.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
name: copy workflow versioning
2+
description: Copy workflow steps specific to versioning
3+
4+
runs:
5+
using: "composite"
6+
steps:
7+
- name: add version.json file (in order to deploy versioning workflows)
8+
if: hashFiles(format('{0}/version.json', env.TARGET_REPO_DIR)) == ''
9+
working-directory: ${{ env.TARGET_REPO_DIR }}
10+
shell: bash
11+
run: |
12+
git fetch origin --unshallow # we need the entire commit history
13+
version=$(git describe --tags --abbrev=0 || true) # highest released version on current branch
14+
if [[ -n "$version" ]]; then # only deply version.json if there's at least one release
15+
printf '{"version": "%s"}' "$version" | jq . > version.json
16+
git add version.json
17+
git commit -m "add version.json file"
18+
fi

‎.github/workflows/check-3rd-party.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
11
on:
22
pull_request:
33
paths-ignore:
4-
- 'config.json'
5-
- 'config-testing.json'
4+
- 'configs/*.json'
65

76
name: Check 3rd Party
87

‎.github/workflows/check-config.sh‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,15 @@
33
set -e
44

55
file=$1
6+
source=${2:-$file}
67
entries=$(mktemp)
78
entries_sorted=$(mktemp)
89

9-
jq -r ".[].target" $file > $entries
10+
jq -r ".repositories[].target" $file > $entries
1011
sort -u $entries > $entries_sorted
1112
status=0
1213
if ! output=$(diff -y $entries $entries_sorted); then
13-
echo "Targets in config.json not sorted alphabetically:"
14+
echo "Targets in $source not sorted alphabetically:"
1415
echo "$output"
1516
status=1
1617
fi

‎.github/workflows/check-config.yml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
on:
22
pull_request:
33
paths:
4-
- 'config.json'
4+
- 'configs/*.json'
55

66
name: Check Config
77

@@ -12,5 +12,10 @@ jobs:
1212
targets: ${{ steps.set-matrix.outputs.targets }}
1313
steps:
1414
- uses: actions/checkout@v2
15-
- name: check if config.json is sorted alphabetically
16-
run: .github/workflows/check-config.sh config.json
15+
- name: check if config files are sorted alphabetically
16+
run: |
17+
for config in configs/*.json; do
18+
echo "::group::$config"
19+
.github/workflows/check-config.sh $config
20+
echo "::endgroup::"
21+
done

‎.github/workflows/copy-workflow.yml‎

Lines changed: 34 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,6 @@ on:
88
head_commit_url:
99
description: "github.event.head_commit.url of the dispatcher"
1010
required: true
11-
files:
12-
description: "List of files to deploy"
13-
required: true
1411
targets:
1512
description: "List of repositories to deploy to"
1613
required: true
@@ -28,7 +25,7 @@ jobs:
2825
TEMPLATE_REPO_DIR: "template-repo"
2926
TEMPLATE_DIR: "templates"
3027
NEEDS_UPDATE: 0
31-
INITIAL_TEST_DEPLOYMENT: 0
28+
INITIAL_WORKFLOW_DEPLOYMENT: 0
3229
GO_VERSION_BUMP: 0
3330
FILES: ""
3431
GITHUB_USER: "web3-bot"
@@ -48,11 +45,6 @@ jobs:
4845
uses: actions/checkout@v2
4946
with:
5047
path: ${{ env.TEMPLATE_REPO_DIR }}
51-
- uses: actions/setup-go@v2
52-
with:
53-
# This should be the same Go version we use in the go-check workflow.
54-
# go mod tidy, go vet, staticcheck and gofmt might behave differently depending on the version.
55-
go-version: "1.17.x"
5648
- name: determine GitHub default branch
5749
working-directory: ${{ env.TARGET_REPO_DIR }}
5850
run: |
@@ -63,105 +55,58 @@ jobs:
6355
run: |
6456
git config user.name ${{ env.GITHUB_USER }}
6557
git config user.email ${{ env.GITHUB_EMAIL }}
66-
- name: is initial test workflow deployment
58+
- name: determine files to add
59+
# By setting the environment variable, it's possible to programmatically add / modify this list.
60+
# See https://github.com/protocol/.github/blob/38135c75e47839623bf9b2748275d8c6167a8fa8/.github/workflows/copy-workflow.yml#L163-L168 for an example, how we used to make use of this.
6761
run: |
68-
if [[ ! -f $TARGET_REPO_DIR/.github/workflows/go-test.yml ]]; then
69-
echo "INITIAL_TEST_DEPLOYMENT=1" >> $GITHUB_ENV
70-
fi
71-
- name: remove Travis (on initial deployment)
72-
if: ${{ env.INITIAL_TEST_DEPLOYMENT == 1 }}
62+
files=${{ toJson(toJson(matrix.cfg.files)) }}
63+
files=$(echo -e "$files" | jq -c '.')
64+
echo "FILES=$files" >> $GITHUB_ENV
65+
- name: is initial workflow deployment
66+
# INITIAL_WORKFLOW_DEPLOYMENT=1 iff none of the files in the target repository exist yet
67+
run: |
68+
initial_workflow_deployment=1
69+
for f in $(jq -r '.[]' <<< ${{ toJson(env.FILES) }}); do
70+
if [[ -f $TARGET_REPO_DIR/$f ]]; then
71+
initial_workflow_deployment=0
72+
break
73+
fi
74+
done
75+
echo "INITIAL_WORKFLOW_DEPLOYMENT=$initial_workflow_deployment" >> $GITHUB_ENV
76+
- name: remove Travis (on initial workflow deployment)
77+
if: ${{ env.INITIAL_WORKFLOW_DEPLOYMENT == 1 }}
7378
working-directory: ${{ env.TARGET_REPO_DIR }}
7479
run: |
7580
if [[ -f .travis.yml ]]; then
7681
git rm .travis.yml
7782
git commit -m "disable Travis"
7883
fi
79-
- name: remove CircleCI (on initial deployment)
80-
if: ${{ env.INITIAL_TEST_DEPLOYMENT == 1 }}
84+
- name: remove CircleCI (on initial workflow deployment)
85+
if: ${{ env.INITIAL_WORKFLOW_DEPLOYMENT == 1 }}
8186
working-directory: ${{ env.TARGET_REPO_DIR }}
8287
run: |
8388
if [[ -d .circleci ]]; then
8489
git rm -r .circleci
8590
git commit -m "disable CircleCI"
8691
fi
87-
- name: remove gx (on initial deployment)
88-
if: ${{ env.INITIAL_TEST_DEPLOYMENT == 1 }}
92+
- name: remove gx (on initial workflow deployment)
93+
if: ${{ env.INITIAL_WORKFLOW_DEPLOYMENT == 1 }}
8994
working-directory: ${{ env.TARGET_REPO_DIR }}
9095
run: |
9196
if [[ -d .gx ]]; then
9297
git rm -r .gx
9398
git commit -m "remove .gx"
9499
fi
95-
- name: add version.json file (in order to deploy versioning workflows)
96-
if: hashFiles(format('{0}/version.json', env.TARGET_REPO_DIR)) == ''
97-
working-directory: ${{ env.TARGET_REPO_DIR }}
98-
run: |
99-
git fetch origin --unshallow # we need the entire commit history
100-
version=$(git describe --tags --abbrev=0 || true) # highest released version on current branch
101-
if [[ -n "$version" ]]; then # only deply version.json if there's at least one release
102-
printf '{"version": "%s"}' "$version" | jq . > version.json
103-
git add version.json
104-
git commit -m "add version.json file"
105-
fi
106-
- name: bump go.mod go version if needed
107-
uses: protocol/[email protected]
108-
with:
109-
working-directory: ${{ env.TARGET_REPO_DIR }}
110-
run: |
111-
# We want our modules to support two Go versions at a time.
112-
# As of August 2021, Go 1.17 is the latest stable.
113-
# go.mod's Go version declares the language version being used.
114-
# As such, it has to be the minimum of all Go versions supported.
115-
# Bump this every six months, as new Go versions come out.
116-
TARGET_VERSION=1.16
117-
118-
# Note that the "<" comparison doesn't understand semver,
119-
# but it should be good enough for the foreseeable future.
120-
CURRENT_VERSION=$(go list -m -f {{.GoVersion}})
121-
122-
if [[ $CURRENT_VERSION < $TARGET_VERSION ]]; then
123-
echo "GO_VERSION_BUMP=1" >> $GITHUB_ENV
124-
125-
# Update the version in go.mod. This alone ensures there's a diff.
126-
go mod edit -go $TARGET_VERSION
127-
128-
# In the future, "go fix" may make changes to Go code,
129-
# such as to adapt to language changes or API deprecations.
130-
# This is largely a no-op as of Go 1.17, and that's fine.
131-
go fix ./...
132-
git add .
133-
134-
# We don't tidy, because the next step does that.
135-
# Separate commits also help with reviews.
136-
git commit -m "bump go.mod to Go $TARGET_VERSION and run go fix"
137-
fi
138-
- name: go mod tidy (on initial deployment and on new Go version)
139-
if: ${{ env.INITIAL_TEST_DEPLOYMENT == 1 || env.GO_VERSION_BUMP == 1}}
140-
uses: protocol/[email protected]
141-
with:
142-
working-directory: ${{ env.TARGET_REPO_DIR }}
143-
run: |
144-
go mod tidy
145-
if ! git diff --quiet; then
146-
git add .
147-
git commit -m "run go mod tidy"
148-
fi
149-
- name: gofmt -s (on initial deployment and on new Go version)
150-
if: ${{ env.INITIAL_TEST_DEPLOYMENT == 1 || env.GO_VERSION_BUMP == 1}}
151-
working-directory: ${{ env.TARGET_REPO_DIR }}
152-
run: |
153-
gofmt -s -w .
154-
if ! git diff --quiet; then
155-
git add .
156-
git commit -m "run gofmt -s"
157-
fi
158-
- name: determine files to add
159-
# By setting the environment variable, it's possible to programmatically add / modify this list.
160-
# See https://github.com/protocol/.github/blob/38135c75e47839623bf9b2748275d8c6167a8fa8/.github/workflows/copy-workflow.yml#L163-L168 for an example, how we used to make use of this.
161-
run: echo "FILES=${{ toJson(github.event.inputs.files) }}" >> $GITHUB_ENV
100+
- name: Run steps specific to go
101+
if: matrix.cfg.deploy_go
102+
# use of ${{ env.TEMPATE_REPO_DIR }} is not allowed here
103+
uses: ./template-repo/.github/actions/copy-workflow-go
104+
- name: Run steps specific to versioning
105+
if: matrix.cfg.deploy_versioning
106+
uses: ./template-repo/.github/actions/copy-workflow-versioning
162107
- name: Add files
163108
run: |
164-
for f in $(jq -r ".[]" <<< ${{ toJson(env.FILES) }}); do
109+
for f in $(jq -r '.[]' <<< ${{ toJson(env.FILES) }}); do
165110
echo -e "\nProcessing $f."
166111
# add DO NOT EDIT header
167112
tmp=$(mktemp)
@@ -194,9 +139,9 @@ jobs:
194139
done
195140
- name: Check if we need to create a PR
196141
working-directory: ${{ env.TARGET_REPO_DIR }}
197-
run: echo "NEEDS_UPDATE=$(git rev-list HEAD...origin/$(git rev-parse --abbrev-ref HEAD) --ignore-submodules --count)" >> $GITHUB_ENV
142+
run: echo "NEEDS_UPDATE=$(git rev-list HEAD...origin/$(git rev-parse --abbrev-ref HEAD) --ignore-submodules --count 2> /dev/null || echo 1)" >> $GITHUB_ENV
198143
- name: Create Pull Request
199-
if: ${{ env.NEEDS_UPDATE }}
144+
if: ${{ env.NEEDS_UPDATE != 0 }}
200145
uses: peter-evans/create-pull-request@83dbed188f76ab04433c639ec214df65e26bc15c # https://github.com/peter-evans/create-pull-request/pull/856
201146
with:
202147
path: ${{ env.TARGET_REPO_DIR }}

0 commit comments

Comments
 (0)