Rarely error occurs on suzuki-shunsuke/tfaction/list-targets step #1023

ponkio-o · 2023-06-28T10:51:35Z

ponkio-o
Jun 28, 2023
Sponsor

tfaction version

We have encountered problems with some versions.

v0.5.25
v0.6.2

Overview

Although the exact cause is unknown, the following error occurs in suzuki-shunsuke/tfaction/list-targets in rare cases.
I have encountered this several times, all in the workflow to run terraform apply

Run suzuki-shunsuke/tfaction/[email protected]
Run tempfile="$(mktemp)"
time="2023-05-10T08:40:10Z" level=info msg="download and unarchive the package" aqua_version=2.5.1 env=linux/amd64 exe_name=github-comment package=suzuki-shunsuke/github-comment package_name=suzuki-shunsuke/github-comment package_version=v6.0.0 program=aqua registry=standard
time="2023-05-10T08:40:10Z" level=info msg="verify a package with slsa-verifier" aqua_version=2.5.1 env=linux/amd64 exe_name=github-comment package=suzuki-shunsuke/github-comment package_name=suzuki-shunsuke/github-comment package_version=v6.0.0 program=aqua registry=standard
time="2023-05-10T08:40:10Z" level=info msg="download and unarchive the package" aqua_version=2.5.1 env=linux/amd64 exe_name=github-comment package=suzuki-shunsuke/github-comment package_name=slsa-framework/slsa-verifier package_version=v2.2.0 program=aqua registry=
Verified signature against tlog entry index 18608072 at URL: https://rekor.sigstore.dev/api/v1/log/entries/24296fb24b8ad77a014f62c5e49b6ab8b99a3277d765fbc8df7c57e023de41b6fbb2b5ebb770c227
Verified build using builder https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v1.5.0 at commit 2b23e1cf2de3a938da53d58397ba17fe67b066fd
Verifying artifact /tmp/416872678: PASSED

PASSED: Verified SLSA provenance
time="2023-05-10T08:40:12Z" level=warning msg="list associated prs" error="associated pull request isn't found" org=MyOrg repo=MyRepo sha=553fdebf5f10823c02dafaa85da74e1d84c844e9
time="2023-05-10T08:40:12Z" level=info msg="download and unarchive the package" aqua_version=2.5.1 env=linux/amd64 exe_name=ci-info package=suzuki-shunsuke/ci-info package_name=suzuki-shunsuke/ci-info package_version=v2.1.2 program=aqua registry=standard
time="2023-05-10T08:40:13Z" level=info msg="verify a package with slsa-verifier" aqua_version=2.5.1 env=linux/amd64 exe_name=ci-info package=suzuki-shunsuke/ci-info package_name=suzuki-shunsuke/ci-info package_version=v2.1.2 program=aqua registry=standard
Verified signature against tlog entry index 11165794 at URL: https://rekor.sigstore.dev/api/v1/log/entries/24296fb24b8ad77a41190e6f40a14ebb5556d7270744dab7d7cca1294ce4f9e0393d9c2a42bac775
Verified build using builder https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v1.4.0 at commit 4a047e648dd0b9d0de1be356421d5d043c38d080
Verifying artifact /tmp/749567826: PASSED

PASSED: Verified SLSA provenance
cp: cannot create regular file '/env': Permission denied
Error: Process completed with exit code 1.

There appears to be an error when executing ci-info.

time="2023-05-10T08:40:12Z" level=warning msg="list associated prs" error="associated pull request isn't found" org=MyOrg repo=MyRepo sha=553fdebf5f10823c02dafaa85da74e1d84c844e9

tfaction/list-targets/action.yaml

Lines 22 to 27 in a496b98

    
                   tempfile="$(mktemp)" 
        
                   github-comment exec -- ci-info run | sed "s/^export //" > "${tempfile}" 
        
                   cat "$tempfile" >> "$GITHUB_ENV" 
        
                   . "$tempfile" 
        
                   cp "$tempfile" "${CI_INFO_TEMP_DIR}/env" 
        
                 shell: bash

How to reproduce

tfaction-root.yaml

---
target_groups:
- working_directory: envs/staging
  target: envs/staging
  aws_region: ap-northeast-1
  s3_bucket_name_plan_file: 'MyRepo-1234123412341234'
  s3_bucket_name_tfmigrate_history: 'MyRepo-1234123412341234'

  terraform_plan_config:
    aws_assume_role_arn: arn:aws:iam::1234123412341234:role/gha-terraform-plan-myrepo
  tfmigrate_plan_config:
    aws_assume_role_arn: arn:aws:iam::1234123412341234:role/gha-tfmigrate-plan-myrepo
  terraform_apply_config:
    aws_assume_role_arn: arn:aws:iam::1234123412341234:role/gha-terraform-apply-myrepo
  tfmigrate_apply_config:
    aws_assume_role_arn: arn:aws:iam::1234123412341234:role/gha-tfmigrate-apply-myrepo

- working_directory: envs/production
  target: envs/production
  aws_region: ap-northeast-1
  s3_bucket_name_plan_file: 'MyRepo-1234123412341234'
  s3_bucket_name_tfmigrate_history: 'MyRepo-1234123412341234'

  terraform_plan_config:
    aws_assume_role_arn: arn:aws:iam::1234123412341234:role/gha-terraform-plan-myrepo
  tfmigrate_plan_config:
    aws_assume_role_arn: arn:aws:iam::1234123412341234:role/gha-tfmigrate-plan-myrepo
  terraform_apply_config:
    aws_assume_role_arn: arn:aws:iam::1234123412341234:role/gha-terraform-apply-myrepo
  tfmigrate_apply_config:
    aws_assume_role_arn: arn:aws:iam::1234123412341234:role/gha-tfmigrate-apply-myrepo

tfaction.yaml

{}

GitHub Actions Workflow

---
name: apply
on:
  push:
    branches: [main]
env:
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  TFACTION_IS_APPLY: 'true'
permissions:
  id-token: write
  contents: read
  pull-requests: write
jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      tfmigrate_targets: ${{ steps.list-targets.outputs.tfmigrate_targets }}
      terraform_targets: ${{ steps.list-targets.outputs.terraform_targets }}
    steps:
      - uses: actions/checkout@v3
      - uses: aquaproj/[email protected]
        with:
          aqua_version: v2.9.0

      - uses: suzuki-shunsuke/tfaction/[email protected]
        id: list-targets

  tfmigrate-apply:
    name: "tfmigrate-apply (${{matrix.target.target}})"
    runs-on: ${{matrix.target.runs_on}}
    needs: setup
    # if services is empty, the build job is skipped
    if: "join(fromJSON(needs.setup.outputs.tfmigrate_targets), '') != ''"
    strategy:
      fail-fast: false
      matrix:
        target: ${{fromJSON(needs.setup.outputs.tfmigrate_targets)}}
    env:
      TFACTION_TARGET: ${{matrix.target.target}}
      TFACTION_JOB_TYPE: tfmigrate
    steps:
      - uses: actions/checkout@v3

      - uses: aquaproj/[email protected]
        with:
          aqua_version: v2.9.0

      - uses: suzuki-shunsuke/tfaction/[email protected]
        with:
          secrets: ${{ toJSON(secrets) }}

      - uses: suzuki-shunsuke/tfaction/[email protected]
        id: setup
        with:
          github_app_token: ${{ secrets.GIT_PERSONAL_TOKEN }}

      - uses: suzuki-shunsuke/tfaction/[email protected]
        id: target-config

      - uses: ./.github/actions/decrypt-tfvars
        with:
          working_dir: ${{ steps.target-config.outputs.working_directory }}
          kms_key_id: ${{ secrets.KMS_KEY_ID }}
          aws_region: ${{ steps.target-config.outputs.aws_region }}

      - uses: suzuki-shunsuke/tfaction/[email protected]
        with:
          github_app_token: ${{ secrets.GIT_PERSONAL_TOKEN }}

      - uses: suzuki-shunsuke/tfaction/[email protected]
        if: failure()
        with:
          github_app_token: ${{ secrets.GIT_PERSONAL_TOKEN }}

  terraform-apply:
    name: "terraform-apply (${{matrix.target.target}})"
    runs-on: ${{matrix.target.runs_on}}
    needs: setup
    # if services is empty, the build job is skipped
    if: "join(fromJSON(needs.setup.outputs.terraform_targets), '') != ''"
    strategy:
      fail-fast: false
      matrix:
        target: ${{fromJSON(needs.setup.outputs.terraform_targets)}}
    env:
      TFACTION_TARGET: ${{matrix.target.target}}
      TFACTION_JOB_TYPE: terraform
    steps:
      - uses: actions/checkout@v3

      - uses: aquaproj/[email protected]
        with:
          aqua_version: v2.9.0

      - uses: suzuki-shunsuke/tfaction/[email protected]
        with:
          secrets: ${{ toJSON(secrets) }}

      - uses: suzuki-shunsuke/tfaction/[email protected]
        id: setup
        with:
          github_app_token: ${{ secrets.GIT_PERSONAL_TOKEN }}

      - uses: suzuki-shunsuke/tfaction/[email protected]
        id: target-config

      - uses: ./.github/actions/decrypt-tfvars
        with:
          working_dir: ${{ steps.target-config.outputs.working_directory }}
          kms_key_id: ${{ secrets.KMS_KEY_ID }}
          aws_region: ${{ steps.target-config.outputs.aws_region }}

      - uses: suzuki-shunsuke/tfaction/[email protected]
        with:
          github_app_token: ${{ secrets.GIT_PERSONAL_TOKEN }}

      - uses: suzuki-shunsuke/tfaction/[email protected]
        if: failure()
        with:
          github_app_token: ${{ secrets.GIT_PERSONAL_TOKEN }}

GitHub Actions' log

△ Run suzuki-shunsuke/tfaction/[email protected]
  with:
    github_token: ***
  env:
    GITHUB_TOKEN: ***
    TFACTION_IS_APPLY: true
△ Run tempfile="$(mktemp)"
  tempfile="$(mktemp)"
  github-comment exec -- ci-info run | sed "s/^export //" > "${tempfile}"
  cat "$tempfile" >> "$GITHUB_ENV"
  . "$tempfile"
  cp "$tempfile" "${CI_INFO_TEMP_DIR}/env"
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    GITHUB_TOKEN: ***
    TFACTION_IS_APPLY: true
time="2023-05-10T08:40:10Z" level=info msg="download and unarchive the package" aqua_version=2.5.1 env=linux/amd64 exe_name=github-comment package=suzuki-shunsuke/github-comment package_name=suzuki-shunsuke/github-comment package_version=v6.0.0 program=aqua registry=standard
time="2023-05-10T08:40:10Z" level=info msg="verify a package with slsa-verifier" aqua_version=2.5.1 env=linux/amd64 exe_name=github-comment package=suzuki-shunsuke/github-comment package_name=suzuki-shunsuke/github-comment package_version=v6.0.0 program=aqua registry=standard
time="2023-05-10T08:40:10Z" level=info msg="download and unarchive the package" aqua_version=2.5.1 env=linux/amd64 exe_name=github-comment package=suzuki-shunsuke/github-comment package_name=slsa-framework/slsa-verifier package_version=v2.2.0 program=aqua registry=
Verified signature against tlog entry index 18608072 at URL: https://rekor.sigstore.dev/api/v1/log/entries/24296fb24b8ad77a014f62c5e49b6ab8b99a3277d765fbc8df7c57e023de41b6fbb2b5ebb770c227
Verified build using builder https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v1.5.0 at commit 2b23e1cf2de3a938da53d58397ba17fe67b066fd
Verifying artifact /tmp/416872678: PASSED

PASSED: Verified SLSA provenance
time="2023-05-10T08:40:12Z" level=warning msg="list associated prs" error="associated pull request isn't found" org=MyOrg repo=MyRepo sha=553fdebf5f10823c02dafaa85da74e1d84c844e9
time="2023-05-10T08:40:12Z" level=info msg="download and unarchive the package" aqua_version=2.5.1 env=linux/amd64 exe_name=ci-info package=suzuki-shunsuke/ci-info package_name=suzuki-shunsuke/ci-info package_version=v2.1.2 program=aqua registry=standard
time="2023-05-10T08:40:13Z" level=info msg="verify a package with slsa-verifier" aqua_version=2.5.1 env=linux/amd64 exe_name=ci-info package=suzuki-shunsuke/ci-info package_name=suzuki-shunsuke/ci-info package_version=v2.1.2 program=aqua registry=standard
Verified signature against tlog entry index 11165794 at URL: https://rekor.sigstore.dev/api/v1/log/entries/24296fb24b8ad77a41190e6f40a14ebb5556d7270744dab7d7cca1294ce4f9e0393d9c2a42bac775
Verified build using builder https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v1.4.0 at commit 4a047e648dd0b9d0de1be356421d5d043c38d080
Verifying artifact /tmp/749567826: PASSED

PASSED: Verified SLSA provenance
cp: cannot create regular file '/env': Permission denied
Error: Process completed with exit code 1.

Expected behaviour

Target is correctly selected

Actual behaviour

An error occures

Important Factoids

Conditions of occurrence are not known
I've only encountered this in the apply workflow
"Re-run all jobs" or "Re-run failled jobs" may encounter the same error no matter how many times it is run

Reference

No response

Answered by ponkio-o

Jun 28, 2023

Sorry, maybe it is a temporary problem 🤔 We ran into the problem about an hour and a half ago.
https://www.githubstatus.com/incidents/yr8dbc2p1ny4

View full answer

suzuki-shunsuke · 2023-06-28T10:59:41Z

suzuki-shunsuke
Jun 28, 2023
Maintainer

Can you reproduce the warning if you run ci-info in your laptop?
⚠️ Specify the commit hash which the issue occurs.

$ ci-info run --owner "$OWENR" --repo "$REPO" --sha "$SHA"

3 replies

ponkio-o Jun 28, 2023
Author Sponsor

Thank you for your response!

The error was not reproduced in the laptop.

$ ci-info run --owner <myorg> --repo <myrepo> --sha 2fb4eb9b01c1d1be747ba0a75155bb0c8d691310 --log-level DEBUG
DEBU[0000] get pull request from SHA                     owner=<myorg> repo=<myrpeo> sha=2fb4eb9b01c1d1be747ba0a75155bb0c8d691310
DEBU[0000] the number of pull requests assosicated with the commit  size=1
export CI_INFO_IS_PR=false
export CI_INFO_HAS_ASSOCIATED_PR=true
export CI_INFO_PR_NUMBER=139
export CI_INFO_BASE_REF=main
export CI_INFO_HEAD_REF=ponkioo/apply-prod
export CI_INFO_PR_AUTHOR=ponkio-o
export CI_INFO_PR_MERGED=true
export CI_INFO_TEMP_DIR=/var/folders/c4/17dhk9fx7s1btjjmzml88_g80000gp/T/ci-info689129194
export CI_INFO_REPO_OWNER=<myorg>
export CI_INFO_REPO_NAME=<myrepo>

$ cat /var/folders/c4/17dhk9fx7s1btjjmzml88_g80000gp/T/ci-info689129194/labels.txt
environments/production
environments/production/add-or-update

suzuki-shunsuke Jun 28, 2023
Maintainer

Ah, probably the error message list associated prs is outputted by not ci-info but github-comment.

https://github.com/suzuki-shunsuke/github-comment/blob/1924f1ca0e601786015743e2bb709c1588090e36/pkg/api/exec.go#L53-L65

suzuki-shunsuke Jun 28, 2023
Maintainer

Thank you for checking.

cp: cannot create regular file '/env': Permission denied

Hmm. It seems the value of $CI_INFO_TEMP_DIR is empty.

suzuki-shunsuke · 2023-06-28T11:04:10Z

suzuki-shunsuke
Jun 28, 2023
Maintainer

How did you merge the pull request? merge? squash? or rebase? Is the same with other pull requests?
I think ci-info should work regardless this, but I'm asking just in case.

1 reply

ponkio-o Jun 28, 2023
Author Sponsor

I used "Create a merge commit (default)".

suzuki-shunsuke · 2023-06-28T11:07:12Z

suzuki-shunsuke
Jun 28, 2023
Maintainer

BTW, tfaction should use set -eu and set -o pipefail in GitHub Actions run steps.
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsshell

0 replies

ponkio-o · 2023-06-28T11:33:27Z

ponkio-o
Jun 28, 2023
Author Sponsor

Sorry, maybe it is a temporary problem 🤔 We ran into the problem about an hour and a half ago.
https://www.githubstatus.com/incidents/yr8dbc2p1ny4

2 replies

suzuki-shunsuke Jun 28, 2023
Maintainer

Oh, I see.
Then I'd like to close this discussion for now.
If you'll face the same issue later, please let us know.

ponkio-o Jun 28, 2023
Author Sponsor

Thank you for investigation! You've always been a great help.

Uh oh!

Rarely error occurs on suzuki-shunsuke/tfaction/list-targets step #1023

Uh oh!

Uh oh!

ponkio-o Jun 28, 2023 Sponsor

tfaction version

Overview

How to reproduce

Expected behaviour

Actual behaviour

Important Factoids

Reference

Replies: 4 comments · 6 replies

Uh oh!

suzuki-shunsuke Jun 28, 2023 Maintainer

Uh oh!

ponkio-o Jun 28, 2023 Author Sponsor

Uh oh!

suzuki-shunsuke Jun 28, 2023 Maintainer

Uh oh!

Uh oh!

suzuki-shunsuke Jun 28, 2023 Maintainer

Uh oh!

suzuki-shunsuke Jun 28, 2023 Maintainer

Uh oh!

ponkio-o Jun 28, 2023 Author Sponsor

Uh oh!

suzuki-shunsuke Jun 28, 2023 Maintainer

Uh oh!

ponkio-o Jun 28, 2023 Author Sponsor

Uh oh!

suzuki-shunsuke Jun 28, 2023 Maintainer

Uh oh!

ponkio-o Jun 28, 2023 Author Sponsor

ponkio-o
Jun 28, 2023
Sponsor

Replies: 4 comments 6 replies

suzuki-shunsuke
Jun 28, 2023
Maintainer

ponkio-o Jun 28, 2023
Author Sponsor

suzuki-shunsuke Jun 28, 2023
Maintainer

suzuki-shunsuke Jun 28, 2023
Maintainer

suzuki-shunsuke
Jun 28, 2023
Maintainer

ponkio-o Jun 28, 2023
Author Sponsor

suzuki-shunsuke
Jun 28, 2023
Maintainer

ponkio-o
Jun 28, 2023
Author Sponsor

suzuki-shunsuke Jun 28, 2023
Maintainer

ponkio-o Jun 28, 2023
Author Sponsor