Combine different release pipelines by the use of reusable workflows (o…

…nnx#6277) ### Description  This pull request should be the start of a pipeline (current status of the considerations under onnx#6246) that automatically generates a whls after tagging, signs it and publishes it directly to Pypi to improve supply chain security. As this is all too much at once, the changes will be introduced and tested gradually. Changes related to this PR: Reusable Workflows are used to merge the individual OS release builds together (we need one process as we want to have all artifacts in one pipeline, so there is not need to download them manuelly). The workflows are currently not really "reusable", as we more or less use pypi creditionals in the workflow, among other things. I reused our testpypi weekly repo as I didn't want to change our behavoir for pypi and testpypi. Maybe it would make sense to adapt the process right away so that you can trigger our "Weekly" manually via the Github interface for testing and don't have to wait a week ;-) #### Key points for the review: * How do we want to configure concurrency * What concurrency-groups do we need? * Where do we need "cancel-in-progress"? #### Not part of the pull request: * Extract publish whl to pypi/testpyp * Introduce trusted publishing * ... ### Motivation and Context * see onnx#6246 --------- Signed-off-by: Andreas Fehlner <[email protected]> Co-authored-by: Justin Chu <[email protected]>
yuanyao-nv · Aug 23, 2024 · d45ad99 · d45ad99
1 parent 2c4b0cd
commit d45ad99
Show file tree

Hide file tree

Showing 5 changed files with 127 additions and 67 deletions.
diff --git a/.github/workflows/create_release.yml b/.github/workflows/create_release.yml
@@ -0,0 +1,86 @@
+# Copyright (c) ONNX Project Contributors
+#
+# SPDX-License-Identifier: Apache-2.0
+
+name: Create Releases
+on:
+  schedule:
+    # Run weekly on Monday 00:00
+    - cron:  '00 00 * * MON'
+  push:
+    branches: [main, rel-*]
+  pull_request:
+    branches: [main, rel-*]
+  workflow_dispatch:    
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name == 'workflow_dispatch' }}
+  cancel-in-progress: true
+
+jobs: 
+
+  call-workflow-ubuntu_x86:    
+    strategy:
+      matrix:
+        os: ['ubuntu-latest']    
+    uses: ./.github/workflows/release_linux_x86_64.yml
+    with:            
+      os: "linux_x86_64" 
+
+  call-workflow-ubuntu_aarch64:    
+    strategy:
+      matrix:
+        os: ['ubuntu-latest']    
+    uses: ./.github/workflows/release_linux_aarch64.yml
+    with:            
+      os: "linux_aarch64"      
+
+  call-workflow-win:    
+    strategy:
+      matrix:
+        os: ['windows-latest']    
+    uses: ./.github/workflows/release_win.yml
+    with:      
+      os: "win"      
+
+  call-workflow-mac:    
+    strategy:
+      matrix:
+        os: ['mac-latest']    
+    uses: ./.github/workflows/release_mac.yml
+    with:            
+      os: "macos"   
+
+
+
+  publish_to_testpypi: 
+    # TODO Add a deployment step for reviewing https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments
+    name: Release (Publish to testpypi, onnxweekly)
+    runs-on: ubuntu-latest
+    needs: [call-workflow-ubuntu_x86, call-workflow-ubuntu_aarch64, call-workflow-mac, call-workflow-win]
+    if: (github.ref == 'refs/head/main') && (github.event_name != 'pull_request') && ((needs.call-workflow-mac.result == 'success') || (needs.call-workflow-ubuntu_x86.result == 'success') || (needs.call-workflow-ubuntu_aarch64.result == 'success') || (needs.call-workflow-win.result == 'success'))
+
+    environment:      
+      name: testpypi 
+      url: https://test.pypi.org/p/onnx
+
+    permissions:
+      contents: write  # IMPORTANT: mandatory for making GitHub Releases
+      id-token: write  # IMPORTANT: mandatory for sigstore     
+
+    steps:
+
+      - uses: actions/download-artifact@v4
+        with:
+          pattern: wheels* 
+          path: dist
+          merge-multiple: true
+
+      - name: Publish distribution to TestPyPI
+        if: (github.event_name == 'schedule') && (github.repository_owner == 'onnx')
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:   
+          repository-url: https://test.pypi.org/legacy/
+          verbose: true     
+          print-hash: true  
+
diff --git a/.github/workflows/release_linux_aarch64.yml b/.github/workflows/release_linux_aarch64.yml
@@ -4,23 +4,16 @@
 
 name: LinuxRelease_aarch64
 
-on:
-  schedule:
-    # Run weekly on Monday 00:00
-    - cron:  '00 00 * * MON'
-  push:
-    branches: [main, rel-*]
-  pull_request:
-    branches: [main, rel-*]
-  workflow_dispatch:
+on:  # Specifies the event triggering the workflow
+  workflow_call:  # Indicates that this is a reusable workflow
+    inputs:      
+      os:
+        required: true
+        type: string
 
 permissions:  # set top-level default permissions as security best practice
   contents: read
 
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name == 'workflow_dispatch' }}
-  cancel-in-progress: true
-
 jobs:
   build:
     if: github.event_name != 'pull_request' || startsWith( github.base_ref, 'rel-') || contains( github.event.pull_request.labels.*.name, 'run release CIs')
@@ -83,11 +76,12 @@ jobs:
           pytest && \
           deactivate'
 
-    - uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # v3.1.3
+    - uses: actions/upload-artifact@0b2256b8c012f0828dc542b3febcab082c67f72b
       with:
-        name: wheels
-        path: dist
-
+        name: wheels-${{ inputs.os }}-${{ matrix.python-version }}
+        path: |
+            ./dist/*.whl
+            
     - name: Upload wheel to PyPI weekly
       if: (github.event_name == 'schedule') # Only triggered by weekly event
       run: |

diff --git a/.github/workflows/release_linux_x86_64.yml b/.github/workflows/release_linux_x86_64.yml
@@ -4,23 +4,16 @@
 
 name: LinuxRelease_x86_64
 
-on:
-  schedule:
-    # Run weekly on Monday 00:00
-    - cron:  '00 00 * * MON'
-  push:
-    branches: [main, rel-*]
-  pull_request:
-    branches: [main, rel-*]
-  workflow_dispatch:
+on:  # Specifies the event triggering the workflow
+  workflow_call:  # Indicates that this is a reusable workflow
+    inputs:      
+      os:
+        required: true
+        type: string  
 
 permissions:  # set top-level default permissions as security best practice
   contents: read
 
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name == 'workflow_dispatch' }}
-  cancel-in-progress: true
-
 jobs:
   build:
     if: github.event_name != 'pull_request' || startsWith( github.base_ref, 'rel-') || contains( github.event.pull_request.labels.*.name, 'run release CIs')
@@ -65,10 +58,12 @@ jobs:
         python -m pip install dist/*manylinux2014_x86_64.whl
         pytest
 
-    - uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
+    - uses: actions/upload-artifact@0b2256b8c012f0828dc542b3febcab082c67f72b
       with:
-        name: wheels
-        path: dist
+        name: wheels-${{ inputs.os }}-${{ matrix.python-version }}
+        path: |
+            ./dist/*.whl
+        
 
     - name: Upload wheel to PyPI weekly
       if: (github.event_name == 'schedule') # Only triggered by weekly event

diff --git a/.github/workflows/release_mac.yml b/.github/workflows/release_mac.yml
@@ -4,15 +4,12 @@
 
 name: MacRelease
 
-on:
-  schedule:
-    # Run weekly on Monday 00:00
-    - cron:  '00 00 * * MON'
-  push:
-    branches: [main, rel-*]
-  pull_request:
-    branches: [main, rel-*]
-  workflow_dispatch:
+on:  # Specifies the event triggering the workflow
+  workflow_call:  # Indicates that this is a reusable workflow
+    inputs:      
+      os:
+        required: true
+        type: string
 
 # Use MACOSX_DEPLOYMENT_TARGET=12.0 to produce compatible wheel
 env:
@@ -21,10 +18,6 @@ env:
 permissions:
   contents: read
 
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name == 'workflow_dispatch' }}
-  cancel-in-progress: true
-
 jobs:
   build:
     if: github.event_name != 'pull_request' || startsWith( github.base_ref, 'rel-') || contains( github.event.pull_request.labels.*.name, 'run release CIs')
@@ -68,10 +61,10 @@ jobs:
         fi
         python -m build --wheel
 
-    - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
+    - uses: actions/upload-artifact@v4
       with:
-        name: macos-wheel-${{ matrix.python-version }}
-        path: dist
+        name: wheels-${{ inputs.os }}-${{ matrix.python-version }}
+        path: dist/*.whl
 
   test:
     needs: build
@@ -93,9 +86,9 @@ jobs:
         arch -${{ matrix.target-architecture }} python -m pip install -q --upgrade pip
         arch -${{ matrix.target-architecture }} python -m pip install -q -r requirements-release.txt
 
-    - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
+    - uses: actions/download-artifact@v4
       with:
-        name: macos-wheel-${{ matrix.python-version }}
+        name:  wheels-${{ inputs.os }}-${{ matrix.python-version }}
         path: dist
 
     - name: Test the wheel

diff --git a/.github/workflows/release_win.yml b/.github/workflows/release_win.yml
@@ -4,23 +4,16 @@
 
 name: WindowsRelease
 
-on:
-  schedule:
-    # Run weekly on Monday 00:00
-    - cron:  '00 00 * * MON'
-  push:
-    branches: [main, rel-*]
-  pull_request:
-    branches: [main, rel-*]
-  workflow_dispatch:
+on:  # Specifies the event triggering the workflow
+  workflow_call:  # Indicates that this is a reusable workflow
+    inputs:      
+      os:
+        required: true
+        type: string
 
 permissions:  # set top-level default permissions as security best practice
   contents: read
 
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name == 'workflow_dispatch' }}
-  cancel-in-progress: true
-
 jobs:
   build:
     if: github.event_name != 'pull_request' || startsWith( github.base_ref, 'rel-') || contains( github.event.pull_request.labels.*.name, 'run release CIs')
@@ -79,15 +72,14 @@ jobs:
         }
         python -m build --wheel
         Get-ChildItem -Path dist/*.whl | foreach {python -m pip install --upgrade $_.fullname}
-
     - name: Test the installed wheel
       run: |
         cd onnx
         pytest
 
-    - uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # v3.1.3
+    - uses: actions/upload-artifact@0b2256b8c012f0828dc542b3febcab082c67f72b
       with:
-        name: wheels
+        name: wheels-${{ inputs.os }}-${{ matrix.python-version }}-${{matrix.architecture}}
         path: ./onnx/dist
 
     - name: Upload onnx-weekly wheel to PyPI/PyPI weekly