[v2-10-test] Backport pull_requttest_target removal

This is a bulk change that synchronizes dev/ci scripts for v2-10-test
branch with main #45266 - including follow-ups.

Rather than cherry-picking relevant PRs, this one gets the
latest version of the scripts from main and updates the branch with
some changes to adapt them to v2-10-test (such as bringing back
python 3.8 support, removing some providers checks after the
bulk move of providers and making sure all tests are passing.

This is far easier than cherry-picking the changes, because for
the v2-10-test we stopped cherry-picking CI changes which was
deemed unnecessary (we used to do it for all previous branches)
but this made it far more difficult (if not impossible) to
cherry-pick individual changes.

Fortunately, the CI scripts are maintained in the way that their
latest version **should** in principle work for a v2-* branch and
hopefully after just a few adjustments we should be able to
synchronize the changes from main by updating all relevant
CI/DEV scripts, dockerfile images, workflows, pre-commits etc.

Add actions in codeql workflows to scan github workflow actions (#45534)

* add actions in codeql workflows to scan github workflow actions

* add actions in codeql workflows to scan github workflow actions

CodeQL scanning can run always on all code (#45541)

The CodeQL scannig is fast and having custom configuration to
select which scanning to run should be run makes it unnecessarily
complex

We can just run all CodeQL scans always.

This has been suggested by actions codeql scan itself.

Add explicit permissions for all workflow-run workflows (#45548)

Those workflows inherit permissions from the calling workflows
but it's good to add explicit permissions to indicate what is
needed and in case we will also use the workflows for other purposes
in the future - default permissions for older repos might be
write so it's best to be explicit about the permissions.

Found by CodeQL scanning

Remove contents: write permission from generate-constraints (#45558)

The write permission cannot be set for PRs from forks in the
call workflow - so we have to come back to implicit permissions
and make explicit permissions passing a bit differently.

(cherry picked from commit ae32ebc)

Bump trove-classifiers from 2025.1.7.14 to 2025.1.10.15 (#45561)

Bumps [trove-classifiers](https://github.com/pypa/trove-classifiers) from 2025.1.7.14 to 2025.1.10.15.
- [Release notes](https://github.com/pypa/trove-classifiers/releases)
- [Commits](pypa/trove-classifiers@2025.1.7.14...2025.1.10.15)

---
updated-dependencies:
- dependency-name: trove-classifiers
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit f3fd262)

Add optional --image-file-dir to store loaded files elsewhere (#45564)

While backorting the "pull_request_target" removal to v2-10-test
branches it turned out that there is not enough disk space
on Public runner to load all 5 images and keep the file dump at
the same time in the same filesystem. This PR allows to choose
where the load/save files will be stored and in the github
runner environment we store the files in "/mnt" wnich is a separate
folder with 40GB free.

(cherry picked from commit 6628049)

Fix --from-pr feature for image load and stabilize help

This is a follow-up after #45564 - it fixes the `--from-pr` and
`--from-run` to work (it was failing with file does not exist).

Also found out that gettempdir might return different directory
depending on which is your designated tmp directory (for example
in MacOS this is is a longer path in /var/.....) - so we have
to force the default during help generation to always return
"/tmp" so that the --help images do not change depending on which
system you are and what your tmp directory is.

.dockerignore

@@ -34,12 +34,15 @@
 !chart
 !docs
 !licenses
+!providers/
+!task_sdk/
 
 # Add those folders to the context so that they are available in the CI container
 !scripts
 
 # Add tests and kubernetes_tests to context.
 !tests
+!tests_common
 !kubernetes_tests
 !helm_tests
 !docker_tests
@@ -79,6 +82,7 @@
 airflow/git_version
 # Exclude mode_modules pulled by "yarn" for compilation of www files generated by NPM
 airflow/www/node_modules
+airflow/ui/node_modules
 
 # Exclude link to docs
 airflow/www/static/docs
@@ -121,6 +125,9 @@ docs/_build/
 docs/_api/
 docs/_doctrees/
 
+# Exclude new providers docs generated files
+providers/**/docs/_api/
+
 # files generated by memray
 *.py.*.html
 *.py.*.bin

.github/actions/breeze/action.yml

@@ -21,10 +21,10 @@ description: 'Sets up Python and Breeze'
 inputs:
   python-version:
     description: 'Python version to use'
-    # Version of Python used for reproducibility of the packages built
-    # Python 3.8 tarfile produces different tarballs than Python 3.9+ tarfile that's why we are forcing
-    # Python 3.9 for all release preparation commands to make sure that the tarballs are reproducible
     default: "3.9"
+  use-uv:
+    description: 'Whether to use uv tool'
+    required: true
 outputs:
   host-python-version:
     description: Python version used in host
@@ -36,13 +36,11 @@ runs:
       uses: actions/setup-python@v5
       with:
         python-version: ${{ inputs.python-version }}
-        cache: 'pip'
-        cache-dependency-path: ./dev/breeze/pyproject.toml
+    # NOTE! Installing Breeze without using cache is FASTER than when using cache - uv is so fast and has
+    # so low overhead, that just running upload cache/restore cache is slower than installing it from scratch
     - name: "Install Breeze"
       shell: bash
       run: ./scripts/ci/install_breeze.sh
-      env:
-        PYTHON_VERSION: ${{ inputs.python-version }}
     - name: "Free space"
       shell: bash
       run: breeze ci free-space

.github/actions/install-pre-commit/action.yml

@@ -21,30 +21,53 @@ description: 'Installs pre-commit and related packages'
 inputs:
   python-version:
     description: 'Python version to use'
-    default: 3.9
+    default: "3.9"
   uv-version:
     description: 'uv version to use'
-    default: 0.5.11
+    default: "0.5.17"  # Keep this comment to allow automatic replacement of uv version
   pre-commit-version:
     description: 'pre-commit version to use'
-    default: 4.0.1
-  pre-commit-uv-version:
-    description: 'pre-commit-uv version to use'
-    default: 4.1.4
+    default: "3.5.0"  # Keep this comment to allow automatic replacement of pre-commit version
 runs:
   using: "composite"
   steps:
-    - name: Install pre-commit, uv, and pre-commit-uv
+    - name: Install pre-commit, uv
       shell: bash
-      run: >
-        pip install
-        pre-commit==${{inputs.pre-commit-version}}
-        uv==${{inputs.uv-version}}
-        pre-commit-uv==${{inputs.pre-commit-uv-version}}
-    - name: Cache pre-commit envs
-      uses: actions/cache@v4
+      env:
+        UV_VERSION: ${{inputs.uv-version}}
+        PRE_COMMIT_VERSION: ${{inputs.pre-commit-version}}
+      run: |
+        pip install uv==${UV_VERSION} || true
+        uv tool install pre-commit==${PRE_COMMIT_VERSION} --with uv==${UV_VERSION}
+      working-directory: ${{ github.workspace }}
+    # We need to use tar file with archive to restore all the permissions and symlinks
+    - name: "Delete ~.cache"
+      run: |
+        du ~/ --max-depth=2
+        echo
+        echo Deleting ~/.cache
+        echo
+        rm -rf ~/.cache
+        echo
+      shell: bash
+    - name: "Restore pre-commit cache"
+      uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
       with:
-        path: ~/.cache/pre-commit
-        key: "pre-commit-${{inputs.python-version}}-${{ hashFiles('.pre-commit-config.yaml') }}"
-        restore-keys: |
-          pre-commit-${{inputs.python-version}}-
+        key: cache-pre-commit-v4-${{ inputs.python-version }}-${{ hashFiles('.pre-commit-config.yaml') }}
+        path: /tmp/
+      id: restore-pre-commit-cache
+    - name: "Restore .cache from the tar file"
+      run: tar -C ~ -xzf /tmp/cache-pre-commit.tar.gz
+      shell: bash
+      if: steps.restore-pre-commit-cache.outputs.stash-hit == 'true'
+    - name: "Show restored files"
+      run: |
+        echo "Restored files"
+        du ~/ --max-depth=2
+        echo
+      shell: bash
+      if: steps.restore-pre-commit-cache.outputs.stash-hit == 'true'
+    - name: Install pre-commit hooks
+      shell: bash
+      run: pre-commit install-hooks || (cat ~/.cache/pre-commit/pre-commit.log && exit 1)
+      working-directory: ${{ github.workspace }}

.github/actions/post_tests_success/action.yml

@@ -33,7 +33,7 @@ runs:
     - name: "Upload artifact for warnings"
       uses: actions/upload-artifact@v4
       with:
-        name: test-warnings-${{env.JOB_ID}}
+        name: test-warnings-${{ env.JOB_ID }}
         path: ./files/warnings-*.txt
         retention-days: 7
         if-no-files-found: ignore
@@ -50,5 +50,5 @@ runs:
       if: env.ENABLE_COVERAGE == 'true' && env.TEST_TYPES != 'Helm' && inputs.python-version != '3.12'
       with:
         name: coverage-${{env.JOB_ID}}
-        flags: python-${{env.PYTHON_MAJOR_MINOR_VERSION}},${{env.BACKEND}}-${{env.BACKEND_VERSION}}
+        flags: python-${{ env.PYTHON_MAJOR_MINOR_VERSION }},${{ env.BACKEND }}-${{ env.BACKEND_VERSION }}
         directory: "./files/coverage-reports/"

.github/actions/prepare_all_ci_images/action.yml

@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: 'Prepare all CI images'
+description: 'Recreates current python CI images from artifacts for all python versions'
+inputs:
+  python-versions-list-as-string:
+    description: 'Stringified array of all Python versions to test - separated by spaces.'
+    required: true
+  platform:
+    description: 'Platform for the build - linux/amd64 or linux/arm64'
+    required: true
+runs:
+  using: "composite"
+  steps:
+    - name: "Cleanup docker"
+      run: ./scripts/ci/cleanup_docker.sh
+      shell: bash
+    # TODO: Currently we cannot loop through the list of python versions and have dynamic list of
+    #       tasks. Instead we hardcode all possible python versions and they - but
+    #       this should be implemented in stash action as list of keys to download.
+    #       That includes 3.8 - 3.12 as we are backporting it to v2-10-test branch
+    #       This is captured in https://github.com/apache/airflow/issues/45268
+    - name: "Restore CI docker image ${{ inputs.platform }}:3.8"
+      uses: ./.github/actions/prepare_single_ci_image
+      with:
+        platform: ${{ inputs.platform }}
+        python: "3.8"
+        python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
+    - name: "Restore CI docker image ${{ inputs.platform }}:3.9"
+      uses: ./.github/actions/prepare_single_ci_image
+      with:
+        platform: ${{ inputs.platform }}
+        python: "3.9"
+        python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
+    - name: "Restore CI docker image ${{ inputs.platform }}:3.10"
+      uses: ./.github/actions/prepare_single_ci_image
+      with:
+        platform: ${{ inputs.platform }}
+        python: "3.10"
+        python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
+    - name: "Restore CI docker image ${{ inputs.platform }}:3.11"
+      uses: ./.github/actions/prepare_single_ci_image
+      with:
+        platform: ${{ inputs.platform }}
+        python: "3.11"
+        python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
+    - name: "Restore CI docker image ${{ inputs.platform }}:3.12"
+      uses: ./.github/actions/prepare_single_ci_image
+      with:
+        platform: ${{ inputs.platform }}
+        python: "3.12"
+        python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}

.github/actions/prepare_breeze_and_image/action.yml

@@ -16,30 +16,52 @@
 # under the License.
 #
 ---
-name: 'Prepare breeze && current python image'
-description: 'Installs breeze and pulls current python image'
+name: 'Prepare breeze && current image (CI or PROD)'
+description: 'Installs breeze and recreates current python image from artifact'
 inputs:
-  pull-image-type:
-    description: 'Which image to pull'
-    default: CI
+  python:
+    description: 'Python version for image to prepare'
+    required: true
+  image-type:
+    description: 'Which image type to prepare (ci/prod)'
+    default: "ci"
+  platform:
+    description: 'Platform for the build - linux/amd64 or linux/arm64'
+    required: true
+  use-uv:
+    description: 'Whether to use uv'
+    required: true
 outputs:
   host-python-version:
     description: Python version used in host
     value: ${{ steps.breeze.outputs.host-python-version }}
 runs:
   using: "composite"
   steps:
+    - name: "Cleanup docker"
+      run: ./scripts/ci/cleanup_docker.sh
+      shell: bash
     - name: "Install Breeze"
       uses: ./.github/actions/breeze
+      with:
+        use-uv: ${{ inputs.use-uv }}
       id: breeze
-    - name: Login to ghcr.io
+    - name: Check free space
+      run: df -H
       shell: bash
-      run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
-    - name: Pull CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG }}
+    - name: Make /mnt/ directory writeable
+      run: sudo chown -R ${USER} /mnt
       shell: bash
-      run: breeze ci-image pull --tag-as-latest
-      if: inputs.pull-image-type == 'CI'
-    - name: Pull PROD image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:${{ env.IMAGE_TAG }}
+    - name: "Restore ${{ inputs.image-type }} docker image ${{ inputs.platform }}:${{ inputs.python }}"
+      uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+      with:
+        key: ${{ inputs.image-type }}-image-save-${{ inputs.platform }}-${{ inputs.python }}
+        path: "/mnt/"
+    - name: "Load ${{ inputs.image-type }} image ${{ inputs.platform }}:${{ inputs.python }}"
+      env:
+        PLATFORM: ${{ inputs.platform }}
+        PYTHON: ${{ inputs.python }}
+        IMAGE_TYPE: ${{ inputs.image-type }}
+      run: >
+        breeze ${IMAGE_TYPE}-image load --platform "${PLATFORM}" --python "${PYTHON}" --image-file-dir "/mnt"
       shell: bash
-      run: breeze prod-image pull --tag-as-latest
-      if: inputs.pull-image-type == 'PROD'