Skip to content

[WIP] Optimize GitHub Actions CI for faster Python package installation and unit tests #3335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
+103 −32
Draft

[WIP] Optimize GitHub Actions CI for faster Python package installation and unit tests #3335

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2008d4f
Optimize GitHub Actions CI for faster Python package installation and…
pingsutw Oct 3, 2025
eea31d7
Use uv by default for all package installation in CI
pingsutw Oct 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 103 additions & 32 deletions .github/workflows/pythonbuild.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,25 +50,33 @@ jobs:
- name: "Clear action cache"
uses: ./.github/actions/clear-action-cache
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ~/.cache/pip
# Look to see if there is a cache hit for the corresponding requirements files
path: |
~/.cache/pip
~/Library/Caches/pip
~\AppData\Local\pip\Cache
key: ${{ format('{0}-pip-{1}', runner.os, hashFiles('dev-requirements.in', 'requirements.in')) }}
restore-keys: |
${{ runner.os }}-pip-
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
enable-cache: true
- name: Install dependencies
run: |
pip install uv
make setup-global-uv
uv pip uninstall --system pandas pyarrow
uv pip freeze
- name: Test with coverage
run: |
make unit_test_codecov
env:
PYTEST_XDIST_AUTO_NUM_WORKERS: ${{ matrix.os == 'ubuntu-latest' && '4' || '2' }}
- name: Codecov
uses: codecov/[email protected]
with:
Expand All @@ -93,19 +101,25 @@ jobs:
- name: "Clear action cache"
uses: ./.github/actions/clear-action-cache
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ~/.cache/pip
# Look to see if there is a cache hit for the corresponding requirements files
path: |
~/.cache/pip
~/Library/Caches/pip
~\AppData\Local\pip\Cache
key: ${{ format('{0}-pip-{1}', runner.os, hashFiles('dev-requirements.in', 'requirements.in')) }}
restore-keys: |
${{ runner.os }}-pip-
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
enable-cache: true
- name: Install dependencies
run: |
pip install uv
make setup-global-uv
uv pip uninstall --system pandas pyarrow
uv pip freeze
Expand All @@ -115,6 +129,8 @@ jobs:
if: ${{ matrix.python-version != '3.12' }}
run: |
make unit_test_extras_codecov
env:
PYTEST_XDIST_AUTO_NUM_WORKERS: ${{ matrix.os == 'ubuntu-latest' && '4' || '2' }}
- name: Codecov
uses: codecov/[email protected]
with:
Expand Down Expand Up @@ -145,16 +161,29 @@ jobs:
- name: "Clear action cache"
uses: ./.github/actions/clear-action-cache
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ~/.cache/pip
# Look to see if there is a cache hit for the corresponding requirements files
path: |
~/.cache/pip
~/Library/Caches/pip
~\AppData\Local\pip\Cache
key: ${{ format('{0}-pip-{1}', runner.os, hashFiles('dev-requirements.in', 'requirements.in')) }}
restore-keys: |
${{ runner.os }}-pip-
- name: Cache uv
uses: actions/cache@v4
with:
path: |
~/.cache/uv
~/Library/Caches/uv
~\AppData\Local\uv\Cache
key: ${{ format('{0}-uv-{1}', runner.os, hashFiles('dev-requirements.in', 'requirements.in')) }}
restore-keys: |
${{ runner.os }}-uv-
- name: Install dependencies
run: |
pip install uv
Expand All @@ -164,6 +193,8 @@ jobs:
- name: Test with coverage
run: |
make unit_test_codecov
env:
PYTEST_XDIST_AUTO_NUM_WORKERS: 4
- name: Codecov
uses: codecov/[email protected]
with:
Expand All @@ -182,16 +213,29 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ~/.cache/pip
# Look to see if there is a cache hit for the corresponding requirements files
path: |
~/.cache/pip
~/Library/Caches/pip
~\AppData\Local\pip\Cache
key: ${{ format('{0}-pip-{1}', runner.os, hashFiles('dev-requirements.in', 'requirements.in')) }}
restore-keys: |
${{ runner.os }}-pip-
- name: Cache uv
uses: actions/cache@v4
with:
path: |
~/.cache/uv
~/Library/Caches/uv
~\AppData\Local\uv\Cache
key: ${{ format('{0}-uv-{1}', runner.os, hashFiles('dev-requirements.in', 'requirements.in')) }}
restore-keys: |
${{ runner.os }}-uv-
- name: Install dependencies
run: |
pip install uv
Expand All @@ -200,6 +244,8 @@ jobs:
- name: Test with coverage
run: |
make test_serialization_codecov
env:
PYTEST_XDIST_AUTO_NUM_WORKERS: 4
- name: Codecov
uses: codecov/[email protected]
with:
Expand All @@ -226,16 +272,29 @@ jobs:
- name: "Clear action cache"
uses: ./.github/actions/clear-action-cache # sandbox has disk pressure, so we need to clear the cache to get more disk space.
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ~/.cache/pip
# Look to see if there is a cache hit for the corresponding requirements files
path: |
~/.cache/pip
~/Library/Caches/pip
~\AppData\Local\pip\Cache
key: ${{ format('{0}-pip-{1}', runner.os, hashFiles('dev-requirements.in', 'requirements.in')) }}
restore-keys: |
${{ runner.os }}-pip-
- name: Cache uv
uses: actions/cache@v4
with:
path: |
~/.cache/uv
~/Library/Caches/uv
~\AppData\Local\uv\Cache
key: ${{ format('{0}-uv-{1}', runner.os, hashFiles('dev-requirements.in', 'requirements.in')) }}
restore-keys: |
${{ runner.os }}-uv-
- name: Install dependencies
run: |
pip install uv
Expand Down Expand Up @@ -394,19 +453,25 @@ jobs:
- name: "Clear action cache"
uses: ./.github/actions/clear-action-cache
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v4
with:
# This path is specific to Ubuntu
path: ~/.cache/pip
# Look to see if there is a cache hit for the corresponding requirements files
path: |
~/.cache/pip
~/Library/Caches/pip
~\AppData\Local\pip\Cache
key: ${{ format('{0}-pip-{1}', runner.os, hashFiles('dev-requirements.txt', format('plugins/{0}/requirements.txt', matrix.plugin-names ))) }}
restore-keys: |
${{ runner.os }}-pip-
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
enable-cache: true
- name: Install dependencies
run: |
pip install uv
# TODO: double-check if checking out all tags solves the issue
export SETUPTOOLS_SCM_PRETEND_VERSION="1.999.999"
make setup-global-uv
Expand Down Expand Up @@ -441,19 +506,25 @@ jobs:
- name: Fetch the code
uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: 3.12
- uses: actions/cache@v4
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
path: ~/.cache/pip
enable-cache: true
- name: Cache pip
uses: actions/cache@v4
with:
path: |
~/.cache/pip
~/Library/Caches/pip
~\AppData\Local\pip\Cache
key: ${{ runner.os }}-pip-${{ hashFiles('**/dev-requirements.in') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install uv
make setup-global-uv
uv pip freeze
- name: Lint
Expand Down
Loading