Skip to content

Simplify the setup in CI #542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Feb 20, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .github/workflows/basic-tests-linux-uv.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,13 @@ jobs:
- name: Set up Python (uv)
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.13"

- name: Install uv and dependencies
shell: bash
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10 # tests for backwards compatibility
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval

Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/basic-tests-macos-uv.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,13 @@ jobs:
- name: Set up Python (uv)
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.13"

- name: Install uv and dependencies
shell: bash
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10 # tests for backwards compatibility
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval

Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/basic-tests-old-pytorch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,12 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.13"

- name: Install dependencies
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10 # tests for backwards compatibility
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add torch==${{ matrix.pytorch-version }}
uv add pytest-ruff nbval
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/basic-tests-pip.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.10" # tests for backwards compatibility

- name: Create Virtual Environment and Install Dependencies
run: |
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/basic-tests-pytorch-rc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,12 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.13"

- name: Install dependencies
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10 # tests for backwards compatibility
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval
uv pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
Expand Down
66 changes: 66 additions & 0 deletions .github/workflows/basic-tests-windows-uv-pip.yml.disabled
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
name: Code tests Windows (uv/pip)

on:
push:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
pull_request:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'

jobs:
test:
runs-on: windows-latest

steps:
- name: Checkout Code
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.13"

- name: Install dependencies
shell: pwsh
run: |
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
python -m pip install --upgrade pip
python -m pip install uv
uv venv --python=python3.11
. .\.venv\Scripts\Activate.ps1
$env:UV_PIP_OPTS="--no-binary tensorflow-io-gcs-filesystem"
uv pip install -r requirements.txt
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv pip install pytest-ruff nbval
uv pip install --force-reinstall matplotlib "numpy<2.1"

- name: Run Python Tests
shell: pwsh
run: |
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
. .\.venv\Scripts\Activate.ps1
pytest --ruff setup/02_installing-python-libraries/tests.py
pytest --ruff ch04/01_main-chapter-code/tests.py
pytest --ruff ch05/01_main-chapter-code/tests.py
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
pytest --ruff ch06/01_main-chapter-code/tests.py

- name: Run Jupyter Notebook Tests
shell: pwsh
run: |
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
. .\.venv\Scripts\Activate.ps1
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb
63 changes: 63 additions & 0 deletions .github/workflows/basic-tests-windows-uv.yml.disabled
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: Code tests Windows (uv)

on:
push:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
pull_request:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'

jobs:
test:
runs-on: windows-latest

steps:
- name: Checkout Code
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.13"

- name: Install dependencies
shell: pwsh
run: |
# Prepend local bin directory to PATH
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
uv sync --dev --python=3.10
$env:UV_PIP_OPTS="--no-binary tensorflow-io-gcs-filesystem"
uv pip install -r requirements.txt
uv pip install matplotlib # for some reason Windows requires this
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval

- name: Run Python Tests
shell: pwsh
run: |
. .\.venv\Scripts\Activate.ps1
pytest --ruff setup/02_installing-python-libraries/tests.py
pytest --ruff ch04/01_main-chapter-code/tests.py
pytest --ruff ch05/01_main-chapter-code/tests.py
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
pytest --ruff ch06/01_main-chapter-code/tests.py

- name: Run Jupyter Notebook Tests
shell: pwsh
run: |
. .\.venv\Scripts\Activate.ps1
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb
4 changes: 1 addition & 3 deletions .github/workflows/check-links.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,11 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
python-version: "3.10"

- name: Install dependencies
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv add pytest-ruff pytest-check-links
# Current version of retry doesn't work well if there are broken non-URL links
# pip install pytest pytest-check-links pytest-retry
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/check-spelling-errors.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,12 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
python-version: "3.10"

- name: Install codespell
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10
uv add codespell

- name: Run codespell
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/pep8-linter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,11 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
python-version: "3.13"
- name: Install ruff (a faster flake 8 equivalent)
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10
uv add ruff

- name: Run ruff with exceptions
Expand Down
3 changes: 1 addition & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,7 @@ dependencies = [
"jupyterlab>=4.0",
"tiktoken>=0.5.1",
"matplotlib>=3.7.1",
"tensorflow>=2.18.0; sys_platform != \"win32\"",
"tensorflow-cpu>=2.18.0; sys_platform == \"win32\"",
"tensorflow>=2.18.0",
"tqdm>=4.66.1",
"numpy>=1.26,<2.1",
"pandas>=2.2.1",
Expand Down
19 changes: 9 additions & 10 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
torch >= 2.3.0 # all
jupyterlab >= 4.0 # all
tiktoken >= 0.5.1 # ch02; ch04; ch05
matplotlib >= 3.7.1 # ch04; ch05
tensorflow>=2.18.0; sys_platform != "win32" # ch05 (non-Windows)
tensorflow-cpu>=2.18.0; sys_platform == "win32" # ch05 (Windows)
tqdm >= 4.66.1 # ch05; ch07
numpy >= 1.26, < 2.1 # dependency of several other libraries like torch and pandas
pandas >= 2.2.1 # ch06
psutil >= 5.9.5 # ch07; already installed automatically as dependency of torch
torch >= 2.3.0 # all
jupyterlab >= 4.0 # all
tiktoken >= 0.5.1 # ch02; ch04; ch05
matplotlib >= 3.7.1 # ch04; ch06; ch07
tensorflow>=2.18.0 # ch05; ch06; ch07
tqdm >= 4.66.1 # ch05; ch07
numpy >= 1.26, < 2.1 # dependency of several other libraries like torch and pandas
pandas >= 2.2.1 # ch06
psutil >= 5.9.5 # ch07; already installed automatically as dependency of torch
10 changes: 5 additions & 5 deletions setup/01_optional-python-setup-preferences/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ If it returns 3.10 or newer, no further action is required.

&nbsp;
> [!NOTE]
> I recommend installing a Python version that is at least 1-3 versions older than the most recent release to ensure PyTorch compatibility. For example, if the most recent version is Python 3.13, I recommend installing version 3.10, 3.11, or 3.12.
> I recommend installing a Python version that is at least 2 versions older than the most recent release to ensure PyTorch compatibility. For example, if the most recent version is Python 3.13, I recommend installing version 3.10 or 3.11.

Otherwise, if Python is not installed or is an older version, you can install it for your operating system as described below.

Expand All @@ -62,7 +62,7 @@ Otherwise, if Python is not installed or is an older version, you can install it

```bash
sudo apt update
sudo apt install python3.11 python3.11-venv python3.11-dev
sudo apt install python3.10 python3.10-venv python3.10-dev
```

<br>
Expand Down Expand Up @@ -143,13 +143,13 @@ uv pip install packaging
To install all required packages from a `requirements.txt` file (such as the one located at the top level of this GitHub repository) run the following command, assuming the file is in the same directory as your terminal session:

```bash
uv pip install -U -r requirements.txt
uv pip install -r requirements.txt
```

Alternatively, install the latest dependencies directly from the repository:

```bash
uv pip install -U -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/refs/heads/main/requirements.txt
uv pip install -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/refs/heads/main/requirements.txt
```


Expand Down Expand Up @@ -246,7 +246,7 @@ conda create -n LLMs python=3.10

<img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/setup/01_optional-python-setup-preferences/new-env.png" alt="new-env" width="600px">

> Many scientific computing libraries do not immediately support the newest version of Python. Therefore, when installing PyTorch, it's advisable to use a version of Python that is one to three releases older. For instance, if the latest version of Python is 3.13, using Python 3.10, 3.11, or 3.12 is recommended.
> Many scientific computing libraries do not immediately support the newest version of Python. Therefore, when installing PyTorch, it's advisable to use a version of Python that is one or two releases older. For instance, if the latest version of Python is 3.13, using Python 3.10 or 3.11 is recommended.

Next, activate your new virtual environment (you have to do it every time you open a new terminal window or tab):

Expand Down
2 changes: 2 additions & 0 deletions setup/01_optional-python-setup-preferences/native-uv.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ uv sync --dev --python 3.11
> If you have problems with the following commands above due to certain dependencies (for example, if you are using Windows), you can always fall back to regular pip:
> `uv add pip`
> `uv run python -m pip install -U -r requirements.txt`
>
> Since the TensorFo



Expand Down