Skip to content

[Test Automation] Add schedule job as accepted job for PRs and cron jobs #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

[Test Automation] Add schedule job as accepted job for PRs and cron jobs #270

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a995d91
Test PR Submission - Write Permission
ConnorHack Oct 23, 2024
23fb3d6
Test cron scheduling
ConnorHack Jan 7, 2025
c2da6dc
Remove previous temporary unnecessary commit lines
ConnorHack Jan 9, 2025
21db221
Add schedule job to test runs
ConnorHack Jan 15, 2025
ff7f82a
Set default runner based on repo owner
ConnorHack Jan 15, 2025
9d85614
Allow test summary to show for all event types
ConnorHack Jan 15, 2025
93a540d
Fix test execution based on runner for manual dispatch
ConnorHack Jan 15, 2025
4de2cfe
Merge branch 'meta-llama:main' into main
ConnorHack Jan 27, 2025
568538f
Fix comment blocking
ConnorHack Jan 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 17 additions & 8 deletions .github/workflows/gha_workflow_llama_models_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,13 +63,13 @@ env:

jobs:
execute_workflow:
name: Execute workload on Self-Hosted CPU k8s runner
name: Execute workload on Self-Hosted GPU/CPU k8s runner
permissions:
pull-requests: write
defaults:
run:
shell: bash # default shell to run all steps for a given job.
runs-on: ${{ inputs.runner != '' && inputs.runner || 'llama-models-gha-runner-cpu' }}
runs-on: ${{ inputs.runner != '' && inputs.runner || github.repository_owner == 'meta-llama' && 'llama-models-gha-runner-cpu' || 'llama-models-fork-gha-runner-cpu' }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you comment what these substrings mean? why "fork"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our self-hosted runners are initially set up using Cloud Terraform code. Each runner has distinct attributes, including:

  • A generic label for the runner
  • A specific repository URL

When testing from a fork, a separate runner is required to run independently of the main runner. This is because the forked repository URL differs from the main repository URL, and each runner can only point to one repository.

I adjusted this line so it automatically determines what name to use based on if the repository owner is meta-llama. If it is, then we use the default runner for the main repository. If not, then we use the runner that targets the fork.

if: always()
steps:

Expand Down Expand Up @@ -169,7 +169,7 @@ jobs:

- name: "Installing specific manual_dispatch dependencies"
id: manual_install_pip
if: github.event_name == 'workflow_dispatch'
if: contains(fromJSON('["workflow_dispatch", "schedule"]'), github.event_name)
run: |
echo "[STEP] Installing specific dependencies for manual dispatch workflows"
pip install numpy
Expand Down Expand Up @@ -227,17 +227,26 @@ jobs:
- name: "Manual - Run Tests"
id: manual_run_tests
working-directory: "${{ github.workspace }}"
if: github.event_name == 'workflow_dispatch'
if: contains(fromJSON('["workflow_dispatch", "schedule"]'), github.event_name)
run: |
echo "[STEP] Running PyTest tests at 'GITHUB_WORKSPACE' path: ${GITHUB_WORKSPACE} | path: ${{ github.workspace }}"
free -m
python3 -m pytest --junitxml="${{ github.workspace }}/result.xml"
if [[ "${{ inputs.runner }}" == *"-cpu"* ]]; then
python3 -m pytest --ignore=models/llama3/tests/api/test_generation.py --ignore=llama_models/llama3/tests/api/test_generation.py --junitxml="${{ github.workspace }}/result.xml"
elif [[ "${{ inputs.runner }}" == *"-gpu"* ]]; then
python3 -m pytest --junitxml="${{ github.workspace }}/result.xml"
else
exit 1
fi

#####################
#### ALL TESTING ####
#####################

#### Create test summary ####

- name: "Manual - Test Summary"
- name: "Test Summary"
id: manual_test_summary
if: always() && github.event_name == 'workflow_dispatch'
if: always()
uses: test-summary/action@v2
with:
paths: "${{ github.workspace }}/result.xml"