cicd: add a github workflow for xfails report script #2273
Conversation
|
Note Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported. |
📝 WalkthroughWalkthroughA new GitHub Actions workflow is added that runs weekly to execute a Python script, generate an xfails report, detect changes, and automatically create a pull request if updates are found. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks✅ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
I haven't tested this out yet (I think it needs to be merged first to run the workflow). But it shouldn't interfere with any pre existing code |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
.github/workflows/update-xfails-report.yml (1)
32-38: Consider separating error output from the report.Redirecting stderr to the report file (
2>&1) means any error messages or warnings will be mixed with the actual report content. If the script fails partway through, the report file could contain partial output plus error messages.🔎 Optional refactor for cleaner error handling
- name: Run XFails tracker run: | - python scripts/xfails_tracker.py > reports/xfails_report.txt 2>&1 || { + python scripts/xfails_tracker.py > reports/xfails_report.txt || { echo "Error: XFails tracker script failed" - cat reports/xfails_report.txt + echo "Script output:" + cat reports/xfails_report.txt 2>/dev/null || echo "(no output file generated)" exit 1 }This keeps stderr visible in the workflow logs separately from the report file.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
.github/workflows/update-xfails-report.yml
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Deploy Docs
🔇 Additional comments (5)
.github/workflows/update-xfails-report.yml (5)
1-7: LGTM: Clear workflow configuration.The workflow name is descriptive, and the triggers are well-configured with both scheduled (weekly on Mondays) and manual execution options.
9-11: LGTM: Appropriate permissions.The permissions are correctly scoped for creating branches and pull requests.
13-16: LGTM: Sensible job configuration.The ubuntu-latest runner and 30-minute timeout are appropriate for this reporting task.
29-30: LGTM: Safe directory creation.The
-pflag correctly handles the case where the directory already exists.
40-57: LGTM: Robust change detection.The logic correctly handles both new and modified files, and uses the modern
$GITHUB_OUTPUTsyntax.
| - name: Checkout repository | ||
| uses: actions/[email protected] | ||
| with: | ||
| fetch-depth: 1 # Shallow clone is sufficient for scanning test files | ||
| token: ${{ secrets.GITHUB_TOKEN }} |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
latest version of actions/checkout GitHub Action 2025
💡 Result:
The latest major version is v6 — use actions/checkout@v6. [1][2]
Sources:
[1] actions/checkout GitHub repository README.
[2] Checkout page on the GitHub Marketplace.
Update actions/checkout to the latest version.
The version v4.2.2 is outdated. The latest stable version is v6. Update to actions/checkout@v6 to ensure security patches and bug fixes are included.
🤖 Prompt for AI Agents
.github/workflows/update-xfails-report.yml around lines 18 to 22: the workflow
pins actions/checkout to v4.2.2 which is outdated; update the action reference
to actions/checkout@v6 by replacing the version string so the workflow uses the
latest stable v6 release, keeping the existing with: block (fetch-depth and
token) intact.
| - name: Create Pull Request | ||
| if: steps.check_changes.outputs.changed == 'true' | ||
| uses: peter-evans/create-pull-request@v7 | ||
| with: | ||
| token: ${{ secrets.FLASHINFER_BOT_TOKEN }} | ||
| committer: flashinfer-bot <[email protected]> | ||
| author: flashinfer-bot <[email protected]> | ||
| commit-message: | | ||
| chore: update xfails report | ||
|
|
||
| Auto-generated xfails report based on current test suite markers. | ||
|
|
||
| 🤖 Generated with [Claude Code](https://claude.com/claude-code) | ||
|
|
||
| Co-Authored-By: Claude <[email protected]> | ||
| branch: auto-update-xfails-report | ||
| base: main | ||
| delete-branch: true | ||
| title: 'chore: Update XFails Report' | ||
| body: | | ||
| ## Summary | ||
|
|
||
| This PR updates the xfails report based on the current test suite. | ||
|
|
||
| ## Changes | ||
|
|
||
| - Updated `reports/xfails_report.txt` with current xfail markers from the test suite | ||
|
|
||
| ## How to Review | ||
|
|
||
| 1. Review the changes to `reports/xfails_report.txt` | ||
| 2. Check if the number of xfails has increased or decreased | ||
| 3. Review the reasons for any new xfails | ||
| 4. Consider addressing xfails that may no longer be needed | ||
|
|
||
| ## Notes | ||
|
|
||
| - This is an automated PR generated weekly | ||
| - The report shows all `pytest.mark.xfail` markers in the test suite | ||
| - Grouped by reason for easy analysis | ||
|
|
||
| --- | ||
|
|
||
| 🤖 This PR was automatically generated by the [update-xfails-report workflow](.github/workflows/update-xfails-report.yml) | ||
| labels: | | ||
| automated | ||
| maintenance | ||
| testing |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
peter-evans/create-pull-request latest version GitHub Action
💡 Result:
The latest major release is v8 — use peter-evans/create-pull-request@v8 (you can pin to a specific patch like @v8.x.x). [1][2]
Sources:
[1] GitHub repo README (peter-evans/create-pull-request).
[2] GitHub Marketplace entry for Create Pull Request.
Upgrade peter-evans/create-pull-request to v8.
Update the action version from @v7 to @v8, which is the latest major release. The rest of the PR creation step is well-structured with appropriate bot attribution, comprehensive PR body, and automatic branch cleanup.
🤖 Prompt for AI Agents
.github/workflows/update-xfails-report.yml around lines 59 to 106: the workflow
uses peter-evans/create-pull-request@v7 and needs to be upgraded to @v8; update
the uses line to reference peter-evans/create-pull-request@v8 and ensure any
breaking-change inputs required by v8 are handled (verify token, committer,
author, commit-message, branch, base, delete-branch, title, body, and labels
inputs remain valid in v8 and adjust names or formats if the action changed
them).
| 🤖 Generated with [Claude Code](https://claude.com/claude-code) | ||
|
|
||
| Co-Authored-By: Claude <[email protected]> |
There was a problem hiding this comment.
No need to add these (we should also remove them from the other workflows).
2 participants
📌 Description
Adds workflow that runs weekly on Mondays (and can be triggered manually) to execute scripts/xfails_tracker.py, which scans the test suite for xfail markers and outputs a comprehensive report to reports/xfails_report.txt. If the report has changed, it automatically creates a pull request to commit the updated report to the repository.
🔍 Related Issues
cont. testing item from november roadmap
🚀 Pull Request Checklist
Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete.
✅ Pre-commit Checks
pre-commitby runningpip install pre-commit(or used your preferred method).pre-commit install.pre-commit run --all-filesand fixed any reported issues.🧪 Tests
unittest, etc.).Reviewer Notes
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.