Skip to content

Process replay: move refs to ci-artifacts #37049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
elkoled wants to merge 11 commits into
base: master
Choose a base branch
from
+29 −15
Open

Process replay: move refs to ci-artifacts #37049

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f69e629
rm upload
elkoled Jan 31, 2026
4fc2fec
use ci-artifacts
elkoled Jan 31, 2026
75a02dc
sanitize
elkoled Jan 31, 2026
89b268b
rm ref_commit
elkoled Jan 31, 2026
dc78d74
add ci
elkoled Jan 31, 2026
3737111
handle exept
elkoled Jan 31, 2026
327dbc4
bootstrap
elkoled Jan 31, 2026
0a9e45d
always
elkoled Jan 31, 2026
4cffe72
fix
elkoled Jan 31, 2026
cdd70f4
replay
elkoled Jan 31, 2026
4c79fd1
keep ref_commit fork compatibility
elkoled Jan 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 23 additions & 5 deletions .github/workflows/tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@ concurrency:
env:
PYTHONWARNINGS: error
BASE_IMAGE: openpilot-base
AZURE_TOKEN: ${{ secrets.AZURE_COMMADATACI_OPENPILOTCI_TOKEN }}

DOCKER_LOGIN: docker login ghcr.io -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }}
BUILD: selfdrive/test/docker_build.sh base

Expand Down Expand Up @@ -185,7 +183,7 @@ jobs:
uses: actions/cache@v4
with:
path: .ci_cache/comma_download_cache
key: proc-replay-${{ hashFiles('selfdrive/test/process_replay/ref_commit', 'selfdrive/test/process_replay/test_processes.py') }}
key: proc-replay-${{ hashFiles('selfdrive/test/process_replay/test_processes.py') }}
- name: Build openpilot
run: |
${{ env.RUN }} "scons -j$(nproc)"
Expand All @@ -205,15 +203,35 @@ jobs:
name: process_replay_diff.txt
path: selfdrive/test/process_replay/diff.txt
- name: Upload reference logs
if: false # TODO: move this to github instead of azure
if: github.repository == 'commaai/openpilot' && github.ref == 'refs/heads/master'
run: |
${{ env.RUN }} "unset PYTHONWARNINGS && AZURE_TOKEN='$AZURE_TOKEN' python3 selfdrive/test/process_replay/test_processes.py -j$(nproc) --upload-only"
${{ env.RUN }} "selfdrive/test/process_replay/test_processes.py -j$(nproc) --upload-only"
Copy link
Contributor

@sshane sshane Feb 4, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this still here if it doesn't actually do anything, or does it? Looks like two "push refs" steps

- name: Run regen
if: false
timeout-minutes: 4
run: |
${{ env.RUN }} "ONNXCPU=1 $PYTEST selfdrive/test/process_replay/test_regen.py && \
chmod -R 777 /tmp/comma_download_cache"
- name: Checkout ci-artifacts
if: github.repository == 'commaai/openpilot' && github.ref == 'refs/heads/master'
uses: actions/checkout@v4
with:
repository: commaai/ci-artifacts
ssh-key: ${{ secrets.CI_ARTIFACTS_DEPLOY_KEY }}
path: ${{ github.workspace }}/ci-artifacts
- name: Push refs
if: github.repository == 'commaai/openpilot' && github.ref == 'refs/heads/master'
working-directory: ${{ github.workspace }}/ci-artifacts
run: |
git checkout --orphan process-replay
git rm -rf *
git config user.name "GitHub Actions Bot"
git config user.email "<>"
cp ${{ github.workspace }}/selfdrive/test/process_replay/fakedata/*.zst .
echo "${{ github.sha }}" > ref_commit
git add .
git commit -m "process-replay refs for ${{ github.sha }}"
git push origin process-replay --force

simulator_driving:
name: simulator driving
Expand Down
1 change: 0 additions & 1 deletion selfdrive/test/process_replay/ref_commit
View file
Open in desktop

This file was deleted.

15 changes: 6 additions & 9 deletions selfdrive/test/process_replay/test_processes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,13 @@

from opendbc.car.car_helpers import interface_names
from openpilot.common.git import get_commit
from openpilot.tools.lib.openpilotci import get_url, upload_file
from openpilot.tools.lib.openpilotci import get_url
from openpilot.selfdrive.test.process_replay.compare_logs import compare_logs, format_diff
from openpilot.selfdrive.test.process_replay.process_replay import CONFIGS, PROC_REPLAY_DIR, FAKEDATA, replay_process, \
check_most_messages_valid
from openpilot.tools.lib.filereader import FileReader
from openpilot.tools.lib.logreader import LogReader, save_log
from openpilot.tools.lib.url_file import URLFile

source_segments = [
("HYUNDAI", "02c45f73a2e5c6e9|2021-01-01--19-08-22--1"), # HYUNDAI.HYUNDAI_SONATA
Expand Down Expand Up @@ -64,7 +65,7 @@
# dashcamOnly makes don't need to be tested until a full port is done
excluded_interfaces = ["mock", "body", "psa"]

BASE_URL = "https://commadataci.blob.core.windows.net/openpilotci/"
BASE_URL = "https://raw.githubusercontent.com/commaai/ci-artifacts/refs/heads/process-replay/"
REF_COMMIT_FN = os.path.join(PROC_REPLAY_DIR, "ref_commit")
Copy link
Contributor

@sshane sshane Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

file is deleted but we use it?

EXCLUDED_PROCS = {"modeld", "dmonitoringmodeld"}

Expand All @@ -79,10 +80,7 @@ def run_test_process(data):
save_log(cur_log_fn, log_msgs)

if args.update_refs or args.upload_only:
print(f'Uploading: {os.path.basename(cur_log_fn)}')
assert os.path.exists(cur_log_fn), f"Cannot find log to upload: {cur_log_fn}"
upload_file(cur_log_fn, os.path.basename(cur_log_fn))
os.remove(cur_log_fn)
Comment on lines 82 to -85
Copy link
Contributor

@sshane sshane Feb 4, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to keep this file aware of uploading if it doesn't upload anymore?

Copy link
Contributor

@sshane sshane Feb 4, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to think of a reason to keep the uploading in this file so you can run locally and not just in CI. Safe to remove or can you think of any good reason?

return (segment, cfg.proc_name, res)


Expand Down Expand Up @@ -163,8 +161,7 @@ def test_process(cfg, lr, segment, ref_log_path, new_log_path, ignore_fields=Non
with open(REF_COMMIT_FN) as f:
ref_commit = f.read().strip()
except FileNotFoundError:
print("Couldn't find reference commit")
sys.exit(1)
ref_commit = URLFile(BASE_URL + "ref_commit", cache=False).read().decode().strip()
Comment on lines 161 to +164
Copy link
Contributor

@sshane sshane Feb 4, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same, won't it always hit except? if it's for you to manually test against a commit, should be an argument not file


cur_commit = get_commit()
if not cur_commit:
Expand Down Expand Up @@ -199,11 +196,11 @@ def test_process(cfg, lr, segment, ref_log_path, new_log_path, ignore_fields=Non
if cfg.proc_name not in ('card', 'controlsd', 'lagd') and car_brand not in ('HYUNDAI', 'TOYOTA'):
continue

cur_log_fn = os.path.join(FAKEDATA, f"{segment}_{cfg.proc_name}_{cur_commit}.zst")
cur_log_fn = os.path.join(FAKEDATA, f"{segment}_{cfg.proc_name}_{cur_commit}.zst".replace("|", "_"))
if args.update_refs: # reference logs will not exist if routes were just regenerated
ref_log_path = get_url(*segment.rsplit("--", 1,), "rlog.zst")
else:
ref_log_fn = os.path.join(FAKEDATA, f"{segment}_{cfg.proc_name}_{ref_commit}.zst")
ref_log_fn = os.path.join(FAKEDATA, f"{segment}_{cfg.proc_name}_{ref_commit}.zst".replace("|", "_"))
ref_log_path = ref_log_fn if os.path.exists(ref_log_fn) else BASE_URL + os.path.basename(ref_log_fn)

dat = None if args.upload_only else log_data[segment]
Expand Down
Loading