Skip to content
Draft
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -177,3 +177,33 @@ jobs:
name: yellow-ai-vs-ai-proto-logs
path: |
/tmp/tbots/yellow/proto_*

common-commands:
strategy:
# TODO: set fail-fast to true in production.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remember to remove TODO before merging

fail-fast: false
matrix:
commands:
- ../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/thunderscope:thunderscope_main -- --run_blue --run_diagnostics --interface lo --keyboard_estop --ci_mode
- ../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/ai/hl/stp/tactic/goalie:goalie_tactic_test -- --enable_thunderscope
- ../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/thunderscope:thunderscope_main -- --enable_autoref --ci_mode && ../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/thunderscope:thunderscope_main -- blue_log="$(find /tmp/tbots/blue -maxdepth 1 -type d -name 'proto_*' -printf '/tmp/tbots/blue/%f\n' 2>/dev/null | head -n1)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does the second half of this command do?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay, I am finally done work.

../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/thunderscope:thunderscope_main -- --enable_autoref --ci_mode This first half runs thunderscope once

This part tries to find the previous replay log and display the logs:
../scripts/safe_run.sh bazel run --run_under="xvfb-run" //software/thunderscope:thunderscope_main -- blue_log="$(find /tmp/tbots/blue -maxde4pth 1 -type d -name 'proto_*' -printf '/tmp/tbots/blue/%f\n' 2>/dev/null | head -n1)

This is not ideal, I am trying to think of a way to handle this. Maybe I should allow a new argument for our thunderscope, so we can specify the location for the replay log to save, so that we don't need to lookup for it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah being able to specify the runtime dir + folder name would be ideal


name: Sanity Check on Common Commands
runs-on: ubuntu-20.04
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can use the ubuntu-22.04 runner

steps:
- uses: actions/checkout@v4

- name: Environment Setup
run: |
"${GITHUB_WORKSPACE}"/environment_setup/setup_software.sh

- name: Build Thunderscope
run: |
cd src
bazel build //software/thunderscope:thunderscope_main

- name: Check Common Commands
run: |
cd src
${{ matrix.commands }}

52 changes: 52 additions & 0 deletions scripts/safe_run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
#!/bin/bash
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

safe_run.sh is sort of a weird name that doesn't immediately tell you what this file does

But tbh I don't know a better name... maybe ci_runner.sh is better. At least it tells you that it's only relevant for CI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a high-level description of this script at the top of this file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you re-write this script with python, you can use subprocess and get stdout and stderr logs. Then, you can print the error messages. Because right now, there isn't any feedback for the developers and it will be confusing when this CI job suddenly fails.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, sounds good!
I was trying to use 2> and > to redirect stdout and stderr, but it didn't seem to work.


# Timeout in seconds
# When the time is up and no error was shown, this test will pass
TIME_LIMIT=120 # 2 minutes

# Match Python traceback
ERROR_PATTERN="Traceback (most recent call last):"

# Temporary log file
LOG_FILE=$(mktemp)

# Run the command and record the log
"$@" &> "$LOG_FILE" &
CMD_PID=$!

echo "Process Running in Wrapper with Timeout $TIME_LIMIT ..."

# Time the process
SECONDS=0
while kill -0 $CMD_PID 2>/dev/null; do
# Check if time is up
if [ $SECONDS -ge $TIME_LIMIT ]; then
echo "Time limit reached, stopping process: $CMD_PID"
kill -SIGINT $CMD_PID
wait $CMD_PID
exit 0 # Upon time out and no error, returns 0 status code
fi

# Check if the log contains Traceback
if grep -q "$ERROR_PATTERN" "$LOG_FILE"; then
echo "[Error detected] Potential error found in command output!"
kill -SIGINT $CMD_PID
wait $CMD_PID
exit 1
fi

sleep 1 # Run this loop once per second
done

cat $LOG_FILE
# Get the exit code of the process
wait $CMD_PID
EXIT_CODE=$?

# Clean up log file
rm -f "$LOG_FILE"

# Exit with the command status code
exit $EXIT_CODE


Loading