Adding Linux CI runner setup docs #490

geomin12 · 2025-04-29T23:51:48Z

No description provided.

ScottTodd · 2025-04-30T15:18:35Z

docs/ci_runner_setup/linux/docker_install.sh

This is more than documentation. It should be moved to somewhere more like https://github.com/ROCm/TheRock/tree/main/build_tools/github_action . I've used build_tools/github_actions/runner on other projects (note the plural, "GitHub Actions" is the branding: https://github.com/features/actions)

ScottTodd · 2025-04-30T15:19:29Z

docs/ci_runner_setup/linux/docker_install.sh

+# Add Docker's official GPG key:
+sudo apt-get update
+sudo apt-get install ca-certificates curl -y
+sudo install -m 0755 -d /etc/apt/keyrings
+sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
+sudo chmod a+r /etc/apt/keyrings/docker.asc


Please link to some official documentation for how this is the "official GPG key" or these are the recommended setup steps.

ScottTodd · 2025-04-30T15:20:27Z

docs/ci_runner_setup/linux/docker_install.sh

+  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
+  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \


We may want to support more Linux distributions than Ubuntu at some point. Could put "ubuntu" in the script name somewhere.

ScottTodd · 2025-04-30T15:37:33Z

docs/ci_runner_setup/linux/README.md

+   - If you have a different Linux distribution, follow [ROCm installation quick start guide](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/quick-start.html)
+   - <b>After reboot, please try `rocminfo` and `rocm-smi` to make sure ROCm is loaded and drivers are installed.</b> If there are issues, please try each command in `rocm_install.sh` instead.
+
+1. If docker is not installed, please run `sudo ./docker_install.sh`. This script will download docker for Ubuntu.


Do all the commands in the script need sudo if you always run the script itself with sudo?

ScottTodd · 2025-04-30T15:39:10Z

docs/ci_runner_setup/linux/README.md

+For brand new machines that do not that ROCm or Docker installed, please follow these steps. Otherwise, please skip to step 3.
+
+1. Install ROCm to the machine using `sudo ./rocm_install.sh`. This script will install ROCm 6.4 and AMD drivers for Ubuntu24, then it will reboot the system.
+
+   - Rebooting the system is required to load ROCm.
+   - If you have a different Linux distribution, follow [ROCm installation quick start guide](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/quick-start.html)
+   - <b>After reboot, please try `rocminfo` and `rocm-smi` to make sure ROCm is loaded and drivers are installed.</b> If there are issues, please try each command in `rocm_install.sh` instead.


Why are we installing ROCm on machines? TheRock is ROCm, so we should be building whatever we need as part of our build/test/release workflows. If we need something specific for bootstrapping, let's extract that instead of pulling the full SDK down from some fixed older release.

When setting up the scripts (particuarly for ROCR_VISIBLE_DEVICES) and debugging the machines, it's quite useful to have rocminfo and rocm-smi around. However, it's only used for those commands and the entire other rocm items aren't used.

Should we just use a version of TheRock during setup, figure out which GPUs is what, then remove it? so that way the machines have a fresh system with no ROCm installed?

I would go with the latter. We can pin a known green commit in a config file and either pull a release tarball or the specific artifacts that contain rocm-smi and rocminfo. We could also bundle only those two pre-build binaries in an extra package if it helps.

Can you elaborate on what is needed at runner setup time with visible devices?

In order of my preference:

No tool dependency for system setup, or use some standard Linux tooling

Minimal set of tools, bootstrapped from a stable release of TheRock

Minimal set of tools, bootstrapped from existing ROCm releases (maybe mirrored to S3)

apt install as here

I wouldn't trust test runners if we install ROCm (TheRock, community build) on top of an existing ROCm (non-community build) install. If we run setup on the host and then runners under Docker, that might be safer though.

ScottTodd · 2025-04-30T15:41:18Z

docs/ci_runner_setup/linux/runner_setup_1.sh

+
+# runner setup
+mkdir "actions-runner-$1" && cd "actions-runner-$1"
+curl -o actions-runner-linux-x64-2.323.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.323.0/actions-runner-linux-x64-2.323.0.tar.gz


Can you make the runner version an argument to the script, or do something to select the latest?

ScottTodd · 2025-04-30T15:42:00Z

docs/ci_runner_setup/linux/runner_setup_2.sh

+# svc install
+sudo ./svc.sh install root


What is svc? Where is this svc.sh script coming from?

ScottTodd · 2025-04-30T15:42:32Z

docs/ci_runner_setup/linux/README.md

+
+### Setup
+
+For brand new machines that do not that ROCm or Docker installed, please follow these steps. Otherwise, please skip to step 3.


Grammar/typo errors in this line, but also see my other comments about ROCm installs.

ScottTodd · 2025-04-30T15:43:36Z

docs/ci_runner_setup/linux/README.md

+   ```
+   cd {IDENTIFIER_FROM_STEP_3}
+   ./config.sh --url https://github.com/ROCm --token {TOKEN} --no-default-labels --labels {LABEL}
+   ```


nit: set the language for fenced code blocks when know to help with syntax highlighting (auto detection sometimes guesses correctly)

Suggested change

```

cd {IDENTIFIER_FROM_STEP_3}

./config.sh --url https://github.com/ROCm --token {TOKEN} --no-default-labels --labels {LABEL}

```

```bash

cd {IDENTIFIER_FROM_STEP_3}

./config.sh --url https://github.com/ROCm --token {TOKEN} --no-default-labels --labels {LABEL}

ScottTodd · 2025-04-30T15:44:30Z

docs/ci_runner_setup/linux/README.md

+1. After the runner packages are there, please follow these steps and run the commands:
+
+   - Please retrieve token from [ROCm GitHub runner page](https://github.com/organizations/ROCm/settings/actions/runners/new?arch=x64&os=linux) in the `Configure` tab.
+   - Please add an unique identifying label for this CI runner. Example: Linux gfx1201 -> label `linux-gfx1201-gpu-rocm`. This is the label that will be used in workflows and will be shared amongst other identical machines.


Could provide a list of all current labels for reference instead of a single example. Maybe a link to the runner page would be sufficient, but a table with more information here would be useful for those without access to that page.

marbre · 2025-05-02T06:58:43Z

build_tools/github_action/runner/linux/runner_setup_2.sh

+sudo apt install gfortran git git-lfs ninja-build cmake g++ pkg-config xxd patchelf automake python3-venv python3-dev libegl1-mesa-dev
+
+# svc install
+# This script comes from GitHub action runner tar file


Maybe

Suggested change

# This script comes from GitHub action runner tar file

# This script comes from GitHub actions runner release tarball

? Furthermore, I thought that this only has runsvc.sh, but I might be wrong.

That may have been a previous script! currently, it has ./svc.sh as the script (must have updated the name?)

ScottTodd · 2025-05-09T17:59:35Z

build_tools/github_action/runner/linux/rocm_install.sh

+# ROCm install
+wget https://repo.radeon.com/amdgpu-install/6.4/ubuntu/noble/amdgpu-install_6.4.60400-1_all.deb


Marking "reviewed" until we resolve the rocm install questions. We could punt on that if the existing runners are already doing this and this is just checking in the configuration.

geomin12 added 2 commits April 29, 2025 16:48

Adding docs

ad34fe5

Adding +x permissions

ab7e849

github-project-automation bot added this to TheRock Triage Apr 29, 2025

github-project-automation bot moved this to TODO in TheRock Triage Apr 29, 2025

pre-commit

62b62c2

ScottTodd requested changes Apr 30, 2025

View reviewed changes

geomin12 added 2 commits May 1, 2025 11:20

pr comments

c2f817d

lint

d591f18

geomin12 requested a review from ScottTodd May 1, 2025 18:30

marbre reviewed May 2, 2025

View reviewed changes

PR comment

2a58e8b

geomin12 requested a review from marbre May 2, 2025 16:56

amd-chrissosa self-assigned this May 8, 2025

ScottTodd reviewed May 9, 2025

View reviewed changes

		"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
		$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" \| \


		### Setup

		For brand new machines that do not that ROCm or Docker installed, please follow these steps. Otherwise, please skip to step 3.

	# This script comes from GitHub action runner tar file
	# This script comes from GitHub actions runner release tarball

		# ROCm install
		wget https://repo.radeon.com/amdgpu-install/6.4/ubuntu/noble/amdgpu-install_6.4.60400-1_all.deb

Adding Linux CI runner setup docs #490

Are you sure you want to change the base?

Adding Linux CI runner setup docs #490

Uh oh!

Conversation

geomin12 commented Apr 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!