Introduce tests for artifact structure (#3802)

## Motivation

Progress on #3796. This adds a
more comprehensive set of tests using the actual built artifacts, unlike
the unit test in #3765 that just
does static analysis of the artifact descriptors (toml files).

> [!IMPORTANT]
> The tests are currently FAILING, so I didn't hook them up to any
workflows yet.
>
> Proposed plan:
> 1. (This PR) Land the tests, runnable via workflow_dispatch
> 2. Fix the artifact definitions / population code to the tests start
passing
> 3. Integrate the tests into existing CI workflows

Running these tests continuously will guard against future regressions
that would otherwise go unnoticed or be difficult to debug (race
conditions!).

## Technical Details

Files in artifacts should be unique across all artifacts. If they are
not, we use extra storage and can run into race conditions during
artifact extraction/flattening.

This adds two key tests:
1. Cross-artifact collision testing (e.g. 'blas' vs 'support')
2. Within-artifact collision testing (e.g. 'blas_dev' vs 'blas_test')

## Test Plan

* Fetch various sets of artifacts with `fetch_artifacts.py` locally and
run the tests locally
    ```bash
set
THEROCK_ARTIFACTS_DIR=D:/scratch/claude/artifacts/22681848469-linux-gfx94X-dcgpu
    python -m pytest tests/test_artifact_structure.py -v
    ```
* Triggered test runs with workflow_dispatch in my fork

## Test Result

| Run | Format | Fetch | Validate | Result |
|---|---|---|---|---|
|
[22743296445](https://github.com/ScottTodd/TheRock/actions/runs/22743296445)
| classic `.tar.xz` | 74s | 169s | 2 FAILED |
|
[22743345907](https://github.com/ScottTodd/TheRock/actions/runs/22743345907)
| kpack `.tar.zst` | 12s | 18s | 2 FAILED |

> [!NOTE]
> * These are on github-hosted runners which may have slower network
access to S3 than our eventual S3 hosted runners
> * The multi-arch tests using .zst are much faster than the classic
tests using .xz

Cross-artifact logs snippet:
```
E           Failed: Found 29 cross-artifact collision(s) across 193 archives (see #3796):
E             include/mxDataGenerator/DataGenerator.hpp
E               - blas (blas_dev_gfx94X-dcgpu.tar.xz)
E               - support (support_dev_generic.tar.xz)
E             include/mxDataGenerator/PreSwizzle.hpp
E               - blas (blas_dev_gfx94X-dcgpu.tar.xz)
E               - support (support_dev_generic.tar.xz)
```

(27 files in `include/mxDataGenerator/*`, 9 files in
`lib/cmake/mxDataGenerator/*`, all shared between `blas_dev` and
`support_dev`)

Within-artifact logs snippet:
```
E           Failed: Found within-artifact component collisions in 10 artifact(s) (4682 total files). Components should be disjoint (see #3796):
E             base (33 files):
E               share/rocprofiler-register/tests/CMakeLists.txt  [run, test]
E               share/rocprofiler-register/tests/amdhip/CMakeLists.txt  [run, test]
E               share/rocprofiler-register/tests/amdhip/amdhip.cpp  [run, test]
E               share/rocprofiler-register/tests/amdhip/amdhip.hpp  [run, test]
E               share/rocprofiler-register/tests/bin/CMakeLists.txt  [run, test]
E               ... and 28 more
E             blas (1 files):
E               share/hipsparse/test/hipsparse_clientmatrices.cmake  [dev, test]
E             core-ocl (2 files):
E               share/opencl/ocltst/liboclperf.so  [lib, test]
E               share/opencl/ocltst/liboclruntime.so  [lib, test]
```

Within-artifact summary:
```
  ┌─────────────────────┬─────────────┬───────────────────┐
  │      Artifact       │ Components  │ Overlapping files │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ rocprofiler-compute │ lib vs test │ 4,139             │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ rocprofiler-sdk     │ run vs test │ 452               │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ rocprofiler-sdk     │ lib vs test │ 10                │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ base                │ run vs test │ 33                │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ rocprofiler-compute │ run vs test │ 22                │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ rocrtst             │ run vs test │ 12                │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ rdc                 │ lib vs test │ 4                 │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ rocdecode           │ dev vs test │ 3                 │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ rocgdb              │ dev vs test │ 2                 │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ core-ocl            │ lib vs test │ 2                 │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ blas                │ dev vs test │ 1                 │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ miopen              │ run vs test │ 1                 │
  ├─────────────────────┼─────────────┼───────────────────┤
  │ rocprofiler-compute │ doc vs test │ 1                 │
  └─────────────────────┴─────────────┴───────────────────┘
```

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

---------

Co-authored-by: Claude <[email protected]>

Author: ScottTodd

.github/workflows/test_artifacts_structure.yml

@@ -0,0 +1,74 @@
+# Copyright Advanced Micro Devices, Inc.
+# SPDX-License-Identifier: MIT
+
+name: Test Artifact Structure
+
+on:
+  workflow_dispatch:
+    inputs:
+      artifact_group:
+        type: string
+        description: "Artifact group (e.g. gfx94X-dcgpu, gfx110X-all)"
+        required: true
+      amdgpu_targets:
+        type: string
+        description: "Individual GPU targets for split artifacts (e.g. gfx942, gfx1100)"
+        default: ""
+      artifact_run_id:
+        type: string
+        description: "Run ID to fetch artifacts from (defaults to current run)"
+        default: ""
+  workflow_call:
+    inputs:
+      artifact_group:
+        type: string
+        required: true
+      amdgpu_targets:
+        type: string
+        default: ""
+      artifact_run_id:
+        type: string
+        default: ""
+
+permissions:
+  contents: read
+
+jobs:
+  test_artifact_structure:
+    name: "Validate artifact structure (${{ inputs.artifact_group }})"
+    runs-on: ${{ github.repository_owner == 'ROCm' && 'azure-linux-scale-rocm' || 'ubuntu-24.04' }}
+    env:
+      ARTIFACT_RUN_ID: "${{ inputs.artifact_run_id != '' && inputs.artifact_run_id || github.run_id }}"
+      ARTIFACT_GROUP: ${{ inputs.artifact_group }}
+      AMDGPU_TARGETS: ${{ inputs.amdgpu_targets }}
+      ARTIFACTS_DIR: ${{ github.workspace }}/artifacts
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Setting up Python
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+        with:
+          python-version: "3.12"
+
+      - name: Install test requirements
+        run: |
+          pip install -r requirements-test.txt
+
+      - name: Fetch artifact archives (no extract)
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+        run: |
+          python ./build_tools/fetch_artifacts.py \
+            --run-id=${ARTIFACT_RUN_ID} \
+            --artifact-group=${ARTIFACT_GROUP} \
+            --amdgpu-targets=${AMDGPU_TARGETS} \
+            --output-dir=${ARTIFACTS_DIR} \
+            --no-extract
+
+      - name: Validate artifact structure
+        env:
+          THEROCK_ARTIFACTS_DIR: ${{ env.ARTIFACTS_DIR }}
+        run: |
+          python -m pytest tests/test_artifact_structure.py \
+            -v --log-cli-level=info --timeout=300

tests/test_artifact_structure.py

@@ -0,0 +1,310 @@
+# Copyright Advanced Micro Devices, Inc.
+# SPDX-License-Identifier: MIT
+
+"""Structural validation of artifact archives.
+
+These tests scan artifact archives WITHOUT extracting them and check invariants
+that the build system should maintain. They run on CPU (no GPU required) and
+are designed to catch issues before artifacts are installed or tested on GPU
+runners.
+
+Tests:
+  - Cross-artifact overlaps: no two artifacts should produce files that
+    flatten to the same path (causes silent overwrites or race conditions,
+    see https://github.com/ROCm/TheRock/issues/3758).
+  - Within-artifact component overlaps: different components (lib, run,
+    test, etc.) of the same artifact should contain disjoint files (the
+    component scanner should enforce this via the extends chain).
+  - Manifest validation: every archive should have artifact_manifest.txt
+    as its first member.
+
+Usage:
+    THEROCK_ARTIFACTS_DIR=/path/to/archives \\
+        python -m pytest tests/test_artifact_structure.py -v --log-cli-level=info
+
+THEROCK_ARTIFACTS_DIR should point to a directory containing artifact archives
+(*.tar.zst or *.tar.xz) as produced by the build pipeline.
+"""
+
+import dataclasses
+import logging
+import os
+import sys
+from collections import defaultdict
+from pathlib import Path
+
+import pytest
+
+THIS_DIR = Path(__file__).resolve().parent
+sys.path.insert(0, str(THIS_DIR.parent / "build_tools"))
+
+from _therock_utils.artifacts import ArtifactName, _open_archive_for_read
+
+logger = logging.getLogger(__name__)
+
+ARTIFACTS_DIR = os.getenv("THEROCK_ARTIFACTS_DIR", "")
+
+
+@pytest.fixture(scope="session")
+def artifacts_dir() -> Path:
+    if not ARTIFACTS_DIR:
+        pytest.skip("THEROCK_ARTIFACTS_DIR not set")
+    path = Path(ARTIFACTS_DIR).resolve()
+    if not path.is_dir():
+        pytest.fail(f"THEROCK_ARTIFACTS_DIR is not a directory: {path}")
+    return path
+
+
+def list_archive_files(archive_path: Path) -> tuple[list[str], list[str]]:
+    """List the flattened file paths in an artifact archive.
+
+    Reads artifact_manifest.txt (must be the first tar member) to get basedir
+    prefixes, then strips those prefixes from each file member to produce
+    the flattened output path.
+
+    Returns:
+        (manifest_prefixes, flattened_file_paths)
+    """
+    prefixes: list[str] = []
+    flattened: list[str] = []
+
+    with _open_archive_for_read(archive_path) as tf:
+        # Extract prefixes from the artifact manifest.
+        manifest_member = tf.next()
+        if manifest_member is None or manifest_member.name != "artifact_manifest.txt":
+            raise ValueError(
+                f"{archive_path.name}: expected artifact_manifest.txt as first member, "
+                f"got {manifest_member.name if manifest_member else 'empty archive'}"
+            )
+        with tf.extractfile(manifest_member) as mf:
+            # The artifact_manifest.txt will contain prefix lines like:
+            #   math-libs/BLAS/hipBLAS-common/stage
+            #   math-libs/BLAS/rocRoller/stage
+            #   math-libs/BLAS/hipBLAS/stage
+            prefixes = [
+                line for line in mf.read().decode().splitlines() if line.strip()
+            ]
+
+        # Strip prefixes from each file member to produce flattened paths.
+        # For example:
+        #   prefix:     math-libs/BLAS/hipBLAS/stage
+        #   member:     math-libs/BLAS/hipBLAS/stage/include/hipblas/hipblas.h
+        #   flattened:  include/hipblas/hipblas.h
+        while member := tf.next():
+            if member.isdir():
+                continue
+            name = member.name
+            for prefix in prefixes:
+                prefix_slash = prefix + "/"
+                if name.startswith(prefix_slash):
+                    flattened_path = name[len(prefix_slash) :]
+                    if flattened_path:
+                        flattened.append(flattened_path)
+                    break
+
+    return prefixes, flattened
+
+
+def discover_archives(artifacts_dir: Path) -> list[Path]:
+    """Find all artifact archives in a directory."""
+    archives = []
+    for ext in ("*.tar.zst", "*.tar.xz"):
+        archives.extend(artifacts_dir.glob(ext))
+    return sorted(archives)
+
+
+@dataclasses.dataclass
+class ArchiveInfo:
+    """Metadata and flattened file listing for a single artifact archive."""
+
+    artifact_name: str
+    component: str
+    filename: str
+    flattened_paths: set[str]
+
+
+@dataclasses.dataclass
+class CrossArtifactOverlap:
+    """A flattened file path found in multiple artifacts."""
+
+    path: str
+    sources: list[tuple[str, str]]  # (artifact_name, archive_filename)
+
+
+@dataclasses.dataclass
+class ComponentOverlap:
+    """Files duplicated across components within one artifact."""
+
+    artifact_name: str
+    overlaps: dict[str, list[str]]  # path -> [component_name, ...]
+
+
+@pytest.fixture(scope="session")
+def archive_index(artifacts_dir: Path) -> list[ArchiveInfo]:
+    """Scan all archives once and return a flat list of ArchiveInfo."""
+    archives = discover_archives(artifacts_dir)
+    if not archives:
+        pytest.skip(f"No artifact archives found in {artifacts_dir}")
+
+    index: list[ArchiveInfo] = []
+    skipped: list[str] = []
+
+    for archive_path in archives:
+        archive_name = archive_path.name
+        an = ArtifactName.from_filename(archive_name)
+        if an is None:
+            logger.warning("Skipping unrecognized archive: %s", archive_name)
+            skipped.append(archive_name)
+            continue
+
+        logger.info("Listing %s ...", archive_name)
+        try:
+            _prefixes, flattened_paths = list_archive_files(archive_path)
+        except Exception:
+            logger.exception("Failed to read %s", archive_name)
+            skipped.append(archive_name)
+            continue
+
+        index.append(
+            ArchiveInfo(
+                artifact_name=an.name,
+                component=an.component,
+                filename=archive_name,
+                flattened_paths=set(flattened_paths),
+            )
+        )
+
+    if skipped:
+        logger.warning("Skipped %d archives: %s", len(skipped), skipped)
+
+    artifact_names = {a.artifact_name for a in index}
+    logger.info(
+        "Indexed %d archives across %d artifacts", len(index), len(artifact_names)
+    )
+    return index
+
+
+def _format_cross_artifact_overlaps(overlaps: list[CrossArtifactOverlap]) -> str:
+    """Format cross-artifact overlaps into readable summary."""
+    lines = []
+    for overlap in sorted(overlaps, key=lambda o: o.path):
+        lines.append(f"  {overlap.path}")
+        for label, archive in sorted(overlap.sources):
+            lines.append(f"    - {label} ({archive})")
+    return "\n".join(lines)
+
+
+def _format_component_overlaps(overlaps: list[ComponentOverlap]) -> str:
+    """Format within-artifact component overlaps into readable summary."""
+    lines = []
+    total = 0
+    for overlap in sorted(overlaps, key=lambda o: o.artifact_name):
+        total += len(overlap.overlaps)
+        lines.append(f"  {overlap.artifact_name} ({len(overlap.overlaps)} files):")
+        for fpath in sorted(overlap.overlaps):
+            comps = sorted(set(overlap.overlaps[fpath]))
+            lines.append(f"    {fpath}  [{', '.join(comps)}]")
+    return total, "\n".join(lines)
+
+
+class TestArtifactStructure:
+    """Structural validation of artifact archives."""
+
+    def test_no_cross_artifact_overlaps(self, archive_index: list[ArchiveInfo]):
+        """No two artifacts should contain files that flatten to the same path.
+
+        This catches both same-basedir overlaps (like #3758) and the subtler
+        cross-basedir case where different stage dirs install identically-named
+        files (e.g., two subprojects both installing "bin/sequence.yaml").
+
+        See https://github.com/ROCm/TheRock/issues/3796
+        """
+        # For each flattened path, track which artifacts contain it.
+        # flattened_path -> { artifact_name: set of archive filenames }
+        path_artifacts: dict[str, dict[str, set[str]]] = defaultdict(
+            lambda: defaultdict(set)
+        )
+
+        for info in archive_index:
+            for fpath in info.flattened_paths:
+                path_artifacts[fpath][info.artifact_name].add(info.filename)
+
+        # A cross-artifact overlap is a path claimed by more than one artifact
+        # (e.g. "blas", "support").
+        overlaps: list[CrossArtifactOverlap] = []
+        for fpath, artifact_archives in path_artifacts.items():
+            if len(artifact_archives) > 1:
+                sources = []
+                for artifact_name, archives in artifact_archives.items():
+                    for archive in archives:
+                        sources.append((artifact_name, archive))
+                overlaps.append(CrossArtifactOverlap(path=fpath, sources=sources))
+
+        if overlaps:
+            summary = _format_cross_artifact_overlaps(overlaps)
+            pytest.fail(
+                f"Found {len(overlaps)} cross-artifact overlap(s) across "
+                f"{len(archive_index)} archives "
+                f"(see https://github.com/ROCm/TheRock/issues/3796):\n{summary}"
+            )
+
+        logger.info(
+            "Checked %d unique paths across %d archives, no cross-artifact overlaps",
+            len(path_artifacts),
+            len(archive_index),
+        )
+
+    def test_no_within_artifact_component_overlaps(
+        self, archive_index: list[ArchiveInfo]
+    ):
+        """Components of the same artifact should contain disjoint files.
+
+        The component scanner (artifact_builder.py) enforces disjointness via
+        the extends chain (lib -> run -> dbg -> dev -> doc). However, the
+        "test" component has no extends, so it can re-claim files already taken
+        by other components.
+
+        See https://github.com/ROCm/TheRock/issues/3796
+        """
+        # Group archives by artifact name, merging target variants per component.
+        # artifact_name -> component -> set of flattened paths
+        by_artifact: dict[str, dict[str, set[str]]] = defaultdict(
+            lambda: defaultdict(set)
+        )
+        for info in archive_index:
+            by_artifact[info.artifact_name][info.component].update(info.flattened_paths)
+
+        # A component overlap is a path claimed by more than one component
+        # within a single artifact name (e.g. "blas_run", "blas_test").
+        overlaps: list[ComponentOverlap] = []
+        for artifact_name, component_files in by_artifact.items():
+            component_names = sorted(component_files.keys())
+            # fpath -> list of component names that contain it
+            artifact_overlaps: dict[str, list[str]] = {}
+            for i in range(len(component_names)):
+                for j in range(i + 1, len(component_names)):
+                    c1, c2 = component_names[i], component_names[j]
+                    for fpath in component_files[c1] & component_files[c2]:
+                        artifact_overlaps.setdefault(fpath, []).extend([c1, c2])
+
+            if artifact_overlaps:
+                overlaps.append(
+                    ComponentOverlap(
+                        artifact_name=artifact_name, overlaps=artifact_overlaps
+                    )
+                )
+
+        if overlaps:
+            total, summary = _format_component_overlaps(overlaps)
+            pytest.fail(
+                f"Found within-artifact component overlaps in "
+                f"{len(overlaps)} artifact(s) ({total} total files). "
+                f"Components (_run, _test, etc.) should be disjoint "
+                f"(see https://github.com/ROCm/TheRock/issues/3796):\n{summary}"
+            )
+
+        artifact_names = {a.artifact_name for a in archive_index}
+        logger.info(
+            "Checked %d artifacts, no within-artifact component overlaps",
+            len(artifact_names),
+        )