Registry Images collector does not pull images the same way K8S does

[Mahoney]

Bug Description

We have a Registry Images check failing, despite the cluster we run the collector against actually being able to pull the image.

Expected Behavior

If the K8S cluster can pull the image, the registry image collector can pull the image and the check passes.

Steps To Reproduce

1. Build a docker image foo/my-multi-arch-image:latest for the following architectures and push it to Dockerhub
   • linux/arm64
   • linux/amd64
2. Start a KinD cluster on a computer running with an Apple M1 Max chip
3. Start a pod in the cluster that uses that image - observe that it is pulled successfully
4. Run the following collector against the cluster:

apiVersion: troubleshoot.sh/v1beta2
kind: Preflight
metadata:
  name: images-can-be-pulled
spec:
  collectors:
    - registryImages:
        images:
          - "foo/my-multi-arch-image:latest"

5. Observe that the check fails as so:

{
  "images": {
    "foo/my-multi-arch-image:latest": {
      "exists": false,
      "error": "failed to get image manifest: choosing image instance: no image found in image index for architecture \"arm64\", variant \"v8\", OS \"darwin\""
    }
  }
}

Additional Context

Include the following information.

• Troubleshoot version: 0.114.0
• Operating system: macOs
• Operating system version: 15.2
• KinD: 0.26.0
• Kubernetes: 1.32.0

[Mahoney]

Just ran it against a new EKS cluster, using nodes from AMI amazon-eks-node-1.30-v20241213 with Architecture x86_64.

We get the same error message:

{
  "images": {
    "foo/my-multi-arch-image:latest": {
      "exists": false,
      "error": "failed to get image manifest: choosing image instance: no image found in image index for architecture \"arm64\", variant \"v8\", OS \"darwin\""
    }
  }
}

That seems very suspicious - like it's taking the architecture from the box where kubectl preflight is running, rather than the node where the image should be being pulled.

[xavpaice]

Are you running these checks from a separate machine, rather than in a pod in the cluster? These preflights are designed to be run by a process (e.g. kotsadm) running in a pod in the cluster, and some can give odd results if you run preflight from the host machine.

[Mahoney]

Outside the cluster, I am running curl https://krew.sh/preflight | bash to install the preflight plugin to kubectl, and then (still outside the cluster) kubectl preflight preflight.yml (where preflight.yml contains the collector above, amongst other collectors and checks).

Is this incorrect? I'm not always the best at reading documentation, but I've found it quite challenging to understand both whatI should run, and where, between Preflight and SupportBundle, and what they take responsibility for running, and where they run it.