[QUESTION/HELP] How to use multiple caching proxies at once?

[cowwoc]

Question / Where do you need Help?

Given this warning when creating a new cluster: WARN[0000] More than one registry specified, but the LocalRegistryHostingV1 spec only supports one -> Selecting the first one: k3d-docker-io.localhost how are we supposed to use multiple cache-on-pull docker registries against a single cluster?

My understanding is that I need to specify one local registry per remote registry I am caching (e.g. one for docker hub, another for github repository, etc). Next, I provide k3d with repositories.yaml that indicates which proxy is caching which remote registry.

But what's the point of doing all of this if k3d can only make use of one caching proxy at a time. Am I missing something?

[cowwoc]

To clarify, if I spin up multiple registries using k3d registry create and pass them into a cluster using k3d cluster create "--disable-default-registry-endpoint@all:*" --registry-use X --registry-use Y --registry-use Z then I see pulls working for X but failing for Y and `Z.

The error I get is:

Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1": failed to pull and unpack image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1": failed to resolve reference "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1": failed to do request: Head "http://k3d-registry-k8s-io.localhost:5003/v2/ingress-nginx/kube-webhook-certgen/manifests/v1.5.1?ns=registry.k8s.io": dial tcp 172.19.0.5:5003: connect: connection refused

I am only aware of two ways to have k3d pull from multiple cached repositories:

1. https://github.com/ligfx/k3d-registry-dockerd
2. Install the open-source version of https://www.sonatype.com/products/sonatype-nexus-oss-download

• Create a "docker (proxy)" for each repository
• Create a "docker (group)" that delegates to the aforementioned repositories
• Point k3d at the "docker (group)" repository. So from k3d's perspective, it only sees a single local repository.

[cowwoc]

This works for me:

1. Invoke k3d registry create once per registry you want to cache. Map each registry to a different host port and path.
2. Configure the new cluster to use the aforementioned registries.

For example:

#!/bin/sh

SCRIPT_PATH=$(dirname "$(realpath "$0")")
export HOST_PATH="${SCRIPT_PATH}"
mkdir -p "${SCRIPT_PATH}\cache"

# Run caching registries that listen on host ports
k3d registry create docker.io --port 5001 --proxy-remote-url https://registry-1.docker.io --volume "${SCRIPT_PATH}/cache/docker.io:/var/lib/registry" --no-help
k3d registry create registry.k8s.io --port 5002 --proxy-remote-url https://registry.k8s.io --volume "${SCRIPT_PATH}/cache/registry.k8s.io:/var/lib/registry" --no-help
k3d cluster create foobar --config k3d-config.yaml

where k3d-config.yaml contains:

apiVersion: k3d.io/v1alpha5
kind: Simple
options:
  k3s:
    extraArgs:
      - arg: "--disable-default-registry-endpoint"
        nodeFilters:
          - "all:*"
registries:
  config: |
    mirrors:
      "docker.io":
        endpoint:
          - http://host.docker.internal:5001
      "registry.k8s.io":
        endpoint:
          - http://host.docker.internal:5002

Please provide such an example in the official documentation. It scales a lot better than --registry-use which is limited to a maximum of one registry.

[cowwoc]

Note, due to microsoft/WSL#12891 (comment) you'll want to avoid ports in the 5xxx range on Windows.