[BUG] Version 5.8.2 breaks k3d-registry-dockerd

[cowwoc]

Version 5.8.1 works fine. Version 5.8.2 fails as follows...

What did you do

• How was the cluster created?

  • k3d cluster create foobar --config k3d-config.yaml

• What did you do afterwards?

  • docker ps
  • curl "http://localhost:53947/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io" (where port 53947 corresponds to the registry's port on the host) returns:
    curl: (52) Empty reply from server
  • Connect to the registry using docker exec -it 662043c3ef578e6eb3848cf9b55c8dc9a7e0c310e5ce35d3fca72dff7f4ac3bf /bin/sh
  • curl "http://localhost:5000/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io" returns index.html

What did you expect to happen

I expect to get the same response whether I am hitting the registry from inside or outside the container.

Which OS & Architecture

• output of k3d runtime-info:

arch: x86_64
cgroupdriver: cgroupfs
cgroupversion: "1"
endpoint: /var/run/docker.sock
filesystem: UNKNOWN
infoname: docker-desktop
name: docker
os: Docker Desktop
ostype: linux
version: 27.5.1

Which version of k3d

• output of k3d version:

k3d version v5.8.2
k3s version v1.31.5-k3s1 (default)

Which version of docker

• output of docker version

Client:
 Version:           27.5.1
 API version:       1.47
 Go version:        go1.22.11
 Git commit:        9f9e405
 Built:             Wed Jan 22 13:41:44 2025
 OS/Arch:           windows/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.38.0 (181591)
 Engine:
  Version:          27.5.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.11
  Git commit:       4c9b3b0
  Built:            Wed Jan 22 13:41:17 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e946
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• output of docker info:

Client:
 Version:    27.5.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.7.3
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-ai.exe
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.20.1-desktop.2
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.4-desktop.1
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.38
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.4
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.16.1
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-scout.exe

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 14
 Server Version: 27.5.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: nvidia runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.1.12-0-g51d5e946
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.31GiB
 Name: docker-desktop
 ID: ab037406-13e1-4f2f-b317-349ebb46e439
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

[cowwoc]

I think I figured out the problem.

Version 5.8.1 shows port 63100->5000/tcp being exported, meaning the host port 63100 is mapped to the container port 5000.

Version 5.8.2 shows 5000/tcp, 62954->62954/tcp being exported, meaning the registry is listening on port 5000 but it is not being exported. Then the host's port 62954 is mapped to the container port 62954, but no one is listening on that port.

[yakom]

@cowwoc hi, what's in k3d-config.yaml?

[yakom]

@cowwoc it's the one mentioned here i take?

[pianofab]

Yes, upvoting the brokenness.

[iwilltry42]

@cowwoc would you want to write a test case for our E2E tests that would've caught this?
That would be dope.

@ligfx As said earlier, I'd also be up for integrating this registry approach more tightly with k3d.

[yakom]

the change i've introduced is working when the default registry image is used. i didn't realize the implication of the fact that the image can be freely substituted. therefore, i think i'll narrow the scope of the change to the default registry image case.

[ligfx]

Looks like there’s a new environment variable, REGISTRY_HTTP_ADDR, that custom registries can use to figure out which port to listen on? Is that correct?

[yakom]

@ligfx yes, but REGISTRY_HTTP_ADDR is used to configure the default registry image speciffically, as per their doc. other images might not have their port configurable or require a different option/variable. k3d can't know. so it may be considered more of a design question for @iwilltry42 to decide what to do, but given that any image can be used, i assumed it's better to narrow the scope of the change.

[cowwoc]

@iwilltry42 Sorry, I'm not able to write an E2E test for this. I don't code in Golang at all yet...

[iwilltry42]

No problem @cowwoc !

So, I have no idea how popular it is to use custom registry images, but I think we can be a little opinionated here and have 1st class support for

• the official (default) registry image from docker -> already supports the REGISTRY_HTTP_ADDR env var
• @ligfx custom registry image -> we can surely add support there fairly easily

2nd class support would just be the ability to use the --port syntax to allow setting the container port to whatever is needed for that image to work. In that case, I also think it's fine to always set REGISTRY_HTTP_ADDR to the chosen port regardless of the image as I don't think it would cause any harm - either it's respected by the image or not.

[iwilltry42]

FYI: I reverted the breaking change and pushed v5.8.3 - we'll re-introduce this change (maybe slightly adjusted) including more docs and more verbose release notes again at a later time.