Image load does not work in an HA setup

[dtantsur]

What Happened?

I'm migrating from a single-node test to a test in the HA scenario, and I'm seeing an odd behavior with loading an image. It looks like the image loaded with image load is not present on all nodes, even though the source code does include a loop over Nodes.

Here is the detailed output of the tool:

+ minikube image load --logtostderr localhost/controller:test
I0404 12:00:25.395509   24681 out.go:345] Setting OutFile to fd 1 ...
I0404 12:00:25.395553   24681 out.go:392] TERM=,COLORTERM=, which probably does not support color
I0404 12:00:25.395559   24681 out.go:358] Setting ErrFile to fd 2...
I0404 12:00:25.395570   24681 out.go:392] TERM=,COLORTERM=, which probably does not support color
I0404 12:00:25.395926   24681 root.go:338] Updating PATH: /home/runner/.minikube/bin
W0404 12:00:25.396120   24681 root.go:314] Error reading config file at /home/runner/.minikube/config/config.json: open /home/runner/.minikube/config/config.json: no such file or directory
I0404 12:00:25.397140   24681 config.go:182] Loaded profile config "minikube": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.32.0
I0404 12:00:25.397303   24681 cache.go:107] acquiring lock: {Name:mk59267ffc2bdae07a62c64253fb7bd5ca55a846 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0404 12:00:25.397582   24681 image.go:135] retrieving image: localhost/controller:test
I0404 12:00:25.398653   24681 image.go:178] daemon lookup for localhost/controller:test: Error response from daemon: No such image: localhost/controller:test
W0404 12:00:26.114699   24681 image.go:188] authn lookup for localhost/controller:test (trying anon): GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]]
I0404 12:00:26.530978   24681 image.go:192] remote lookup for localhost/controller:test: GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]]
I0404 12:00:26.531038   24681 cache.go:96] cache image "localhost/controller:test" -> "/home/runner/.minikube/cache/images/amd64/localhost/controller_test" took 1.133772783s
W0404 12:00:26.531322   24681 out.go:270] ! The image 'localhost/controller:test' was not found; unable to add it to cache.
! The image 'localhost/controller:test' was not found; unable to add it to cache.
I0404 12:00:26.531366   24681 cache.go:87] Successfully saved all images to host disk.
I0404 12:00:26.531616   24681 config.go:182] Loaded profile config "minikube": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.32.0
I0404 12:00:26.532180   24681 cli_runner.go:164] Run: docker container inspect minikube --format={{.State.Status}}
I0404 12:00:26.545692   24681 ssh_runner.go:195] Run: systemctl --version
I0404 12:00:26.545746   24681 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0404 12:00:26.559656   24681 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:32772 SSHKeyPath:/home/runner/.minikube/machines/minikube/id_rsa Username:docker}
I0404 12:00:26.657783   24681 cache_images.go:88] LoadCachedImages start: [localhost/controller:test]
I0404 12:00:26.658892   24681 image.go:135] retrieving image: localhost/controller:test
I0404 12:00:26.659632   24681 image.go:178] daemon lookup for localhost/controller:test: Error response from daemon: No such image: localhost/controller:test
W0404 12:00:27.137772   24681 image.go:188] authn lookup for localhost/controller:test (trying anon): GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]]
I0404 12:00:27.556426   24681 image.go:192] remote lookup for localhost/controller:test: GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]]
I0404 12:00:27.556469   24681 image.go:94] error retrieve Image localhost/controller:test ref GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]] 
I0404 12:00:27.556491   24681 cache_images.go:116] "localhost/controller:test" needs transfer: got empty img digest "" for localhost/controller:test
I0404 12:00:27.556529   24681 docker.go:337] Removing image: localhost/controller:test
I0404 12:00:27.556600   24681 ssh_runner.go:195] Run: docker rmi localhost/controller:test
I0404 12:00:27.579404   24681 cache_images.go:289] Loading image from: /home/runner/.minikube/cache/images/amd64/localhost/controller_test
I0404 12:00:27.579459   24681 cache_images.go:92] duration metric: took 921.653492ms to LoadCachedImages
W0404 12:00:27.579471   24681 cache_images.go:253] Failed to load cached images for "minikube": LoadCachedImages: stat /home/runner/.minikube/cache/images/amd64/localhost/controller_test: no such file or directory
I0404 12:00:27.579757   24681 cli_runner.go:164] Run: docker container inspect minikube-m02 --format={{.State.Status}}
I0404 12:00:27.591543   24681 cache_images.go:88] LoadCachedImages start: [localhost/controller:test]
I0404 12:00:27.592313   24681 image.go:135] retrieving image: localhost/controller:test
I0404 12:00:27.592830   24681 image.go:178] daemon lookup for localhost/controller:test: Error response from daemon: No such image: localhost/controller:test
W0404 12:00:28.062489   24681 image.go:188] authn lookup for localhost/controller:test (trying anon): GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]]
I0404 12:00:28.471699   24681 image.go:192] remote lookup for localhost/controller:test: GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]]
I0404 12:00:28.471735   24681 image.go:94] error retrieve Image localhost/controller:test ref GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]] 
I0404 12:00:28.471757   24681 cache_images.go:116] "localhost/controller:test" needs transfer: got empty img digest "" for localhost/controller:test
I0404 12:00:28.471790   24681 docker.go:337] Removing image: localhost/controller:test
I0404 12:00:28.471861   24681 ssh_runner.go:195] Run: docker rmi localhost/controller:test
I0404 12:00:28.471926   24681 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube-m02
I0404 12:00:28.485919   24681 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:32777 SSHKeyPath:/home/runner/.minikube/machines/minikube-m02/id_rsa Username:docker}
I0404 12:00:28.580216   24681 cache_images.go:289] Loading image from: /home/runner/.minikube/cache/images/amd64/localhost/controller_test
I0404 12:00:28.580293   24681 cache_images.go:92] duration metric: took 988.734762ms to LoadCachedImages
W0404 12:00:28.580306   24681 cache_images.go:253] Failed to load cached images for "minikube": LoadCachedImages: stat /home/runner/.minikube/cache/images/amd64/localhost/controller_test: no such file or directory
I0404 12:00:28.580600   24681 cli_runner.go:164] Run: docker container inspect minikube-m03 --format={{.State.Status}}
I0404 12:00:28.594113   24681 cache_images.go:88] LoadCachedImages start: [localhost/controller:test]
I0404 12:00:28.594803   24681 image.go:135] retrieving image: localhost/controller:test
I0404 12:00:28.595360   24681 image.go:178] daemon lookup for localhost/controller:test: Error response from daemon: No such image: localhost/controller:test
W0404 12:00:29.054139   24681 image.go:188] authn lookup for localhost/controller:test (trying anon): GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]]
I0404 12:00:29.463147   24681 image.go:192] remote lookup for localhost/controller:test: GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]]
I0404 12:00:29.463181   24681 image.go:94] error retrieve Image localhost/controller:test ref GET https://index.docker.io/v2/localhost/controller/manifests/test: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:localhost/controller Type:repository]] 
I0404 12:00:29.463201   24681 cache_images.go:116] "localhost/controller:test" needs transfer: got empty img digest "" for localhost/controller:test
I0404 12:00:29.463233   24681 docker.go:337] Removing image: localhost/controller:test
I0404 12:00:29.463336   24681 ssh_runner.go:195] Run: docker rmi localhost/controller:test
I0404 12:00:29.463401   24681 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube-m03
I0404 12:00:29.477735   24681 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:32782 SSHKeyPath:/home/runner/.minikube/machines/minikube-m03/id_rsa Username:docker}
I0404 12:00:29.572374   24681 cache_images.go:289] Loading image from: /home/runner/.minikube/cache/images/amd64/localhost/controller_test
I0404 12:00:29.572438   24681 cache_images.go:92] duration metric: took 978.308601ms to LoadCachedImages
W0404 12:00:29.572455   24681 cache_images.go:253] Failed to load cached images for "minikube": LoadCachedImages: stat /home/runner/.minikube/cache/images/amd64/localhost/controller_test: no such file or directory
I0404 12:00:29.572466   24681 cache_images.go:265] failed pushing to: minikube minikube-m02 minikube-m03

So, it just silently failed and returned success after not doing anything?

Attach the log file

minikube-full.log

Operating System

Ubuntu

Driver

None

[dtantsur]

In case it matters: I'm switching from image build to image load because image build --all makes me run out of space on the github actions worker. Or so it seems, I'm not 100% sure.

[dtantsur]

Oh, an idea: could it be an incompatibility with podman?

In any case, silently succeeding is not the right thing to do when the upload failed on all nodes.