Talos upgrade in Air-gapped Environments

[marcopornic]

Hello everyone !

I’ve gone through all the discussion topics, but I can’t find any information on updating Talos in an AirGap environment.
I successfully generated an ISO with the image cache feature, and everything is working fine. My cluster is up and running.

Now, how should I proceed to update Talos or Kubernetes? In the documentation, it suggests using the --image ghcr.io/siderolabs/installer:vX.X.X image, but for that, I need an internet connection.
Is it possible to update Talos using a new ISO?

Thanks for your feedback!

[smira]

I think it's the same question as how would you upgrade your applications in air-gapped environment? How would you upgrade Kubernetes?

Talos upgrades are delivered via a container image, same way as your application image.

If you have more than a single machine running in the air-gapped environment, the proper way is to set up a local registry mirror, and redirect all image pulls from Talos to use that mirror.

If you have a single machine, that might sound like an overkill, and in that case Image Cache sounds like a better way. At the moment Talos doesn't support updating Image Cache after the initial install.

[smira]

In other words, can I upgrade using a new ISO without reinstalling everything from scratch?

At the moment, no, this might become available in Talos 1.11.

At the moment the Talos upgrade image installer is a container image, so you make it available to your host same way as any other new container image you need during the upgrade process.

Would Zarf be a suitable tool to help automate or facilitate this kind of upgrade in a Talos air-gapped setup? (I am currently trying to deploy it in the cluster)

I don't know, I would like to hear from you how it goes!

[marcopornic]

Hi, and thanks for your feedback!

The integration of Zarf into Talos is progressing well on my side, but I'm currently blocked at a specific step.
Zarf automatically deploys a container registry inside a pod, and this pod then tries to access the registry using 127.0.0.1.

Here are the logs from the pod:

:~/talos# kubectl describe pod zarf-docker-registry-77746f4b77-6w6j8 -n zarf

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  23s               default-scheduler  Successfully assigned zarf/zarf-docker-registry-77746f4b77-6w6j8 to talos-cjf-b3b
  Normal   BackOff    22s               kubelet            Back-off pulling image "127.0.0.1:31865/library/registry:2.8.3"
  Warning  Failed     22s               kubelet            Error: ImagePullBackOff
  Normal   Pulling    8s (x2 over 23s)  kubelet            Pulling image "127.0.0.1:31865/library/registry:2.8.3"
  Warning  Failed     8s (x2 over 23s)  kubelet            Failed to pull image "127.0.0.1:31865/library/registry:2.8.3": rpc error: code = NotFound desc = failed to pull and unpack image "127.0.0.1:31865/library/registry:2.8.3": failed to resolve reference "127.0.0.1:31865/library/registry:2.8.3": 127.0.0.1:31865/library/registry:2.8.3: not found
  Warning  Failed     8s (x2 over 23s)  kubelet            Error: ErrImagePull

Interestingly, by using port forwarding to the pod, I'm able to successfully pull the image — so it is definitely available in the registry:

root@controlplane:~/talos/zarf# kubectl -n zarf port-forward svc/zarf-injector 5000:5000
Forwarding from 127.0.0.1:5000 -> 5000
Forwarding from [::1]:5000 -> 5000
Handling connection for 5000
Handling connection for 5000
Handling connection for 5000
Handling connection for 5000
Handling connection for 5000
Handling connection for 5000
Handling connection for 5000
Handling connection for 5000
Handling connection for 5000
Handling connection for 5000

root@controlplane:~/talos/zarf# docker pull localhost:5000/library/registry:2.8.3
2.8.3: Pulling from library/registry
44cf07d57ee4: Pull complete
bbbdd6c6894b: Pull complete
8e82f80af0de: Pull complete
3493bf46cdec: Pull complete
6d464ea18732: Pull complete
Digest: sha256:46faa9a1ae6813194b53921a370f2f4f8c5e1aae228a89bceafef5847a6a3278
Status: Downloaded newer image for localhost:5000/library/registry:2.8.3
localhost:5000/library/registry:2.8.3

Is the use of 127.0.0.1 blocked or restricted in Talos?

Thanks for your feedback!

[smira]

127.0.0.1 is a loopback address, Talos doesn't restrict it.

I'm not sure how it is supposed to work though, as kubelet/CRI run with host network, so for them 127.0.0.1 would be a host address.

[marcopornic]

Zarf expose a nodeport service for the registry. I've gone a bit further by adding this configuration in Talos:

machine:
  registries:
    mirrors:
      "127.0.0.1:31540":
        endpoints:
          - http://10.10.205.17:31540/

I'm using the NodePort of the zarf-docker-registry. The problem is that I don't know in advance which NodePort will be used, and when the pod is recreated, it will change :(

[hmonsalv]

@marcopornic , probably you've already found a solution for this, but in case you didn't: if you're running Cilium in you're k8s cluster, then you can redirect k8s traffic (for example from your Zarf pod) to your localhost to reach that registry. We talked about this 2 weeks ago in a TalosCon25 session "Cutting the Cord: An Air-Gapped Journey with Talos and Cilium", will be published soon, keep an eye on that.