certSANs and Internal, Pub IP after talosctl apply

[nobleess]

I create VM with Talos in Cloud
My machine got two ip example (int: 1.1.1.1, pub: 2.2.2.2)

I forgot add public IP in slice machine.certSANs
How i can path this, after talosctl apply -i

How i have
rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for 1.1.1.1 not 2.2.2.2"
Help)))

[smira]

If you can't connect over 1.1.1.1, there's no way to fix it, just re-provision the VM.

[nobleess]

hello
Can I configure a new node through the node entity?

[smira]

Please create a separate discussion with all the details for your question. I don't quite understand what you're asking about.

[agusdallalba]

I'm trying out Talos for the first time and I just hit this issue.

I'd say it's a bug in talosctl gen config, but it may also be a documentation issue.

I was following https://www.talos.dev/v1.10/introduction/prodnotes/, provisioning the first node using DNS for endpoint HA.

talosctl gen config "$CLUSTER_NAME" \
    https://kube.example.com:6443 \
    --with-secrets secrets.yaml \
    --config-patch @config-patch.yaml \
    --output-dir _out

This generates a controlplane.yaml that contains:

machine:
    certSANs: []
cluster:
    controlPlane:
        endpoint: https://kube.example.com:6443
    apiServer:
        certSANs:
            - kube.example.com

Notice the empty machine.certSANs.

I applied this controlplane.yaml to the first node and everything appeared to go well.

talosctl --insecure --nodes $IP apply-config --file _out/controlplane.yaml
talosctl config merge _out/talosconfig
talosctl config endpoint kube.example.com

In that last command I deviated from the documentation, because it made sense to me that the kube API endpoint would be in the same host as the talos API endpoint.

However any command after this fails with the confusing error message error executing bootstrap: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for talos-j9b-tb5, not kube.example.com", where talos-j9b-tb5 is generated randomly.

Here I thought I had messed up and had to reprovision. Despite appearances this is a client error, and is possible to fix it by setting talosctl config endpoint $IP.

In the case of @nobleess, where $IP was unreachable, they probably could have salvaged the node by editing /etc/hosts and adding a line like talos-j9b-tb5 $IP.

I see two simultaneous solutions to this.

• It makes sense to me that the kube API endpoint would be in the same host as the talos API endpoint. Therefore talosctl gen config should set machine.certSANs[] to the endpoint hostname in control plane nodes, the same way it already does for cluster.apiserver.certSANs[].

  In fact the --additional-sans flag of talosctl gen config (which is a cluster-scoped command, not a machine-scoped command) also includes the host in both lists:

  talosctl gen config "$CLUSTER_NAME" \
      https://ENDPOINT:6443 \
      --output-dir _out \
      --additional-sans ADDITIONAL
  

  Results in _out/controlplane.yaml containing:

  machine:
      certSANs:
          - ADDITIONAL
  cluster:
      controlPlane:
          endpoint: https://ENDPOINT:6443
      apiServer:
          certSANs:
              - ENDPOINT
              - ADDITIONAL
  

• As a (very useful) hack, there should be a talosctl global flag that allows you to ignore the CN/SAN of the endpoint certificate. This could be useful to recover a cluster that has been renumbered.

Kind regards.

[Iheanacho-ai]

Hi @agusdallalba

I totally understand where you are coming from, the thing is,

Talos cares about two different APIs

• the Kubernetes API, the one you set for its HA endpoint as ( https://kube.example.com:6443)
• The Talos API, (used by talosctl to talk to the nodes)

When you ran talosctl gen config, it added kube.example.com to the Kubernetes API certs (so Kubernetes works).

But it did not add kube.example.com to Talos API certs (machine.certSANs)

So when you told talosctl to connect to the Talos API at kube.example.com.

The Talos API cert did not recognize that name, and thus the TLS verification failed.

Thats, why step 12 of the production clusters documentation asks that you configure your endpoint: https://www.talos.dev/v1.10/introduction/prodnotes/#step-12-set-endpoints-of-your-control-plane-nodes

@smira and @rothgar do you have a different answer?

[smira]

Talos and Kubernetes APIs are different in the way HA is achieved:

• Kubernetes API is always a single endpoint, which should be backed by a proper TCP load-balancer to achieve true HA for external cluster access (in-cluster access is made HA with Talos's KubePrism)
• Talos API is HA via client-side loadbalancing and automatic failover; so our recommended path is to put IP addresses of controlplane node as endpoints.

Talos automatically manages cert SANs, but it can't know about external DNS names, so these should be added to the machine config manually.

There is no "lock out" scenario here, as IPs are always pushed to the cert SANs automatically.