Unable to access cluster due to certificate expiration with odd time

[willgorman]

I have a 4 node raspberry pi Talos cluster that I'm no longer able to access the talos api due to what appears to be an expired certificate. However the expiration date looks to be much too old.

 talosctl reboot
◰ watching nodes: [192.168.6.104 192.168.6.105 192.168.6.106 192.168.6.107]
    * 192.168.6.104: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-10-31T18:42:09-05:00 is after 1970-01-02T00:00:14Z"
    * 192.168.6.105: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-10-31T18:42:09-05:00 is after 1970-01-02T00:00:13Z"
    * 192.168.6.106: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-10-31T18:42:09-05:00 is after 1970-01-02T00:00:17Z"
    * 192.168.6.107: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-10-31T18:42:09-05:00 is after 1970-01-02T00:00:13Z"

I think this looks different than the issue in #9457 because the config info shows the certificate is still valid for a month

 talosctl config info
Current context:     kuberry
Nodes:               192.168.6.104, 192.168.6.105, 192.168.6.106, 192.168.6.107
Endpoints:           192.168.6.105, 192.168.6.104, 192.168.6.106
Roles:               os:admin
Certificate expires: 1 month from now (2025-12-01)

The kubernetes cluster isn't accessible at all currently

 kubectl get nodes
E1031 18:45:49.527331    5861 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.6.104:6443/api?timeout=32s\": dial tcp 192.168.6.104:6443: connect: connection refused"
E1031 18:45:49.529042    5861 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.6.104:6443/api?timeout=32s\": dial tcp 192.168.6.104:6443: connect: connection refused"
E1031 18:45:49.530646    5861 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.6.104:6443/api?timeout=32s\": dial tcp 192.168.6.104:6443: connect: connection refused"
E1031 18:45:49.532316    5861 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.6.104:6443/api?timeout=32s\": dial tcp 192.168.6.104:6443: connect: connection refused"
E1031 18:45:49.533895    5861 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.6.104:6443/api?timeout=32s\": dial tcp 192.168.6.104:6443: connect: connection refused"
The connection to the server 192.168.6.104:6443 was refused - did you specify the right host or port?

Is there anything I can try to recover access to the talos api for this cluster?

[willgorman]

I found https://www.reddit.com/r/TalosLinux/comments/1mu0uxp/talosctl_commands_fail_with_tls_verification_on/ that looks like a similar issue. I lost access after all the nodes rebooted suddenly due to a power loss, but I've rebooted nodes in the past with no issues.

[willgorman]

Figured it out. I did what I should have done in the first place and connected a monitor to one of the nodes. The console had a bunch of errors like "time query error with server 192.168.4.1" and "kiss of death received". Apparently the nodes are using my eero router for NTP, which is a little surprising to me since I think the default in the machine configs is to use cloudflare and I haven't changed that. I also don't know why the router NTP server was sending an error but I restarted the eero network and then the time query errors stopped and the talosctl certificate errors went away. Unfortunately, etcd is failing to get healthy now but at least I have a chance to figure that out now that I can interact with the api again.