Certificate expiration problem

[pptfz]

An error was reported in etcd pod for certificate expiration, how should I update the certificate? I did not find relevant instructions in the official document, because my native language is not English, so I may have some difficulties in searching, could you help me find out how to deal with the certificate expiration, thank you

$ kubectl logs -f etcd-0 
{"level":"warn","ts":"2024-03-29T11:04:19.083Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:6191","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:22.547Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:56424","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:23.864Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:7304","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:24.859Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:48735","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:24.871Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:28562","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:25.866Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:12775","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:26.392Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:37674","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:27.181Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:22013","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:28.366Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:50554","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority"}
WARNING: 2024/03/29 11:04:28 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2024-03-29T11:04:28.779Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:7672","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:29.652Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:34941","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:32.298Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:28518","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:34.528Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:50468","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:38.800Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:9520","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:40.515Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:13362","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:04:51.620Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:52088","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority"}
WARNING: 2024/03/29 11:04:51 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2024-03-29T11:05:26.579Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:53560","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority"}
WARNING: 2024/03/29 11:05:26 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2024-03-29T11:06:44.427Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:57668","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority"}
WARNING: 2024/03/29 11:06:44 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2024-03-29T11:08:42.877Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34742","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority"}
WARNING: 2024/03/29 11:08:42 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2024-03-29T11:09:52.888Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:33800","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:09:53.884Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:7872","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:09:53.895Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:61508","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:09:54.891Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:16420","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:09:55.656Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:48604","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:09:56.761Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:33691","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:09:58.386Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:42731","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:09:59.760Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:12541","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:10:02.904Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:15257","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:10:04.151Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:41527","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:10:10.238Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:24551","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:10:11.590Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.80.4.234:16347","server-name":"etcd-client.karmada-system.svc.cluster.local","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-03-29T11:10:38.564Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:39222","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority"}
WARNING: 2024/03/29 11:10:38 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
{"level":"warn","ts":"2024-03-29T11:12:36.870Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:46760","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority"}
WARNING: 2024/03/29 11:12:36 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate". Reconnecting...
^C

I searched for certificate expiration and found no results

[jmhbnz]

Hi @pptfz - Thanks for your question. Is this etcd instance a built in etcd instance for your kubernetes cluster? Or is it a standalone instance deployed on top of kubernetes for some other workload?

How have you configured your tls settings for etcd currently?

[pptfz]

We are using karmada Open source project

The following are errors for the installed pod and the current pod,I have put forward my problem to karmada project, but I have not received any reply yet. I checked the log of these defective Pods and found that the error was indicating that the certificate had expired, because it contained information about etcd connection error, so I would like to ask whether it was related to the expiration of etcd certificate

$ kubectl get pod
NAME                                               READY   STATUS             RESTARTS            AGE
etcd-0                                             1/1     Running            22688 (38s ago)     569d
karmada-apiserver-6dd844fdfd-stzzg                 0/1     CrashLoopBackOff   62176 (4m54s ago)   569d
karmada-controller-manager-7dbf7c6578-kmztg        0/1     CrashLoopBackOff   57450 (4m9s ago)    326d
karmada-kube-controller-manager-656cdc675f-cj6vw   0/1     CrashLoopBackOff   57286 (3m42s ago)   569d
karmada-scheduler-764fbdcd6d-6jhrd                 1/1     Running            1                   569d
karmada-webhook-6489787db4-wc9pp                   1/1     Running            0                   569d

$ kubectl logs -f karmada-apiserver-6dd844fdfd-stzzg
W0328 02:20:08.682680       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://etcd-client.karmada-system.svc.cluster.local:2379/  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2024-03-28T02:20:08Z is after 2023-09-05T06:51:02Z". Reconnecting...

$ kubectl logs -f karmada-controller-manager-7dbf7c6578-kmztg
E0328 02:26:10.815013       1 controllermanager.go:78] failed to build controller manager: Get "https://karmada-apiserver.karmada-system.svc.cluster.local:5443/api?timeout=32s": dial tcp 10.254.52.163:5443: connect: connection refused
Get "https://karmada-apiserver.karmada-system.svc.cluster.local:5443/api?timeout=32s": dial tcp 10.254.52.163:5443: connect: connection refused

$ karmada-kube-controller-manager-656cdc675f-cj6vw
I0328 02:26:42.065358       1 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/etc/karmada/pki/server-ca.crt"
unable to load configmap based request-header-client-ca-file: Get "https://karmada-apiserver.karmada-system.svc.cluster.local:5443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 10.254.52.163:5443: connect: connection refused