Expose a metric on the data plane to tell how long the mTLS identity cert is going to expire

[jijiechen]

Description

We've received a few cases on mTLS identity cert renewing this year and there is not an OOTB way to track the expiration of current effective identity cert in the data plane. To do so, people either need to aggregate & combine metrics at control plane side, or they need to programmatically retrieve the identify cert info from the data plane XDS admin API.

If an identity cert is not renewed in time as expected, the traffic from/to the data plane will fail immediately. So people want to discover this risk earlier. Sometimes, they can solve the issue manually by restarting the workload or control plane as a workaround.

To make this process easier, it's helpful to expose a metric to tell how long the mTLS identity cert is going to expire so that one can make an alert based on it.

[lahabana]

I feel like there's one like this already but I may be mixing things up

[slonka]

Triage: I think there is days_until_first_cert_expiring and there is also expiration_unix_time_seconds - can we just use that? WDYT @lahabana ?

This listener.self_inbound_dp_secondary.ssl.certificate.kri_mid_meshidentity-migration___identity-migration_.expiration_unix_time_seconds: 1762800586 - seems better because days_until_first_cert_expiring - will not give useful info if rotation is 1 day.

Check if this metric expiration_unix_time_seconds is in 2.7 release

[lobkovilya]

Metric expiration_unix_time_seconds was introduced relatively recently and due to the bug it appears in prometheus stats only starting 1.35.2. So basically we can use it only in 2.12