etcd_server_healthcheck not available until after readyz endpoint has been hit once

[AndrewJackson2020]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

It appears that the etcd_server_healthcheck fields in the /metrics endpoint are not available until after the readyz endpoint has been hit.

What did you expect to happen?

The behaviour that I expect is shown in the below code block. I would expect the fields in etcd_server_healthcheck to still show up but with zero count.

> cd `mktemp -d`

> etcd  --log-level fatal &

> curl http://localhost:2379/metrics | grep 'etcd_server_healthcheck'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  156k    0  156k    0     0  27.0M      0 --:--:-- --:--:-- --:--:-- 30.6M# HELP etcd_server_healthcheck The result of each kind of healthcheck.
# TYPE etcd_server_healthcheck gauge
etcd_server_healthcheck{name="data_corruption",type="readyz"} 0
etcd_server_healthcheck{name="linearizable_read",type="readyz"} 0
etcd_server_healthcheck{name="non_learner",type="readyz"} 0
etcd_server_healthcheck{name="serializable_read",type="readyz"} 0
# HELP etcd_server_healthchecks_total The total number of each kind of healthcheck.
# TYPE etcd_server_healthchecks_total counter
etcd_server_healthchecks_total{name="data_corruption",status="success",type="readyz"} 0
etcd_server_healthchecks_total{name="linearizable_read",status="success",type="readyz"} 0
etcd_server_healthchecks_total{name="non_learner",status="success",type="readyz"} 0
etcd_server_healthchecks_total{name="serializable_read",status="success",type="readyz"} 0

> curl http://localhost:2379/readyz
ok

> curl http://localhost:2379/metrics | grep 'etcd_server_healthcheck'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP etcd_server_healthcheck The result of each kind of healthcheck.
# TYPE etcd_server_healthcheck gauge
etcd_server_healthcheck{name="data_corruption",type="readyz"} 1
etcd_server_healthcheck{name="linearizable_read",type="readyz"} 1
etcd_server_healthcheck{name="non_learner",type="readyz"} 1
etcd_server_healthcheck{name="serializable_read",type="readyz"} 1
# HELP etcd_server_healthchecks_total The total number of each kind of healthcheck.
# TYPE etcd_server_healthchecks_total counter
etcd_server_healthchecks_total{name="data_corruption",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="linearizable_read",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="non_learner",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="serializable_read",status="success",type="readyz"} 1

> pkill etcd

> etcd  --log-level fatal &

> curl http://localhost:2379/metrics | grep 'etcd_server_healthcheck'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  156k    0  156k    0     0  27.0M      0 --:--:-- --:--:-- --:--:-- 30.6M# HELP etcd_server_healthcheck The result of each kind of healthcheck.
# TYPE etcd_server_healthcheck gauge
etcd_server_healthcheck{name="data_corruption",type="readyz"} 0
etcd_server_healthcheck{name="linearizable_read",type="readyz"} 0
etcd_server_healthcheck{name="non_learner",type="readyz"} 0
etcd_server_healthcheck{name="serializable_read",type="readyz"} 0
# HELP etcd_server_healthchecks_total The total number of each kind of healthcheck.
# TYPE etcd_server_healthchecks_total counter
etcd_server_healthchecks_total{name="data_corruption",status="success",type="readyz"} 0
etcd_server_healthchecks_total{name="linearizable_read",status="success",type="readyz"} 0
etcd_server_healthchecks_total{name="non_learner",status="success",type="readyz"} 0
etcd_server_healthchecks_total{name="serializable_read",status="success",type="readyz"} 0

How can we reproduce it (as minimally and precisely as possible)?

See below for script that I used to repro this issue.

> cd `mktemp -d`

> etcd  --log-level fatal &

> curl http://localhost:2379/metrics | grep 'etcd_server_healthcheck'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  156k    0  156k    0     0  27.0M      0 --:--:-- --:--:-- --:--:-- 30.6M

> curl http://localhost:2379/readyz
ok

> curl http://localhost:2379/metrics | grep 'etcd_server_healthcheck'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP etcd_server_healthcheck The result of each kind of healthcheck.
# TYPE etcd_server_healthcheck gauge
etcd_server_healthcheck{name="data_corruption",type="readyz"} 1
etcd_server_healthcheck{name="linearizable_read",type="readyz"} 1
etcd_server_healthcheck{name="non_learner",type="readyz"} 1
etcd_server_healthcheck{name="serializable_read",type="readyz"} 1
# HELP etcd_server_healthchecks_total The total number of each kind of healthcheck.
# TYPE etcd_server_healthchecks_total counter
etcd_server_healthchecks_total{name="data_corruption",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="linearizable_read",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="non_learner",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="serializable_read",status="success",type="readyz"} 1

> pkill etcd

> etcd  --log-level fatal &

> curl http://localhost:2379/metrics | grep 'etcd_server_healthcheck'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  156k    0  156k    0     0  27.0M      0 --:--:-- --:--:-- --:--:-- 30.6M

Anything else we need to know?

No response

Etcd version (please run commands below)

 > etcd --version
etcd Version: 3.6.5
Git SHA: a061450
Go Version: go1.24.7
Go OS/Arch: linux/amd64

 > etcdctl version
etcdctl version: 3.6.5
API version: 3.6

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

[gclough]

Shouldn't these always be 1 when the system is healthy, even if the endpoint hasn't yet been polled?

etcd_server_healthchecks_total{name="data_corruption",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="linearizable_read",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="non_learner",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="serializable_read",status="success",type="readyz"} 1

Making the initial state in /metrics:

# HELP etcd_server_healthcheck The result of each kind of healthcheck.
# TYPE etcd_server_healthcheck gauge
etcd_server_healthcheck{name="data_corruption",type="readyz"} 1
etcd_server_healthcheck{name="linearizable_read",type="readyz"} 1
etcd_server_healthcheck{name="non_learner",type="readyz"} 1
etcd_server_healthcheck{name="serializable_read",type="readyz"} 1
# HELP etcd_server_healthchecks_total The total number of each kind of healthcheck.
# TYPE etcd_server_healthchecks_total counter
etcd_server_healthchecks_total{name="data_corruption",status="success",type="readyz"} 0
etcd_server_healthchecks_total{name="linearizable_read",status="success",type="readyz"} 0
etcd_server_healthchecks_total{name="non_learner",status="success",type="readyz"} 0
etcd_server_healthchecks_total{name="serializable_read",status="success",type="readyz"} 0

I had presumed (yeah I know... "presumed") that was a binary yes/no, but I can't find any details in the doco:

https://etcd.io/docs/v3.6/metrics/
https://etcd.io/docs/v3.6/metrics/etcd-metrics-latest.txt

[serathius]

cc @siyuanfoundation

[gclough]

As a workaround we've added this to the service we use to start etcd, so it polls immediately after startup:

# wait for startup, then poll /readyz and /livez endpoints to ensure stats appear in /metrics endpoint
# https://github.com/etcd-io/etcd/issues/20741
ExecStartPost=/bin/bash -c "\
  while ! /usr/bin/nc -z localhost {{ etcd_listen_client_port }}; do \
    echo $(/usr/bin/date): Waiting on etcd cluster to start listening on {{ etcd_listen_client_port }}; \
    sleep 1; \
  done; \
  echo livez: $(/usr/bin/curl --silent {{ etcd_listen_client_protocol }}://localhost:{{ etcd_listen_client_port }}/livez); \
  echo readyz: $(/usr/bin/curl --silent {{ etcd_listen_client_protocol }}://localhost:{{ etcd_listen_client_port }}/readyz); \
"

[siyuanfoundation]

Thanks for raising the issue.
The proper way of using the etcd health checks is the client would check the readyz before starting to use etcd, and periodically probe the endpoint throughout the lifetime of etcd. So we always assume the endpoint would be probed from the beginning.
@gclough @AndrewJackson2020 Can you help me understand why you would depend on the metrics at the initial setup?

If we have to assume the client does not probe the endpoint regularly, then we would need to add periodic healthchecks within etcd to make sure the metrics are reflecting the current state, which would be redundant for normal use cases.

[AndrewJackson2020]

The usecase here is there is a prometheus scrape/alert that assumes that field exists and is problematic when it does not exist, not sure about all of the details though.

then we would need to add periodic healthchecks within etcd to make sure the metrics are reflecting the current state

I don't think it would need to be periodic healthchecks within etcd, would just need to be a onetime thing on startup, similar to what @gclough alluded to with his systemd service file ExecStartPost snippet. Have not poked around the code but kind of surprising its not possible to show 0 instead of non existent for those metrics so that no internal healthcheck is required at all. Have not had capacity to poke around the etcd codebase too much.

[siyuanfoundation]

etcd/server/etcdserver/api/etcdhttp/health.go

Line 298 in bc6a17c

func (reg *CheckRegistry) runHealthChecks(ctx context.Context, checkNames ...string) HealthStatus {

is where the health check is run and metrics get recorded.
Feel free to raise a PR to improve it.

[AndrewJackson2020]

Probably not the cleanest solution but the below woks for me.

diff --git a/server/etcdserver/api/etcdhttp/health.go b/server/etcdserver/api/etcdhttp/health.go
index 26ed4ca..1598956 100644
--- a/server/etcdserver/api/etcdhttp/health.go
+++ b/server/etcdserver/api/etcdhttp/health.go
@@ -145,6 +145,45 @@ func init() {
        prometheus.MustRegister(healthFailed)
        prometheus.MustRegister(healthCheckGauge)
        prometheus.MustRegister(healthCheckCounter)
+
+       healthCheckGauge.With(prometheus.Labels{
+               "type":   "readyz",
+               "name":   "data_corruption",
+       }).Add(0)
+       healthCheckGauge.With(prometheus.Labels{
+               "type":   "readyz",
+               "name":   "linearizable_read",
+       }).Add(0)
+       healthCheckGauge.With(prometheus.Labels{
+               "type":   "readyz",
+               "name":   "non_learner",
+       }).Add(0)
+       healthCheckGauge.With(prometheus.Labels{
+               "type":   "readyz",
+               "name":   "serializable_read",
+       }).Add(0)
+
+
+       healthCheckCounter.With(prometheus.Labels{
+               "type":   "readyz",
+               "name":   "serializable_read",
+               "status": "success",
+       }).Add(0)
+       healthCheckCounter.With(prometheus.Labels{
+               "type":   "readyz",
+               "name":   "non_learner",
+               "status": "success",
+       }).Add(0)
+       healthCheckCounter.With(prometheus.Labels{
+               "type":   "readyz",
+               "name":   "linearizable_read",
+               "status": "success",
+       }).Add(0)
+       healthCheckCounter.With(prometheus.Labels{
+               "type":   "readyz",
+               "name":   "data_corruption",
+               "status": "success",
+       }).Add(0)
 }

 // Health defines etcd server health status.

[siyuanfoundation]

If you just need some value instead of success checks, the easiest you can do is

--- a/server/etcdserver/api/etcdhttp/health.go
+++ b/server/etcdserver/api/etcdhttp/health.go
@@ -293,6 +293,8 @@ func (reg *CheckRegistry) InstallHTTPEndpoints(lg *zap.Logger, mux *http.ServeMu
                        return reg.runHealthChecks(r.Context(), check)
                }))
        }
+       // initial run of the health checks to populate the initial values of the metrics
+       reg.runHealthChecks(context.Background(), checkNames...)
 }

but linearizable_read will not be ready during this initialization phase.

[gclough]

Thanks for raising the issue. The proper way of using the etcd health checks is the client would check the readyz before starting to use etcd, and periodically probe the endpoint throughout the lifetime of etcd. So we always assume the endpoint would be probed from the beginning. @gclough @AndrewJackson2020 Can you help me understand why you would depend on the metrics at the initial setup?

I have Prometheus scraping the /metrics endpoint, checking the values of livez and readyz in the returned data:

etcd_server_healthcheck{name="data_corruption",type="readyz"} 1
etcd_server_healthcheck{name="linearizable_read",type="readyz"} 1
etcd_server_healthcheck{name="non_learner",type="readyz"} 1
etcd_server_healthcheck{name="serializable_read",type="livez"} 1
etcd_server_healthcheck{name="serializable_read",type="readyz"} 1
etcd_server_healthchecks_total{name="data_corruption",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="linearizable_read",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="non_learner",status="success",type="readyz"} 1
etcd_server_healthchecks_total{name="serializable_read",status="success",type="livez"} 1
etcd_server_healthchecks_total{name="serializable_read",status="success",type="readyz"} 1

I wasn't intending on polling the separate \livez and \readyz endpoints if I can get that info directly from \metrics.

If we have to assume the client does not probe the endpoint regularly, then we would need to add periodic healthchecks within etcd to make sure the metrics are reflecting the current state, which would be redundant for normal use cases.

If I read that correctly, then the above values are only ever updated in /metrics as and when the corresponding /livez & /readyz endpoints are polled? This means the data in /metrics represents the state during the most recent healthcheck, and is stale if /livez and /readyz are not regularly polled?

Would it make sense to call that out explicitly in the returned data from /metrics by adding an extra field?

etcd_server_healthcheck{name="data_corruption",type="readyz",last_checked="2025-10-03T10:19:00+01:00"} 1

[AndrewJackson2020]

Would it make sense to call that out explicitly in the returned data from /metrics by adding an extra field?

etcd_server_healthcheck{name="data_corruption",type="readyz",last_checked="2025-10-03T10:19:00+01:00"} 1

Or something like this

etcd_server_healthcheck_last_checked{type="readyz"} "2025-10-03T10:19:00+01:00"

[serathius]

I'm not following what problem we are trying to solve. The metrics are not a way to get the health status of etcd, we have dedicated endpoints for that. Metrics are meant to monitor availability aggregated over time. They are exposed in prometheus format, a standard monitoring protocol that doesn't guarantee values on freshly started server nor provides dates for a counter value. We will not expand nor customize Prometheus protocol.