Why is etcd db continuous growing? #9507
Comments
|
Can you try with etcd 3.3? |
|
Yes, we can try 3.3, but I'd like to know why, what causes this problem? Why are there so many http2server logs? Thanks so much for your support. |
|
3.3 uses new BoltDB storage backend with freelist fsync disabled (see https://coreos.com/blog/announcing-etcd-3.3). |
|
Thanks @gyuho , that means the db growing is due to the freelist fsync, right? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue:
etcd db is continuous growing until exceed the disk quota, even if there is no data written into db.
etcd version:
etcd Version: 3.2.5
Git SHA: d0d1a87
Go Version: go1.8.3
Go OS/Arch: linux/amd64
Finding:
The log "transport: http2Server.HandleStreams failed to read frame" is continuously printed.
Is this why db is increasing?
tail -f /var/log/messages| grep etcd
2018-03-29T05:03:24.792574-04:00 sps-me-auxiliary-0 etcd: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.20:2379->192.168.3.31:54138: read: connection reset by peer
2018-03-29T05:03:49.534597-04:00 sps-me-auxiliary-0 etcd: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.20:2379->192.168.3.10:39098: read: connection reset by peer
2018-03-29T05:04:43.474917-04:00 sps-me-auxiliary-0 etcd: start to snapshot (applied: 92038445, lastsnap: 91938444)
2018-03-29T05:04:43.477961-04:00 sps-me-auxiliary-0 etcd: saved snapshot at index 92038445
2018-03-29T05:04:43.478293-04:00 sps-me-auxiliary-0 etcd: compacted raft log at 92033445
2018-03-29T05:04:51.172359-04:00 sps-me-auxiliary-0 etcd: purged file /appdata/etcd/member/snap/0000000000000005-000000000574c404.snap successfully
2018-03-29T05:05:40.531373-04:00 sps-me-auxiliary-0 etcd: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.20:2379->192.168.3.40:43440: read: connection reset by peer
2018-03-29T05:05:55.992022-04:00 sps-me-auxiliary-0 etcd: transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.3.20:2379->192.168.3.30:33946: read: connection reset by peer
Metrics:
curl http://192.168.3.20:2379/metrics
HELP etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds Bucketed histogram of db compaction pause duration.
TYPE etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds histogram
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="1"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="2"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="4"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="8"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="16"} 2
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="32"} 27
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="64"} 29
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="128"} 1579
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="256"} 1590
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="512"} 1590
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="1024"} 1590
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="2048"} 1590
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="4096"} 1590
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="+Inf"} 1590
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_sum 153500
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_count 1590
HELP etcd_debugging_mvcc_db_compaction_total_duration_milliseconds Bucketed histogram of db compaction total duration.
TYPE etcd_debugging_mvcc_db_compaction_total_duration_milliseconds histogram
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="100"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="200"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="400"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="800"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="1600"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="3200"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="6400"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="12800"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="25600"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="51200"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="102400"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="204800"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="409600"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="819200"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="+Inf"} 1
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_sum 0
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_count 1
HELP etcd_debugging_mvcc_db_total_size_in_bytes Total size of the underlying database in bytes.
TYPE etcd_debugging_mvcc_db_total_size_in_bytes gauge
etcd_debugging_mvcc_db_total_size_in_bytes 4.1443328e+07
HELP etcd_debugging_mvcc_delete_total Total number of deletes seen by this member.
TYPE etcd_debugging_mvcc_delete_total counter
etcd_debugging_mvcc_delete_total 122
HELP etcd_debugging_mvcc_events_total Total number of events sent by this member.
TYPE etcd_debugging_mvcc_events_total counter
etcd_debugging_mvcc_events_total 0
HELP etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds Bucketed histogram of index compaction pause duration.
TYPE etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds histogram
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="0.5"} 0
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="1"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="2"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="4"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="8"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="16"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="32"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="64"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="128"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="256"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="512"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="1024"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="+Inf"} 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_sum 1
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_count 1
HELP etcd_debugging_mvcc_keys_total Total number of keys.
TYPE etcd_debugging_mvcc_keys_total gauge
etcd_debugging_mvcc_keys_total 2201
HELP etcd_debugging_mvcc_pending_events_total Total number of pending events to be sent.
TYPE etcd_debugging_mvcc_pending_events_total gauge
etcd_debugging_mvcc_pending_events_total 0
HELP etcd_debugging_mvcc_put_total Total number of puts seen by this member.
TYPE etcd_debugging_mvcc_put_total counter
etcd_debugging_mvcc_put_total 1.572619e+07
HELP etcd_debugging_mvcc_range_total Total number of ranges seen by this member.
TYPE etcd_debugging_mvcc_range_total counter
etcd_debugging_mvcc_range_total 3.41330237e+08
HELP etcd_debugging_mvcc_slow_watcher_total Total number of unsynced slow watchers.
TYPE etcd_debugging_mvcc_slow_watcher_total gauge
etcd_debugging_mvcc_slow_watcher_total 0
HELP etcd_debugging_mvcc_txn_total Total number of txns seen by this member.
TYPE etcd_debugging_mvcc_txn_total counter
etcd_debugging_mvcc_txn_total 270575
HELP etcd_debugging_mvcc_watch_stream_total Total number of watch streams.
TYPE etcd_debugging_mvcc_watch_stream_total gauge
etcd_debugging_mvcc_watch_stream_total 10
HELP etcd_debugging_mvcc_watcher_total Total number of watchers.
TYPE etcd_debugging_mvcc_watcher_total gauge
etcd_debugging_mvcc_watcher_total 38
HELP etcd_debugging_server_lease_expired_total The total number of expired leases.
TYPE etcd_debugging_server_lease_expired_total counter
etcd_debugging_server_lease_expired_total 1.592529e+07
HELP etcd_debugging_snap_save_marshalling_duration_seconds The marshalling cost distributions of save called by snapshot.
TYPE etcd_debugging_snap_save_marshalling_duration_seconds histogram
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.001"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.002"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.004"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.008"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.016"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.032"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.064"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.128"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.256"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.512"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="1.024"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="2.048"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="4.096"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="8.192"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="+Inf"} 915
etcd_debugging_snap_save_marshalling_duration_seconds_sum 0.019256652000000013
etcd_debugging_snap_save_marshalling_duration_seconds_count 915
HELP etcd_debugging_snap_save_total_duration_seconds The total latency distributions of save called by snapshot.
TYPE etcd_debugging_snap_save_total_duration_seconds histogram
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.001"} 1
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.002"} 849
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.004"} 914
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.008"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.016"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.032"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.064"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.128"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.256"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.512"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="1.024"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="2.048"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="4.096"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="8.192"} 915
etcd_debugging_snap_save_total_duration_seconds_bucket{le="+Inf"} 915
etcd_debugging_snap_save_total_duration_seconds_sum 1.4390841170000006
etcd_debugging_snap_save_total_duration_seconds_count 915
HELP etcd_debugging_store_expires_total Total number of expired keys.
TYPE etcd_debugging_store_expires_total counter
etcd_debugging_store_expires_total 0
HELP etcd_debugging_store_reads_total Total number of reads action by (get/getRecursive), local to this member.
TYPE etcd_debugging_store_reads_total counter
etcd_debugging_store_reads_total{action="get"} 15506
etcd_debugging_store_reads_total{action="getRecursive"} 4
HELP etcd_debugging_store_watch_requests_total Total number of incoming watch requests (new or reestablished).
TYPE etcd_debugging_store_watch_requests_total counter
etcd_debugging_store_watch_requests_total 0
HELP etcd_debugging_store_watchers Count of currently active watchers.
TYPE etcd_debugging_store_watchers gauge
etcd_debugging_store_watchers 0
HELP etcd_debugging_store_writes_total Total number of writes (e.g. set/compareAndDelete) seen by this member.
TYPE etcd_debugging_store_writes_total counter
etcd_debugging_store_writes_total{action="set"} 3
HELP etcd_disk_backend_commit_duration_seconds The latency distributions of commit called by backend.
TYPE etcd_disk_backend_commit_duration_seconds histogram
etcd_disk_backend_commit_duration_seconds_bucket{le="0.001"} 136909
etcd_disk_backend_commit_duration_seconds_bucket{le="0.002"} 2.144543e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.004"} 2.362002e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.008"} 2.364931e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.016"} 2.366637e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.032"} 2.366828e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.064"} 2.366875e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.128"} 2.366882e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.256"} 2.366887e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.512"} 2.366887e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="1.024"} 2.366887e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="2.048"} 2.366887e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="4.096"} 2.366887e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="8.192"} 2.366887e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="+Inf"} 2.366887e+06
etcd_disk_backend_commit_duration_seconds_sum 3356.71715119799
etcd_disk_backend_commit_duration_seconds_count 2.366887e+06
HELP etcd_disk_backend_snapshot_duration_seconds The latency distribution of backend snapshots.
TYPE etcd_disk_backend_snapshot_duration_seconds histogram
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.01"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.02"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.04"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.08"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.16"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.32"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.64"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="1.28"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="2.56"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="5.12"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="10.24"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="20.48"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="40.96"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="81.92"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="163.84"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="327.68"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="655.36"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="+Inf"} 1
etcd_disk_backend_snapshot_duration_seconds_sum 0.096143281
etcd_disk_backend_snapshot_duration_seconds_count 1
HELP etcd_disk_wal_fsync_duration_seconds The latency distributions of fsync called by wal.
TYPE etcd_disk_wal_fsync_duration_seconds histogram
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 7.55358e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 7.8382061e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 7.8529558e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.008"} 7.8544221e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.016"} 7.8550015e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.032"} 7.8550316e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.064"} 7.8550375e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.128"} 7.855043e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.256"} 7.8550547e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.512"} 7.8550547e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="1.024"} 7.8550547e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="2.048"} 7.8550547e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="4.096"} 7.8550547e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="8.192"} 7.8550547e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="+Inf"} 7.8550547e+07
etcd_disk_wal_fsync_duration_seconds_sum 46928.33328664243
etcd_disk_wal_fsync_duration_seconds_count 7.8550547e+07
HELP etcd_grpc_proxy_cache_hits_total Total number of cache hits
TYPE etcd_grpc_proxy_cache_hits_total gauge
etcd_grpc_proxy_cache_hits_total 0
HELP etcd_grpc_proxy_cache_keys_total Total number of keys/ranges cached
TYPE etcd_grpc_proxy_cache_keys_total gauge
etcd_grpc_proxy_cache_keys_total 0
HELP etcd_grpc_proxy_cache_misses_total Total number of cache misses
TYPE etcd_grpc_proxy_cache_misses_total gauge
etcd_grpc_proxy_cache_misses_total 0
HELP etcd_grpc_proxy_events_coalescing_total Total number of events coalescing
TYPE etcd_grpc_proxy_events_coalescing_total counter
etcd_grpc_proxy_events_coalescing_total 0
HELP etcd_grpc_proxy_watchers_coalescing_total Total number of current watchers coalescing
TYPE etcd_grpc_proxy_watchers_coalescing_total gauge
etcd_grpc_proxy_watchers_coalescing_total 0
HELP etcd_http_failed_total Counter of handle failures of requests (non-watches), by method (GET/PUT etc.) and code (400, 500 etc.).
TYPE etcd_http_failed_total counter
etcd_http_failed_total{code="404",method="GET"} 9
HELP etcd_http_received_total Counter of requests received into the system (successfully parsed and authd).
TYPE etcd_http_received_total counter
etcd_http_received_total{method="GET"} 9
HELP etcd_network_client_grpc_received_bytes_total The total number of bytes received from grpc clients.
TYPE etcd_network_client_grpc_received_bytes_total counter
etcd_network_client_grpc_received_bytes_total 5.922995388e+10
HELP etcd_network_client_grpc_sent_bytes_total The total number of bytes sent to grpc clients.
TYPE etcd_network_client_grpc_sent_bytes_total counter
etcd_network_client_grpc_sent_bytes_total 7.0564059569e+10
HELP etcd_network_peer_received_bytes_total The total number of bytes received from peers.
TYPE etcd_network_peer_received_bytes_total counter
etcd_network_peer_received_bytes_total{From="0"} 3.4672288e+07
etcd_network_peer_received_bytes_total{From="372fab067049cb8e"} 2.702327416e+10
etcd_network_peer_received_bytes_total{From="a1fbc63905319eb6"} 2.7511420949e+10
HELP etcd_network_peer_round_trip_time_seconds Round-Trip-Time histogram between peers.
TYPE etcd_network_peer_round_trip_time_seconds histogram
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.0001"} 0
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.0002"} 1
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.0004"} 58
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.0008"} 17210
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.0016"} 17296
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.0032"} 17302
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.0064"} 17303
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.0128"} 17303
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.0256"} 17304
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.0512"} 17304
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.1024"} 17304
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.2048"} 17304
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.4096"} 17304
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="0.8192"} 17304
etcd_network_peer_round_trip_time_seconds_bucket{To="372fab067049cb8e",le="+Inf"} 17304
etcd_network_peer_round_trip_time_seconds_sum{To="372fab067049cb8e"} 8.635961042999993
etcd_network_peer_round_trip_time_seconds_count{To="372fab067049cb8e"} 17304
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.0001"} 1
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.0002"} 1
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.0004"} 2851
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.0008"} 17260
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.0016"} 17292
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.0032"} 17294
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.0064"} 17295
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.0128"} 17295
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.0256"} 17296
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.0512"} 17296
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.1024"} 17296
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.2048"} 17296
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.4096"} 17296
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="0.8192"} 17296
etcd_network_peer_round_trip_time_seconds_bucket{To="a1fbc63905319eb6",le="+Inf"} 17296
etcd_network_peer_round_trip_time_seconds_sum{To="a1fbc63905319eb6"} 7.467577441999992
etcd_network_peer_round_trip_time_seconds_count{To="a1fbc63905319eb6"} 17296
HELP etcd_network_peer_sent_bytes_total The total number of bytes sent to peers.
TYPE etcd_network_peer_sent_bytes_total counter
etcd_network_peer_sent_bytes_total{To="372fab067049cb8e"} 3.4195159139e+10
etcd_network_peer_sent_bytes_total{To="a1fbc63905319eb6"} 3.4707172536e+10
HELP etcd_network_peer_sent_failures_total The total number of send failures from peers.
TYPE etcd_network_peer_sent_failures_total counter
etcd_network_peer_sent_failures_total{To="372fab067049cb8e"} 431389
etcd_network_peer_sent_failures_total{To="a1fbc63905319eb6"} 2
HELP etcd_server_has_leader Whether or not a leader exists. 1 is existence, 0 is not.
TYPE etcd_server_has_leader gauge
etcd_server_has_leader 1
HELP etcd_server_leader_changes_seen_total The number of leader changes seen.
TYPE etcd_server_leader_changes_seen_total counter
etcd_server_leader_changes_seen_total 3
HELP etcd_server_proposals_applied_total The total number of consensus proposals applied.
TYPE etcd_server_proposals_applied_total gauge
etcd_server_proposals_applied_total 9.2086175e+07
HELP etcd_server_proposals_committed_total The total number of consensus proposals committed.
TYPE etcd_server_proposals_committed_total gauge
etcd_server_proposals_committed_total 9.2086175e+07
HELP etcd_server_proposals_failed_total The total number of failed proposals seen.
TYPE etcd_server_proposals_failed_total counter
etcd_server_proposals_failed_total 0
HELP etcd_server_proposals_pending The current number of pending proposals to commit.
TYPE etcd_server_proposals_pending gauge
etcd_server_proposals_pending 0
HELP go_gc_duration_seconds A summary of the GC invocation durations.
TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.3167e-05
go_gc_duration_seconds{quantile="0.25"} 5.7157e-05
go_gc_duration_seconds{quantile="0.5"} 6.6448e-05
go_gc_duration_seconds{quantile="0.75"} 8.2414e-05
go_gc_duration_seconds{quantile="1"} 0.000297498
go_gc_duration_seconds_sum 3.5945791480000002
go_gc_duration_seconds_count 44627
HELP go_goroutines Number of goroutines that currently exist.
TYPE go_goroutines gauge
go_goroutines 231
HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 4.6222312e+07
HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 5.671141677832e+12
HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 2.363597e+06
HELP go_memstats_frees_total Total number of frees.
TYPE go_memstats_frees_total counter
go_memstats_frees_total 6.2705332148e+10
HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.8393856e+07
HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 4.6222312e+07
HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 9.67524352e+08
HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 6.26688e+07
HELP go_memstats_heap_objects Number of allocated objects.
TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 206356
HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 9.13358848e+08
HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.030193152e+09
HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.522314497247059e+09
HELP go_memstats_lookups_total Total number of pointer lookups.
TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 1.541458e+06
HELP go_memstats_mallocs_total Total number of mallocs.
TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 6.2705538504e+10
HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 895736
HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 6.79936e+06
HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 7.0366944e+07
HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 2.099243e+06
HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.540096e+06
HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.540096e+06
HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.081405688e+09
HELP grpc_server_handled_total Total number of RPCs completed on the server, regardless of success or failure.
TYPE grpc_server_handled_total counter
grpc_server_handled_total{grpc_code="OK",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 9
grpc_server_handled_total{grpc_code="OK",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 1
grpc_server_handled_total{grpc_code="OK",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 31
grpc_server_handled_total{grpc_code="OK",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 1.5797704e+07
grpc_server_handled_total{grpc_code="OK",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1.5716109e+07
grpc_server_handled_total{grpc_code="OK",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 3.41059665e+08
grpc_server_handled_total{grpc_code="OK",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 2
grpc_server_handled_total{grpc_code="OK",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 89528
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 4.299109e+07
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 3
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 104048
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 33
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 14
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 2
HELP grpc_server_msg_received_total Total number of RPC stream messages received on the server.
TYPE grpc_server_msg_received_total counter
grpc_server_msg_received_total{grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 9
grpc_server_msg_received_total{grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1
grpc_server_msg_received_total{grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 1
grpc_server_msg_received_total{grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 31
grpc_server_msg_received_total{grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 5.8788794e+07
grpc_server_msg_received_total{grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1.5716112e+07
grpc_server_msg_received_total{grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 3.41059669e+08
grpc_server_msg_received_total{grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 2
grpc_server_msg_received_total{grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 193576
grpc_server_msg_received_total{grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 64
HELP grpc_server_msg_sent_total Total number of gRPC stream messages sent by the server.
TYPE grpc_server_msg_sent_total counter
grpc_server_msg_sent_total{grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 9
grpc_server_msg_sent_total{grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 1
grpc_server_msg_sent_total{grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 31
grpc_server_msg_sent_total{grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 1.5797704e+07
grpc_server_msg_sent_total{grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1.5716109e+07
grpc_server_msg_sent_total{grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 3.41059665e+08
grpc_server_msg_sent_total{grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 2
grpc_server_msg_sent_total{grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 89528
grpc_server_msg_sent_total{grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 64
HELP grpc_server_started_total Total number of RPCs started on the server.
TYPE grpc_server_started_total counter
grpc_server_started_total{grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 9
grpc_server_started_total{grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1
grpc_server_started_total{grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 1
grpc_server_started_total{grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 31
grpc_server_started_total{grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 5.8788794e+07
grpc_server_started_total{grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 36
grpc_server_started_total{grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1.5716112e+07
grpc_server_started_total{grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 3.41059669e+08
grpc_server_started_total{grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 2
grpc_server_started_total{grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 193576
grpc_server_started_total{grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 24
HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} NaN
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} NaN
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="prometheus"} 7533.451
http_request_duration_microseconds_count{handler="prometheus"} 3
HELP http_request_size_bytes The HTTP request sizes in bytes.
TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} NaN
http_request_size_bytes{handler="prometheus",quantile="0.9"} NaN
http_request_size_bytes{handler="prometheus",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="prometheus"} 198
http_request_size_bytes_count{handler="prometheus"} 3
HELP http_requests_total Total number of HTTP requests made.
TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 3
HELP http_response_size_bytes The HTTP response sizes in bytes.
TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} NaN
http_response_size_bytes{handler="prometheus",quantile="0.9"} NaN
http_response_size_bytes{handler="prometheus",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="prometheus"} 101898
http_response_size_bytes_count{handler="prometheus"} 3
HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 204360.99
HELP process_max_fds Maximum number of open file descriptors.
TYPE process_max_fds gauge
process_max_fds 65536
HELP process_open_fds Number of open file descriptors.
TYPE process_open_fds gauge
process_open_fds 57
HELP process_resident_memory_bytes Resident memory size in bytes.
TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.16420352e+08
HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
TYPE process_start_time_seconds gauge
process_start_time_seconds 1.52179801596e+09
HELP process_virtual_memory_bytes Virtual memory size in bytes.
TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.31682304e+09
The text was updated successfully, but these errors were encountered: