How to resolve defragmentation under large data volumes

[Black-max12138]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via [email protected].
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

As the etcd data grows, the size of the etcd database increases accordingly. When the etcd data reaches 4.1GB, a single defragmentation operation takes about 15 seconds. If this defragmentation occurs on the primary node, the cluster cannot write any data during this time. Is there any good way to handle this situation?
We thought of one solution: setting a threshold for the database. If the primary node reaches this threshold, we could elect a new primary node and then perform disk defragmentation on the old primary node. Would this approach be feasible? Or is there any other better solution?

What did you expect to happen?

Defragmentation should not affect cluster operations.

How can we reproduce it (as minimally and precisely as possible)?

Constructing a situation where the database size is relatively large

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here

$ etcdctl version
# paste output here

3.5.11

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

[ahrtr]

https://github.com/ahrtr/etcd-defrag this is the best solution the community can provide for now.

It supports rule based automatic compaction + defragmentation.

You can also set --move-leader to move the leadership before performing defragmentation on the leader, defaults to false.

Please read the readme of the project.