failed to purge snap db file

[kelen121]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

etcd exit, log:
{"level":"fatal","ts":"2024-06-01T22:11:59.898283+0800","caller":"etcdserver/server.go:887","msg":"failed to purge snap db file","error":"open /data/trustworkspace/data/etcd/member/snap: permission denied","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).purgeFile\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:887\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2761"}

What did you expect to happen?

why exit?

How can we reproduce it (as minimally and precisely as possible)?

The etcd process exits irregularly.

Anything else we need to know?

No response

Etcd version (please run commands below)

etcd Version: 3.5.12
Git SHA: e7b3bb6
Go Version: go1.20.13
Go OS/Arch: linux/amd64

Etcd configuration (command line flags or environment variables)

name: 'default'

Path to the data directory.

data-dir: /data/trustworkspace/data/etcd

Path to the dedicated wal directory.

wal-dir: /data/trustworkspace/data/etcd-wal

#snapshot: false

Number of committed transactions to trigger a snapshot to disk.

snapshot-count: 1000000000000

Time (in milliseconds) of a heartbeat interval.

heartbeat-interval: 100

Time (in milliseconds) for an election to timeout.

election-timeout: 1000

Raise alarms when backend size exceeds the given quota. 0 means use the

default quota.

quota-backend-bytes: 0

List of comma separated URLs to listen on for client traffic.

listen-client-urls: http://10.21.20.209:2379

Maximum number of snapshot files to retain (0 is unlimited).

max-snapshots: 5

Maximum number of wal files to retain (0 is unlimited).

max-wals: 5

List of this member's client URLs to advertise to the public.

The URLs needed to be a comma-separated list.

advertise-client-urls: http://******:2379

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

No response

Relevant log output

No response

[kelen121]

{"level":"warn","ts":"2024-06-01T21:00:29.370356+0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"KMrdlShHiPnPkfzs.3120"}
{"level":"info","ts":"2024-06-01T21:05:30.474473+0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"fSlTYFvrhGsYotQp.3121"}
{"level":"warn","ts":"2024-06-01T21:10:58.003982+0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"fSlTYFvrhGsYotQp.3121"}
{"level":"info","ts":"2024-06-01T21:15:58.473863+0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"WjwHoyppYGeOgXWn.3122"}
{"level":"warn","ts":"2024-06-01T21:21:26.639016+0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"WjwHoyppYGeOgXWn.3122"}
{"level":"info","ts":"2024-06-01T21:26:27.473623+0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"nvRPEgXByBrJKfun.3123"}
{"level":"warn","ts":"2024-06-01T21:31:55.271978+0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"nvRPEgXByBrJKfun.3123"}
{"level":"info","ts":"2024-06-01T21:36:55.474155+0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"rlqIsSqsEwSsmMas.3124"}
{"level":"warn","ts":"2024-06-01T21:42:23.907282+0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"rlqIsSqsEwSsmMas.3124"}
{"level":"info","ts":"2024-06-01T21:47:24.474196+0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"QphKDbYtErENSquF.3125"}
{"level":"warn","ts":"2024-06-01T21:52:52.54077+0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"QphKDbYtErENSquF.3125"}
{"level":"info","ts":"2024-06-01T21:57:53.474383+0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"gQGiCpHxEkOeaTvf.3126"}
{"level":"warn","ts":"2024-06-01T22:03:21.176855+0800","caller":"auth/store.go:1183","msg":"invalid auth token","token":"gQGiCpHxEkOeaTvf.3126"}
{"level":"info","ts":"2024-06-01T22:08:21.473633+0800","caller":"auth/simple_token.go:170","msg":"deleted a simple token","user-name":"root","token":"IMSxAQofTGOcJShk.3127"}
{"level":"fatal","ts":"2024-06-01T22:11:59.898283+0800","caller":"etcdserver/server.go:887","msg":"failed to purge snap db file","error":"open /data/trustworkspace/data/etcd/member/snap: permission denied","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).purgeFile\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:887\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).GoAttach.func1\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2761"}

[jmhbnz]

Hi @kelen121 - As shown in the error logs above, the etcd server process is exiting because it cannot write the snapshot file. Please check and confirm which user the process is running under, and that the user can write to the file.

If the issue persists please provide clear step by step reproduce instructions so we can attempt to reproduce the same scenario at our end. Thanks.

[kelen121]

I am running as the trustworkspace user and have already granted the user 777 permissions, as follows:
[root@tws-dev logs]# ll /data/trustworkspace/data/etcd/member/snap
总用量 16408
-rwxrwxrwx. 1 trustworkspace trustworkspace 17166336 5月 31 17:54 db

[jmhbnz]

Thanks for the update @kelen121. Is this issue limited to a single machine or can you recreate it elsewhere?

Please provide step by step process to recreate the issue so we can attempt it on our side.

[kelen121]

[root@tws-dev logs]# ll /data/trustworkspace/data/etcd/member/snap
总用量 16408
-rwxrwxrwx. 1 trustworkspace trustworkspace 17166336 5月 31 17:54 db