br backup failed with error “rpc error: code = Unavailable desc = etcdserver: leader changed” after injection pd leader io delay

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run br backup
2、injection pd leader io delay

2. What did you expect to see? (Required)

br backup succeed

3. What did you see instead (Required)

br backup failed
br.log.2024-10-31T04.27.51Z.zip

Detail BR log in /tmp/br.log.2024-10-31T04.27.51Z
{level:warn,ts:2024-10-31T04:28:54.823024Z,logger:etcd-client,caller:[email protected]/retry_interceptor.go:62,msg:retrying of unary invoker failed,target:etcd-endpoints://0xc0022a08c0/tc-pd.ha-test-br-backup-tps-7643347-1-341:2379,attempt:0,error:rpc error: code = Unavailable desc = etcdserver: leader changed}
{level:warn,ts:2024-10-31T04:28:54.966914Z,logger:etcd-client,caller:[email protected]/retry_interceptor.go:62,msg:retrying of unary invoker failed,target:etcd-endpoints://0xc0022a08c0/tc-pd.ha-test-br-backup-tps-7643347-1-341:237"

4. What is your TiDB version? (Required)

./br -V
Release Version: v8.5.0-alpha-19-g49c3eba4b0
Git Commit Hash: 49c3eba
Git Branch: HEAD
Go Version: go1.23.2
UTC Build Time: 2024-10-30 15:51:01
Race Enabled: false

[Lily2025]

/assign Leavrth

[Lily2025]

/severity major

[Leavrth]

[tikv:9001]PD server timeout: 
github.com/pingcap/errors.AddStack
	/root/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:178
github.com/pingcap/errors.(*Error).GenWithStackByArgs
	/root/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:175
github.com/pingcap/tidb/pkg/store/driver/error.ToTiDBErr
	/workspace/source/tidb/pkg/store/driver/error/error.go:114
github.com/pingcap/tidb/pkg/store/driver/txn.(*tikvSnapshot).Iter
	/workspace/source/tidb/pkg/store/driver/txn/snapshot.go:72
github.com/pingcap/tidb/pkg/structure.(*TxStructure).IterateHash
	/workspace/source/tidb/pkg/structure/hash.go:226
github.com/pingcap/tidb/pkg/structure.(*TxStructure).HGetAll
	/workspace/source/tidb/pkg/structure/hash.go:157
github.com/pingcap/tidb/pkg/meta.(*Mutator).ListDatabases
	/workspace/source/tidb/pkg/meta/meta.go:1235
github.com/pingcap/tidb/br/pkg/backup.BuildBackupSchemas
	/workspace/source/tidb/br/pkg/backup/client.go:826
github.com/pingcap/tidb/br/pkg/backup.BuildBackupRangeAndInitSchema.func2
	/workspace/source/tidb/br/pkg/backup/client.go:812
github.com/pingcap/tidb/br/pkg/backup.(*Schemas).BackupSchemas
	/workspace/source/tidb/br/pkg/backup/schema.go:95
github.com/pingcap/tidb/br/pkg/task.RunBackup
	/workspace/source/tidb/br/pkg/task/backup.go:713
main.runBackupCommand
	/workspace/source/tidb/br/cmd/br/backup.go:56
main.newDBBackupCommand.func1
	/workspace/source/tidb/br/cmd/br/backup.go:163
github.com/spf13/cobra.(*Command).execute
	/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
	/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117
github.com/spf13/cobra.(*Command).Execute
	/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041
main.main
	/workspace/source/tidb/br/cmd/br/main.go:36
runtime.main
	/usr/local/go/src/runtime/proc.go:272
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1700