[Bug]: Failed to recover observer after K8s cluster recovers from abnormal status

[chris-sun-star]

Describe the bug

observer keeps waiting for the pod to be ready, but the pod has not been created successfully

Environment

a one node obcluster

Fast reproduce steps

delete the observer pod, wait observer create the pod and starts waiting the pod to be ready, delete the pod

Expected behavior

No response

Actual behavior

No response

Additional context

observer recover procedure can rerun from the start, maybe change the policy to startover and do some check before doing every task will fix this