Safeguard wrong execution order when restoring from backup

[pat-s]

Today I wanted to restore from a pgbackrest backup.

I've set the settings outlined in the doc and then ran deploy_pgcluster.yml. AFAICS though, I should first have run configure_pgcluster.yml as otherwise the backup action is not being picked up (?) and a new etcd cluster is initialized instead (?). Maybe I also needed to have set dcs_exists: true?

So right now my cluster is in a bad state as the etcd seems to have reinitialized and is now failing at

TASK [etcd : Wait for port 2379 to become open on the host] ******************************************************************************************************************
ok: [10.10.1.2]
ok: [10.10.1.3]
ok: [10.10.1.1]
FAILED - RETRYING: [10.10.1.2]: Wait until the etcd cluster is healthy (10 retries left).
FAILED - RETRYING: [10.10.1.1]: Wait until the etcd cluster is healthy (10 retries left).
[...]

I tried running configure_pgcluster.yml again followed by deploy_pgcluster.yml without success.

Is there any way to recover from this? I should have functional backups in S3. Happy to reinitialize everything. I think I have a broken etcd now as well.

---

From a user PoV, it would be great to have some guardrails saving one from executing one step too early or without all necessary settings in place. Specifically, running deploy_pgcluster.yml a second time with a functional cluster (and dcs_exists: false) will result in a broken etcd.

Also it feels like that all of these "DONTS" should be highlighted more prominently (or at all) in the "restore" documentation page. Another point there which is unclear: at the bottom, the command for PIT is shown. Yet it is not clear if one should just use deploy_pgcluster.yml without -t to perform a delta restore or if another tag is needed for this.

[vitabaks]

The deploy_pgcluster.yml designed to deploy a new cluster or clone an existing one. For restore existing cluster, use the point_in_time_recovery tag as described in the documentation — this ensures the playbook skips the extra steps and executes only what’s needed for recovery.

I’ll take your feedback into account — it might be worth creating a dedicated playbook, restore_pgcluster.yml, to avoid similar confusion in the future.

---

Free Support Discontinuation Notice

[pat-s]

Thanks. I think overall there should be multiple safeguards preventing the (wrong) secondary execution of deploy_.

Is there any way to recover from this state? I might not be the last one ending up with a broken etcd.

Is destroy + create new + restore from backup the easiest/only way?

[vitabaks]

I answered below, see if it's right for you.

But yes, you can destroy and deploy again (with patroni_cluster_bootstrap_method: "pgbackrest"):

ansible-playbook remove_cluster.yml -e "remove_etcd=true"
ansible-playbook deploy_pgcluster.yml -e "postgresql_exists=true"

Or

ansible-playbook remove_cluster.yml -e "remove_postgres=true remove_etcd=true"
ansible-playbook deploy_pgcluster.yml

[pat-s]

Thanks! I tried these, but this didn't properly cleanup etcd. Clearing /var/lib/etcd/* helped.

I was also able to properly restore from a backup, after having wiped /var/lib/pgdata.

---

The underlying issue which caused all of this was that pgbouncer failed and didn't properly startup anymore. The reason was that var/log/pgbouncer/ was owned by pgbouncer, causing write errors for user postgres. I don't know why that happened, can't recall if I ran any PB shortly before.

Thanks again for the help, it pushed my into the right direction and I was able to uncover some backup-related usage things which I'll take into account for the future.

[vitabaks]

the etcd seems to have reinitialized and is now failing at

You can try to fix the current ETCD cluster by examining its logs first:

sudo journalctl -u etcd -n 50 -f

Or deploy it again, first stopping the etcd process and deleting its data directory.

# on each node of the etcd cluster
sudo systemctl stop etcd
sudo rm -f /var/lib/etcd/*
sudo systemctl start etcd

[pat-s]

That worked, thanks.

Meanwhile I wiped both the PG cluster and etcd, but still can't property init/restore as the task starting PG itself fails (after the restore happened) (and with PG data dir wiped before).

I've read the related issues about restoring and everything should be in place. The restore via pgbackrest finishes successfully.
Writing this one the phone, hence no logs.
If something of the above rings a bell, let me know!