How to manually replace old cluster nodes and add new ones in different datacenters?

[algoritmsystems]

Hi,

I initially had a 3-node PostgreSQL cluster configured with Patroni:

patronictl list
+ Cluster: postgres-cluster (7447180793296821424) +----+-----------+
| Member | Host        | Role         | State     | TL | Lag in MB |
+--------+-------------+--------------+-----------+----+-----------+
| db-1   | 172.16.9.30 | Sync Standby | streaming | 24 |         0 |
| db-2   | 172.16.9.31 | Sync Standby | streaming | 24 |         0 |
| db-3   | 172.16.9.32 | Leader       | running   | 24 |           |

Later, I added 4 more nodes:

patronictl list
+ Cluster: postgres-cluster (7447180793296821424) +----+-----------+
| Member | Host        | Role         | State     | TL | Lag in MB |
|--------|-------------|--------------|-----------|----|-----------|
| db-1   | 172.16.9.30 | Sync Standby | streaming | 26 |         0 |
| db-2   | 172.16.9.31 | Sync Standby | streaming | 26 |         0 |
| db-3   | 172.16.9.32 | Leader       | running   | 26 |           |
| db-4   | 172.16.9.41 | Sync Standby | streaming | 26 |         0 |
| db-5   | 172.16.9.42 | Sync Standby | streaming | 26 |         0 |
| db-6   | 172.16.9.43 | Sync Standby | streaming | 26 |         0 |
| db-7   | 172.16.9.44 | Sync Standby | streaming | 26 |         0 |

As DCS used etcd!

cat /etc/etcd/etcd.conf 
ETCD_NAME="db-1"
ETCD_LISTEN_CLIENT_URLS="http://172.16.9.30:2379,http://127.0.0.1:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.9.30:2379"
ETCD_LISTEN_PEER_URLS="http://172.16.9.30:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.9.30:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-postgres-cluster"
ETCD_INITIAL_CLUSTER="db-1=http://172.16.9.30:2380,db-2=http://172.16.9.31:2380,db-3=http://172.16.9.32:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_ELECTION_TIMEOUT="5000"
ETCD_HEARTBEAT_INTERVAL="1000"
ETCD_INITIAL_ELECTION_TICK_ADVANCE="false"
ETCD_AUTO_COMPACTION_RETENTION="1"


etcdctl member list
59f503c5914e5cba, started, db-1, http://172.16.9.30:2380, http://172.16.9.30:2379, false
791c49c7afe4bb54, started, db-3, http://172.16.9.32:2380, http://172.16.9.32:2379, false
dd8a60b00d165c33, started, db-2, http://172.16.9.31:2380, http://172.16.9.31:2379, false

Now, I need to:
-Add 3 more nodes in the new datacenter: 172.16.9.45, 172.16.9.46, 172.16.9.47
-And remove the original nodes from old: 172.16.9.30, 172.16.9.31, 172.16.9.32 (they're only with etcd now)

What is the correct manual procedure for doing this safely?
Thanks!

[vitabaks]

Hi @algoritmsystems

At the moment, DCS cluster scaling automation is not yet implemented (it’s planned here).
For now, you can manually use etcdctl member add / remove as described in the official etcd documentation.

Alternatively, you can consider a support package — we’ll assist you with the setup and can also prioritize automation of etcd cluster scaling as part of the roadmap.

---

Free Support Discontinuation Notice