Cluster Recovery

[jhuynh08]

I have been running a personal homelab cluster for over a month, with a single node (n1) control plane on Omni, with daily backups configured. I was dealing with some issues with Ceph and trying to do a Ceph Cluster restore, but due to the Ceph cluster issue, the k8s-api was getting hit pretty hard and so was dealing with intermittent outages.

I was then trying to do a swap out and change the machines in the cluster (a1, a2, a3) being CPs and n1 switching over to worker.

Typically, I would do ⁠ omnictl delete machineset -control-planes ⁠ but this cluster was stuck in deletion and I accidentally ended up deleting the cluster itself (big mistake).

So now when I tried to do ⁠ omnictl cluster template sync -f  ⁠ - it isn’t able to validate the cluster UUID, since the cluster itself is deleted.

I then did rclone of the old cluster’s snapshot over to the new cluster’s object storage location so that it would exist, and modified the cluster template with the new UUID, but it wouldn’t take, likely due to the snapshot itself being recorded in omni’s etcd, since the error was: ⁠ * error getting backup data: failed to get: resource BackupDatas.omni.sidero.dev(default/@undefined) doesn't exist ⁠

I know I should’ve backed up the etcd manually with ⁠ talosctl ⁠ and this would’ve made life so much easier, but now I’m stuck.

Would it be possible to restore the previous snapshot somehow?

[rothgar]

Are these the steps you followed to restore the cluster backup? https://omni.siderolabs.com/how-to-guides/restore-etcd-of-a-cluster-managed-by-cluster-templates

[jhuynh08]

Hi,

Yes those are the exact instructions I used. I get this error when attempting to recover the etcd from the previous cluster to a new cluster.

[jhuynh08]

Should I use the new cluster's UUID in the template instead of the cluster that was deleted?

[brantgurga]

The etc backups are encrypted with a key tying it to that cluster. That's why the recovery instructions say not to delete the cluster. If you've deleted the cluster, I don't know that you can recover since there's no means to decrypt the etcd backup.