snapshot backup of multi-instances application should only take snapshot of the same PVC once

[famfop]

Describe the problem/challenge you have

Currently it is particularly difficult to configure a backup of a multi-node app. E.g. I am running an elasticsearch cluster with 3 nodes (one master). It is easy to configure a pre-hook to check whether the node is a master node and run snapshot commands correspondingly, but ultimately my PodAnnotations are on all nodes of my app and thus a snapshot-dir is backed up 3 times instead of one - I am using file-system backup but assume this will be the same for volumesnapshot backups.

Describe the solution you'd like
I would like to be able to configure the backup on "service level", i.e. my headless service is aware of my cluster-endpoint, I could already configure snapshots there and only run one backup of the data - if my service was aware of that data.

Anything else you would like to add:
The same applies to mongo and probably all "multi-node" or "horizontally scaled" services.

[sseago]

VolumeGroupSnapshot support is on the road map. That should help use cases like this.

[blackpiglet]

I want to check what is expected for the cluster application's backup.
Does the ES cluster's multiple instances mount the same volume?

If the same volume is mounted multiple times, Velero should find a way to back up it only once instead of multiple times.
If the ES cluster instances mount different volumes, I'm not sure what the expected behavior is.
Theoretically, the data in the volumes can be different.

[famfop]

Yes, the (in my case) three nodes mount the same volume which is reserved for snapshots only. Sure, if I have different volumes I would want to create backups of each of them which would already be solved by the PodAnnotation.

I guess you know better how to implement this but I see different options on how to: Either using Services which would make sense ("Do backup of service X") or at least give the option to create backups of deployments/statefulsets but not on Pod level.

[blackpiglet]

I modified the title to reflect the issue to solve in more detail.

velero/pkg/backup/item_backupper.go

Lines 214 to 230 in 804d73c

	if shouldDoFSBackup {
	// track the volumes backing up by PVB , so that when we backup PVCs/PVs
	// via an item action in the next step, we don't snapshot PVs that will have their data backed up
	// with pod volume backup.
	ib.podVolumeSnapshotTracker.Track(pod, volume.Name)
	if found, pvcName := ib.podVolumeSnapshotTracker.TakenForPodVolume(pod, volume.Name); found {
	log.WithFields(map[string]interface{}{
	"podVolume": volume,
	"pvcName": pvcName,
	}).Info("Pod volume uses a persistent volume claim which has already been backed up from another pod, skipping.")
	continue
	}
	pvbVolumes = append(pvbVolumes, volume.Name)
	} else {
	ib.podVolumeSnapshotTracker.Optout(pod, volume.Name)
	}

The fs-backup already supports the scenario you require. Thus, the issue is Velero should only take one snapshot of the same PVC in one backup.

[famfop]

Well, I noticed that you are right: My backup only includes one pod volume backup, not three as thought. So, this is on me, sorry. From my side the issue may be closed unless you want to keep it open for tracking something, feel free to close.

[blackpiglet]

Got it. I will keep this issue open to track the duplicate snapshot issue in backup.

[blackpiglet]

Need to split this issue for data-mover and CSI snapshots backup.