Restored-volume (clone) gets created only on the snapshot node

[deepakbnsl]

Describe the bug
Snapshot clone doesn't get created on other nodes.

To Reproduce
Cluster with 2 nodes

• N1 has a snapshot of an existing volume.
• create a storage class with node affinity (topology) to N2.
• using above storage class, create a pvc with data source as above snapshot.
• using above pvc, create a pod with node affinity to N2.
• pods gets created on N2 while new volume gets created on N1 only (confirmed by "nvme list-subsys")

Expected behavior
New volume should get created on N2 (for local disk access). And this volume should reference read-only snapshot data internally from N1.

OS info:

• Distro: [e.g. NixOS] : Ubuntu
• Kernel version : 5.15.0-125-generic
• MayaStor revision : 2.7.1
  registry.k8s.io/sig-storage/csi-provisioner:v3.5.0
  registry.k8s.io/sig-storage/csi-attacher:v4.3.0
  registry.k8s.io/sig-storage/csi-snapshotter:v6.3.3
  registry.k8s.io/sig-storage/snapshot-controller:v6.3.3
  registry.k8s.io/sig-storage/csi-resizer:v1.9.3
  docker.io/openebs/mayastor-csi-controller:v2.7.1

[tiagolobocastro]

I suspect the topology feature is not taking the existing snapshot into account CC @sinhaashish
A restored replica of a snapshot is always located on the same pool, this is because it's using COW, so it's not a full copy of the data if you will.

I wonder what the behaviour should be in this case if the topology does not match. We have 2 options:

1. We fail the restored volume creation with invalid arguments
2. We do a full copy of the volume into pools matching the topology

If you want the restored volume to have the correct topology you can increase the replica count of the volume and then decrease it again, though it may not work without this fix: openebs/mayastor-control-plane#906