Add secret details to the clientProfile CR

[Madhu-1]

Today ceph-csi is heavily dependent on the kubernetes sidecar to provide the secrets we have faced this problem multiple times where we needed to change the secrets or where we need to add the additional secrets to all the classes.

This is a feature request to solve the secret override or missing secret problem with ceph-csi without affecting the upgrade. Because we cannot add the additional secrets to the kubernetes PV objects as they are immutabe nor it can be changed as well.

In ClientProfile we are going to expose a set of fields where admin can specify the secrets details per clusterID/profile and cephcsi is going get the secret from kubernetes and use that secret to talk to ceph.

This will solve most of the problem we have with secrets.

[leelavg]

This was brought up yesterday in one of the meetings, we also need to make DR stack aware of these overrides (or extra set). Just leaving not to miss.

[nb-ohad]

This need some discussion, I do think that we have a problem with sticky secret names, but I don't think ceph-csi-operator APIs is the current place to expose such knobs. ceph-csi-operator is about the orchestration of the CSI drivers themselves not usage configuration

[Madhu-1]

This need some discussion, I do think that we have a problem with sticky secret names, but I don't think ceph-csi-operator APIs is the current place to expose such knobs. ceph-csi-operator is about the orchestration of the CSI drivers themselves not usage configuration

I think there is some gap with the assumptions here, CSI needs some configurations that can only be provided by the ceph-csi-operator. not exactly sure what was the decisions even had in yesterday meeting, Do we have some notes about what was discusssed and whats its all about?