Enable users to customize BTP Operator secret and configMap

[PK85]

Description

We will allow to modify and keep user modifications in the btp-manager-secret.

Reason

Users wants to migrate btp-operator resources and configurations from one k8s cluster into new Kyma cluster.

AC:

• 1) support label on our btp manager secret sap-btp-manager to skip reconciliation. If secret deleted then reconciliation is back.
  • 1.1) In the BTP manager plutono add dashboard about number of skipped, and be able to see a list of ShootName from those skipped ones
• 2) if sap-btp-manager secret is changed, immediately reconcile btp-operator
  • 2.1) CM modification is not visible to the btp-operator-controller pod. Only restart of the pod will load it again
  • 2.2) Secret sap-btp-operator-clusterid is always created and INITIAL_CLUSTER_ID from it is always in use. When pod will restart from 2.1 then btp-operator-controller will fail to start because CLUSTER_ID from CM != INITIAL_CLUSTER_ID from secret. panic: ClusterID changed, which is not supported. Please redeploy with --set cluster.id=test_cluster_id goroutine 1 [running]: main.main() /workspace/main.go:140 +0x1d65
  • 2.3) TODO in the BTP-Manager: if user changes CLUSTER_ID in the btp-manager-secret then BTP-Manager deletes a secret sap-btp-operator-clusterid, checks if CM: sap-btp-operator-config has new CLUSTER_ID set, restart sap-btp-operator-controller-manager-58fbb647ff-72ckx pods, checks if sap-btp-operator-clusterid is back with new INITIAL_CLUSTER_ID.
• 3) E2E integration test
  Issue: End-to-end (e2e) integration test for customizing BTP Manager input secret #921
  PR: E2E BTP Manager secret customization test  #935
• 4) docs
• 5) RNs

Agreed label name:

• kyma-project.io/skip-reconciliation: 'true'

NOTE!:

• I changed manually CLUSTER ID in our secret, then config map was having it but still instances were created with previous previous CLUSTER ID, see code: https://github.com/SAP/sap-btp-service-operator/blob/9cebc45aded4d8b58396f913720168b0d1307f73/main.go#L208

[szwedm]

AC:
...

• 3) E2E integration test, ticket: End-to-end (e2e) integration test for customizing BTP Manager input secret #921

PR with the e2e test: #935