Add multi cluster secrets and controller

[mikenairn]

What

Add the required controller logic to watch for multi cluster secrets and register them as "clusters" within the dns operator.

How

• Add watch on secrets in a specified namespace e.g. kuadrant-system with a given label e.g. kuadrant.io/multicluster-kubeconfig: "true"
• The namespace and label should be configurable via flags to the controller:
  • --cluster-secret-namespace The Namespace to look for cluster secrets. default: kuadrant-system
  • --cluster-secret-label The label that identifies a Secret resource as a cluster secret. default: kuadrant.io/multicluster-kubeconfig
• Cluster secrets are expected to contain kubeconfig data, when one is found it should be registered as a "cluster" and a watch for DNSRecord resources started inside a runnable go routine.
• If a cluster secret is removed, any cluster resource and watch created should be removed.
• When a DNSRecord resource gets an update it should call the "reconcile function". This task will just stub this out for now.
• Add a new role to the dns operator that contains the minimum required permissions needed for a "primary" cluster to process the DNSRecord resources on a "remote" cluster.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: remote-cluster-role
rules:
- apiGroups:
    - kuadrant.io
  resources:
    - dnsrecords
  verbs:
    - get
    - list
    - watch
- apiGroups:
    - kuadrant.io
  resources:
    - dnsrecords/status
  verbs:
    - get
    - patch
    - update

Note: For the POC i used the multicluster controller runtime package which is created by the KCP folks, but experimental. It did work well, and was easily integrated using a kubeconfig provider, but we should talk about the use of this and alternatives before committing to it.

Expected Outcomes

• Update local-setup to include a means to create two kind clusters clusters with one configured as a "primary".
• Re-evaluate multicluster controller runtime package and if we can use it.
• Multi cluster secret controller logic added
• Watch maintained on DNSRecords on the remote clusters with appropriate rbac defined.
• Stub function for reconcile called on updates (just log the requests for now).

Note: Does not require DNSRecord reconciliation logic to be completed, follow on tasks will deal with that.

[FourFifthsCode]

Just a heads up, merged custom schema support in kubeconfig provider kubernetes-sigs/multicluster-runtime#52