Managed control planes

[Unix4ever]

Tasks

Beta Give feedback

1. create a controller for handling machine request set which is created for managed control planes #705
   +0
   
2. managed control planes config patch management #725
   +0
   
3. support setting machine patches in the infra provider #728
   +0
   
4. -- POC
5. support special resource operation mode in Omni which doesn't give full access to the user #550
   +0
   
6. block Talos API access to the managed control planes
7. hidden infra provider support #717
   +0
8. control planes monitoring
9. logging into the client's account for the users with godmode access #551
   +0
10. filter the managed resources, not showing them to the user #594
    +0
11. machine set node controller should support the mode which allocates machines from a specific machine request set #595
    +0
    
12. support the mode when the control plane node doesn't register itself in the kubelet
13. disable downgrades for the clusters with managed control plane
14. have more strict list of usable Talos versions for the clusters with managed control planes

[Unix4ever]

Managed control planes creation flow:

1. Create a cluster and enable a feature use managed control planes.
2. MachineRequestsSet should be created (1 or 3 depending on the tier, we can start with 1 node).
3. Create a controlplane MachineSet that gets the machines from the created MachineRequestSet.
4. We should create the ConfigPatch that configures the node right for the managed mode.

Note: managed control planes should be ignored by the machine class controller, and the state validation shouldn't allow allocating the nodes if they are allocated with the readonly annotation.

Need to decide what we're going to do with the etcd backups feature for the cluster.

[Unix4ever]

POC UX:

1. The POC works only with Talos 1.8.1+.
2. Introduce a flag in Omni that will enable managed control planes support.
3. Cluster template ConrolPlanes section gets another configuration setting:

---
kind: ControlPlane
managed:
  enabled: true
  provider: SideroLabs
  size: xs

4. Cluster gets created with the control planes allocated in the configured infra provider for managed control planes.
5. The user can see the created machines, can't do anything with them, can't use talosctl command against them.
6. Kubectl get nodes will show the control planes and get pods will show the kube-system static pods.
7. Cluster should have pre-configured config patches: kubespan, metrics endpoints. They should be read-only/hidden for the user.
8. The POC will be a separate instance of Omni and it's whole state will be dropped after user testing.