How to Autoscaling node VM on VMware VSphere

[walsptr]

I want to know if it is possible to scale nodes/VMs on Talos Linux VMs running on VMware vSphere. Scaling is done based on the workload.

[rothgar]

Autoscaling is accomplished outside of Talos. In general an autoscaler controller watches for unscheduled pods (due to resource constraints) and requests a new node from a virtualized environment (cloud provider, VMware, etc).

How the nodes get joined to the cluster is usually done via automation from the cloud provider (eg user-data) or pre-built images (hard coded values).

vSphere is not a supported cloud provider from the Kubernetes cluster autoscaler or Karpenter so you would need a custom auto scaler that calls vSphere APIs whenever a pod is unschedulable to create a new node that joins the cluster. There are some other open source vSphere autoscalers I found on GitHub but none of them appear to be maintained or support the current Kubernetes releases.

Nothing prevents you from doing this with Talos, but outside of Tanzu proprietary tooling I'm not sure how it would be accomplished without writing something yourself.

[walsptr]

thanks for the reply, so basically i need to write it manually right?

i want ask something again, with omni it is possible? or omni just can use for provisioning to?

[rothgar]

With every Kubernetes provider autoscaling is something that lives "outside" of the provisioning flow because it depends on the Kubernetes APIs. Even in hosted options (EKS, GKE, etc) the provisioning engine doesn't do the autoscaling, an autoscaler does. Sometimes the provider runs an autoscaler for you (eg EKS Auto), but that's usually up to you.

Omni could be part of the autoscaling automation with infrastructure providers, but it's not the current goal because Omni doesn't have access to your Kubernetes APIs.

For example, you could have something like this

graph LR
    Omni["Omni"]
    infra["infra<br/>provider"]
    vSphere["vSphere"]
    
    subgraph vSphere
        as["autoscaler"]
        k8s["k8s"]
    end
    
    as -->|"①"| k8s
    as --> |"②"| Omni
    Omni -->|"③"| infra
    infra -->|"④"| vSphere

    style vSphere fill:#fff4e6,stroke:#ff9800,stroke-width:3px,color:#000
    style k8s fill:#e3f2fd,stroke:#1976d2,stroke-width:2px

Loading

1. You run a cluster inside vSphere that runs a cluster autoscaler. That autoscaler watches for unschedulable pods and if it finds any does a calculation to figure out how many nodes would bee needed to make the pods fit.
2. The autoscaler calls Omni with a request to scale the cluster
3. The Infrastructure Provider is watching Omni cluster size and when it sees an increase in the number of requested nodes calls vSphere to add a VM to the cluster.

I have a video to show how this works without an autoscaler on YouTube.