Scaling Bare-Metal Provisioning to Thousands of Nodes

[rcorujo]

Hello,

I’m evaluating Omni for a large-scale deployment and would like to understand best practices for provisioning many worker nodes concurrently. Specifically, how can I provision, for example, 10,000 bare-metal worker nodes without the image server becoming a bottleneck?

Are there recommended strategies, caching mechanisms, or orchestration techniques to handle this scale efficiently?

Thank you for your guidance.

[utkuozdemir]

Hi,

Assuming you plan to self-host Omni, you can also self-host the image-factory in air-gapped mode: https://github.com/siderolabs/image-factory#air-gapped-mode

As it builds the initramfs for every unique schematic only once, the bottleneck will probably be the network bandwidth between the nodes and the factory.

It also supports CDN cache, which is used already in factory.talos.dev.

[rcorujo]

Thank you again for your response.

I watched the video, but as I understand it, the Image Cache only prevents nodes from pulling the OS image directly from the Internet.

In the video, the following command was used:

talosctl cluster create --vmlinuz-path ./vmlinux-amd64 --initrd-path ./initramfs-amd64.xz --masters 3 --workers 1 ...

Now, suppose I want to provision 10,000 worker nodes using --workers 10000. When all these servers PXE boot simultaneously, wouldn’t they all access the same HTTP or TFTP server concurrently to fetch the kernel and initrd? Wouldn’t that risk saturating the HTTP or TFTP server?

Similarly, once the initrd is loaded, wouldn’t each worker pull the OS image from the same Image Cache, potentially overloading that cache server?

Is there currently any mechanism in Omni to distribute the load during PXE boot, so that not all worker nodes access the same HTTP/TFTP server or the same Image Cache at the same time?

Thank you for your guidance.

[smira]

Image Cache is local to the machine, so part of the answer there is no bottleneck here with 10000 nodes.

If you want to PXE boot 10000 nodes, that happens outside of Omni/Talos reach. For example, if you use Talos Image Factory, you can configure to PXE boot from it, and it will deliver from the CDN. You can make it local to your network as well, but this how you provision your nodes, this happens even before Talos boots.

[rcorujo]

Thank you. So, if I understood your responses correctly:

Neither Omni nor Talos currently:

• Automatically distributes PXE boot traffic across multiple servers.
• Implements peer-to-peer image sharing (like BitTorrent).
• Coordinates large-scale boot waves.

So to handle 10,000-node provisioning, you’d need to:

• Architect your own PXE load distribution layer (multiple TFTP/HTTP servers, DHCP balancing, etc.).
• Optionally mirror Talos Image Factory locally to reduce WAN load.
• Possibly stagger node boots (not all 10k at once).

Thanks.

[rothgar]

Correct. Omni is outside of the boot path and doesn't provide boot assets (image factory does). The image factory has CDN support for distributing boot assets as machines boot. You can distribute images internally via an s3 compatible API endpoint documented here https://github.com/siderolabs/image-factory#cdn-cache

Talos is just the operating system that has the options for local, on-disk image caching and configuring registry mirrors or caches for container images. How you distribute those images is up to you.

As an aside, I don't know any provisioning system or infrastructure that recommends booting 10,000 nodes all at once. In every case you would want to evaluate what your requirements are, what your budget is, and how much resiliency, security, and performance you can afford.

A single Omni instance is tested to handle about 5000 nodes. If you'll need more than that please contact our sales and support teams to understand your requirements better and help you achieve what you need.

[rcorujo]

@rothgar Thank you for your response.