3 controlplanes in 3 geographic locations over Kubespan

[Xysto]

Hello,

I know this setup is not recommended due to latency and etcd limitations, but out of curiosity — is it possible to have three control plane nodes in three different locations, connected only over the internet via KubeSpan?

I tried to test this idea using three Raspberry Pi 4B devices. The control plane configuration went fine with the default settings — each node can “see” the others as KubeSpan peers. However, when I run talosctl bootstrap on one node, the other two never seem to join the cluster, and the first one completes the bootstrap alone, forming a single-node cluster.

I’m obviously missing something important here. Sorry if this is a naïve question, but why don’t the other two nodes join over KubeSpan? Do I need to specify any additional configuration besides kubespan.enabled: true?

[smira]

Please see documentation on network requirements.

If the peers not only "see" each other, but establish a connection, they should automatically form an etcd cluster.

[Xysto]

Thanks for your quick answer :) For additional details, each of my nodes is located behind a separate home gateway. Every gateway is configured to forward UDP port 51820 to the node within its respective local network.

The output of talosctl get kubespanpeerspec on my first node is:

NODE            NAMESPACE   TYPE               ID                                             VERSION   LABEL           ENDPOINTS
192.168.1.xxx   kubespan    KubeSpanPeerSpec   <KUBESPANID-node2>   2         talos-3xm-g2t   ["192.168.1.yyy:51820","[<IPv6-node2>]:51820","<publicIPv4-node2>:51820"]
192.168.1.xxx   kubespan    KubeSpanPeerSpec   <KUBESPANID-node3>   5         talos-p81-6vs   ["192.168.1.zzz:51820","[<IPv6-node3>]:51820","<publicIPv4-node3>:51820"]

And the output of talosctl get kubespanpeerstatuses is:

NODE            NAMESPACE   TYPE                 ID                                             VERSION   LABEL           ENDPOINT                                       STATE   RX        TX
192.168.1.xxx   kubespan    KubeSpanPeerStatus   1xVqOWyvSHMtaCiXeo/GPmfI3wRf2sQGbrInIJUqTDo=   366       talos-3xm-g2t   [<IPv6-node2>]:51820   up      3747548   2564284
192.168.1.xxx   kubespan    KubeSpanPeerStatus   D9G8MimDy96mvXGMSP6rkC4BxYjrmBS6RFR6bXjcens=   220       talos-p81-6vs   [<IPv6-node3>]:51820   up      2253144   1531124

However, talosctl get kubespanendpoints is empty. I think that matches your explanation that the peers are not in an established connection? But why is that the case? I’ve made sure that each gateway performs a straightforward UDP 51820 port forward to its respective node in the local network.

[smira]

If KubeSpan state is up, they can talk to each other, and should establish the cluster.

You need to dig into the logs to see why etcd doesn't join, it should.

[Xysto]

Indeed, etcd is the problem. It refuses to compose the cluster because of the latency (apply requests take ~230ms, but the threshold is <100ms). It was worth the try! Thanks a lot to pointed me in the right direction 😃