joining a cluster with another address creates inconsistent canonical node addresses

[qrkourier]

On ctrl2, I joined the cluster by calling the address of the first node's ctrl plane listener, not its public, advertised address:

ziti agent cluster add --tcp-addr "127.0.0.1:10001" "tls:ziti-ctrl1-controller-ctrl:1280"

My goal in calling the private, internal ctrl address was to form cluster links between the nodes on their shared, private, underlay fabric. In this case, the pod network where they are deployed in k8s.

As a result of the following inconsistency, I fell back to forcing the inter-node ctrl links to hairpin through their respective public, advertised addresses, which is more brittle in terms of network conditions the cluster can survive, but may still be preferable for the simplicity of a single, canonical ctrl address per node.

Assuming a single ctrl address is expected, we should prevent registering a node in the cluster with an alternative address, forcing the node to join with a canonical address matching its advertised host for the ctrl plane.

On ctrl1:

❯ ziti agent cluster list --tcp-addr=127.0.0.1:10001
╭───────┬─────────────────────────────────────────┬───────┬────────┬─────────┬───────────╮
│ ID    │ ADDRESS                                 │ VOTER │ LEADER │ VERSION │ CONNECTED │
├───────┼─────────────────────────────────────────┼───────┼────────┼─────────┼───────────┤
│ ctrl1 │ tls:ctrl1.ziti.172.19.0.3.sslip.io:1280 │ true  │ true   │ v1.6.7  │ true      │
│ ctrl2 │ tls:ctrl2.ziti.172.19.0.3.sslip.io:1281 │ true  │ false  │ v1.6.7  │ true      │
╰───────┴─────────────────────────────────────────┴───────┴────────┴─────────┴───────────╯

On ctrl2, the inconsistency is apparent:

❯ ziti agent cluster list --tcp-addr=127.0.0.1:10001
╭───────┬─────────────────────────────────────────┬───────┬────────┬─────────────────┬───────────╮
│ ID    │ ADDRESS                                 │ VOTER │ LEADER │ VERSION         │ CONNECTED │
├───────┼─────────────────────────────────────────┼───────┼────────┼─────────────────┼───────────┤
│ ctrl1 │ tls:ctrl1.ziti.172.19.0.3.sslip.io:1280 │ true  │ true   │ <not connected> │ false     │
│ ctrl2 │ tls:ctrl2.ziti.172.19.0.3.sslip.io:1281 │ true  │ false  │ v1.6.7          │ true      │
│ ctrl1 │ tls:ziti-ctrl1-controller-ctrl:1280     │ false │ false  │ v1.6.7          │ true      │
╰───────┴─────────────────────────────────────────┴───────┴────────┴─────────────────┴───────────╯

For now, I'll work around this by always hairpinning k8s-cluster-internal traffic through the external address. This seems to be Ziti's internal assumption, but that is more brittle than I'd prefer.