V2: Error connecting to cluster

[bentheiii]

Hi, we're trying to connect to a production cluster, and keep getting the error:

Unable to communicate with server cluster: Failed to connect to host(s). The network connection(s) to cluster nodes may have timed out, or the cluster may be in a state of flux.

activating debug logs in the SDK shows this error:

 Unable to communicate with server cluster: Failed to connect to host(s). The network connection(s) to cluster nodes may have timed out, or the cluster may be in a state of flux.
	Invalid cluster node: Node name has changed: 'A32' => 'A11'
�DEBUG�[0m �[2maerospike_core::cluster�[0m�[2m:�[0m No connections available; seeding...
� INFO�[0m �[2maerospike_core::cluster�[0m�[2m:�[0m Seeding the cluster. Seeds count: 1
�DEBUG�[0m �[2maerospike_core::cluster::node_validator�[0m�[2m:�[0m Resolved aliases for host 10.147.0.15:3000: [Host { name: "10.147.0.15", port: 3000 }]
�DEBUG�[0m �[2maerospike_core::commands::info_command�[0m�[2m:�[0m response from server for info command: "node\tA32\ncluster-name\taerospike-cache\nfeatures\tbatch-any;batch-index;blob-bits;cdt-list;cdt-map;cluster-stable;float;geo;sindex-exists;peers;pipelining;pquery;pscans;query-show;relaxed-sc;replicas;replicas-all;replicas-master;replicas-max;truncate-namespace;udf;xdr"
�DEBUG�[0m �[2maerospike_core::commands::info_command�[0m�[2m:�[0m response from server for info command: "node\tA11\ncluster-name\taerospike-cache\npartition-generation\t1319\nservices\t10.147.0.29:3000;10.147.0.28:3000;10.147.0.38:3000;10.147.0.9:3000;10.147.0.33:3000;10.147.0.27:3000;10.147.0.35:3000;10.147.0.32:3000;10.147.0.34:3000;10.147.0.26:3000;10.147.0.37:3000;10.147.0.25:3000;10.147.0.8:3000;10.147.0.36:3000;10.147.0.91:3000;10.147.0.47:3000;10.147.0.49:3000;10.147.0.58:3000;10.147.0.52:3000;10.147.0.65:3000;10.147.0.89:3000;10.147.0.45:3000;10.147.0.61:3000;10.147.0.50:3000;10.147.0.71:3000;10.147.0.82:3000;10.147.0.60:3000;10.147.0.40:3000;10.147.0.81:3000;10.147.0.73:3000;10.147.0.83:3000;10.147.0.53:3000;10.147.0.42:3000;10.147.0.79:3000;10.147.0.41:3000;10.147.0.80:3000;10.147.0.59:3000;10.147.0.78:3000;10.147.0.51:3000;10.147.0.76:3000;10.147.0.88:3000;10.147.0.87:3000;10.147.0.74:3000;10.147.0.85:3000;10.147.0.48:3000;10.147.0.69:3000;10.147.0.70:3000;10.147.0.93:3000;10.147.0.90:3000;10.147.0.44:3000;10.147.0.63:3000;10.147.0.39:3000;10.147.0.62:3000;10.147.0.46:3000;10.147.0.67:3000;10.147.0.54:3000;10.147.0.84:3000;10.147.0.86:3000;10.147.0.64:3000;10.147.0.77:3000;10.147.0.66:3000;10.147.0.56:3000;10.147.0.75:3000;10.147.0.68:3000;10.147.0.72:3000;10.147.0.92:3000;10.147.0.55:3000;10.147.0.43:3000;10.147.0.57:3000;10.147.0.13:3000;10.147.0.12:3000;10.147.0.21:3000;10.147.0.20:3000;10.147.0.17:3000;10.147.0.24:3000;10.147.0.19:3000;10.147.0.18:3000;10.147.0.22:3000;10.147.0.23:3000"
� WARN�[0m �[2maerospike_core::cluster�[0m�[2m:�[0m Node `A32: 10.147.0.15:3000` refresh failed: Failed to validate node
� INFO�[0m �[2maerospike_core::cluster�[0m�[2m:�[0m Seeding the cluster. Seeds count: 1
�DEBUG�[0m �[2maerospike_core::cluster::node_validator�[0m�[2m:�[0m Resolved aliases for host 10.147.0.15:3000: [Host { name: "10.147.0.15", port: 3000 }]
�DEBUG�[0m �[2maerospike_core::commands::info_command�[0m�[2m:�[0m response from server for info command: "node\tA17\ncluster-name\taerospike-cache\nfeatures\tbatch-any;batch-index;blob-bits;cdt-list;cdt-map;cluster-stable;float;geo;sindex-exists;peers;pipelining;pquery;pscans;query-show;relaxed-sc;replicas;replicas-all;replicas-master;replicas-max;truncate-namespace;udf;xdr"
�DEBUG�[0m �[2maerospike_core::commands::info_command�[0m�[2m:�[0m response from server for info command: "node\tA37\ncluster-name\taerospike-cache\npartition-generation\t790\nservices\t10.147.0.29:3000;10.147.0.30:3000;10.147.0.28:3000;10.147.0.38:3000;10.147.0.9:3000;10.147.0.33:3000;10.147.0.27:3000;10.147.0.35:3000;10.147.0.32:3000;10.147.0.34:3000;10.147.0.26:3000;10.147.0.37:3000;10.147.0.25:3000;10.147.0.8:3000;10.147.0.36:3000;10.147.0.91:3000;10.147.0.47:3000;10.147.0.49:3000;10.147.0.58:3000;10.147.0.52:3000;10.147.0.65:3000;10.147.0.89:3000;10.147.0.45:3000;10.147.0.61:3000;10.147.0.50:3000;10.147.0.71:3000;10.147.0.82:3000;10.147.0.40:3000;10.147.0.81:3000;10.147.0.73:3000;10.147.0.83:3000;10.147.0.53:3000;10.147.0.42:3000;10.147.0.79:3000;10.147.0.41:3000;10.147.0.80:3000;10.147.0.59:3000;10.147.0.78:3000;10.147.0.51:3000;10.147.0.76:3000;10.147.0.88:3000;10.147.0.87:3000;10.147.0.74:3000;10.147.0.85:3000;10.147.0.48:3000;10.147.0.69:3000;10.147.0.70:3000;10.147.0.93:3000;10.147.0.90:3000;10.147.0.44:3000;10.147.0.63:3000;10.147.0.39:3000;10.147.0.62:3000;10.147.0.46:3000;10.147.0.67:3000;10.147.0.54:3000;10.147.0.84:3000;10.147.0.86:3000;10.147.0.64:3000;10.147.0.77:3000;10.147.0.66:3000;10.147.0.56:3000;10.147.0.75:3000;10.147.0.68:3000;10.147.0.72:3000;10.147.0.92:3000;10.147.0.55:3000;10.147.0.43:3000;10.147.0.57:3000;10.147.0.13:3000;10.147.0.12:3000;10.147.0.21:3000;10.147.0.20:3000;10.147.0.17:3000;10.147.0.24:3000;10.147.0.19:3000;10.147.0.18:3000;10.147.0.22:3000;10.147.0.23:3000"
� WARN�[0m �[2maerospike_core::cluster�[0m�[2m:�[0m Node `A17: 10.147.0.15:3000` refresh failed: Failed to validate node
	Invalid cluster node: Node name has changed: 'A17' => 'A37'
DEBUG�[0m �[2maerospike_core::cluster�[0m�[2m:�[0m No connections available; seeding...

After about an hour of retrying it finally does manage to connect, but we still consider it an issue.

[realmgic]

10.147.0.15 is the address of the load balancer (which was created for service discovery by the Aerospike Kubernetes Operator (AKO)). It's not a cluster node, so querying its partition table (tend) might go to different nodes on each run.

This means that this check:

aerospike-client-rust/src/cluster/node.rs

Line 167 in 41312f3

Some(info_name) if info_name == expected => Ok(()),

is problematic.

The real list of nodes is defined by the service command, which returns the access addresses or the alternate access addresses and the seed node itself shouldn't be part of the list (it is added here: https://github.com/aerospike/aerospike-client-rust/blob/41312f30e392df90dd9900e0868f93dd49a06a3e/src/cluster/mod.rs#L296C63-L296C67 - but the IP used it the LB address)

[khaf]

This seems to stem from lack of support for load balancers as seeds in the Rust client. I will address it for the next release.

[khaf]

This issue should be fixed in v2.0.0-alpha6.