Cluster tend failed: java.lang.ArrayIndexOutOfBoundsException

[vanjadardic]

Hey guys, we have been getting ArrayIndexOutOfBoundsException exceptions in tend when upgraded client from v8.1.3 to v8.1.4. This is happening when some node gets a Connection failed error. We have a cluster of 8 nodes deployed in US west coast and the client in located in US east coast. Client configuration is using all default values (including timeout 1000ms and login timeout 5000ms), only configuration we have is minConnsPerNode=4.

First we have a log when using client v8.1.3 with no exception at the end:

2025-01-29 13:22:43.500 | INFO  | main | com.test.AerospikeDataStore | Initializing aerospike data store...
2025-01-29 13:22:43.995 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node A10 192.168.34.95 3000
2025-01-29 13:22:49.733 | DEBUG | main | com.test.AerospikeDataStore | Failed to create connection: Error -8: Connection failed
2025-01-29 13:22:50.998 | DEBUG | main | com.test.AerospikeDataStore | Failed to create connection: Error -8: Connection failed
2025-01-29 13:22:50.998 | INFO  | main | com.test.AerospikeDataStore | Add node B20 192.168.34.98 3000
2025-01-29 13:22:50.998 | INFO  | main | com.test.AerospikeDataStore | Add node A10 192.168.34.95 3000
2025-01-29 13:22:50.998 | INFO  | main | com.test.AerospikeDataStore | Add node B10 192.168.34.97 3000
2025-01-29 13:22:50.998 | INFO  | main | com.test.AerospikeDataStore | Add node A21 192.168.35.96 4000
2025-01-29 13:22:50.999 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node B20 192.168.34.98 3000
2025-01-29 13:22:50.999 | INFO  | main | com.test.AerospikeDataStore | Add node A20 192.168.34.96 3000
2025-01-29 13:22:50.999 | INFO  | main | com.test.AerospikeDataStore | Add node B11 192.168.35.97 4000
2025-01-29 13:22:50.999 | INFO  | main | com.test.AerospikeDataStore | Add node B21 192.168.35.98 4000
2025-01-29 13:22:50.999 | INFO  | main | com.test.AerospikeDataStore | Add node A11 192.168.35.95 4000
2025-01-29 13:22:51.065 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node A21 192.168.35.96 4000
2025-01-29 13:22:51.132 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node B10 192.168.34.97 3000
2025-01-29 13:22:51.198 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node A11 192.168.35.95 4000
2025-01-29 13:22:51.264 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node B21 192.168.35.98 4000
2025-01-29 13:22:51.330 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node B11 192.168.35.97 4000
2025-01-29 13:22:51.396 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node A20 192.168.34.96 3000
2025-01-29 13:22:51.462 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node A10 192.168.34.95 3000
2025-01-29 13:22:51.538 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node B20 192.168.34.98 3000
2025-01-29 13:22:51.604 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node A21 192.168.35.96 4000
2025-01-29 13:22:51.670 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node B10 192.168.34.97 3000
2025-01-29 13:22:51.736 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node A11 192.168.35.95 4000
2025-01-29 13:22:51.802 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node B21 192.168.35.98 4000
2025-01-29 13:22:51.868 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node B11 192.168.35.97 4000
2025-01-29 13:22:51.934 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node A20 192.168.34.96 3000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed la1-as2m1.lan.us 4000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.35.97 4000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.35.98 4000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.35.95 4000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.34.97 3000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed la1-as2m0.lan.us 3000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed la1-as1m0.lan.us 3000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.34.95 3000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed la2-as2m1.lan.us 4000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed la2-as2m0.lan.us 3000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed la2-as1m1.lan.us 4000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed la1-as1m1.lan.us 4000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.34.98 3000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed la2-as1m0.lan.us 3000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.35.96 4000
2025-01-29 13:22:52.000 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.34.96 3000
.
.
.
2025-01-29 13:23:02.282 | WARN  | tend | com.test.AerospikeDataStore | Node A10 192.168.34.95 3000 refresh failed: Error -8: Connection failed
2025-01-29 13:23:02.740 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B20 192.168.34.98 3000
2025-01-29 13:23:02.806 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A21 192.168.35.96 4000
2025-01-29 13:23:02.872 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B10 192.168.34.97 3000
2025-01-29 13:23:02.937 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A11 192.168.35.95 4000
2025-01-29 13:23:03.003 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B21 192.168.35.98 4000
2025-01-29 13:23:03.069 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B11 192.168.35.97 4000
2025-01-29 13:23:03.135 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A20 192.168.34.96 3000
2025-01-29 13:23:04.856 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A10 192.168.34.95 3000
2025-01-29 13:23:04.922 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B20 192.168.34.98 3000
2025-01-29 13:23:04.987 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A21 192.168.35.96 4000
2025-01-29 13:23:05.053 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B10 192.168.34.97 3000
2025-01-29 13:23:05.118 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A11 192.168.35.95 4000
2025-01-29 13:23:05.184 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B21 192.168.35.98 4000
2025-01-29 13:23:05.250 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B11 192.168.35.97 4000
2025-01-29 13:23:05.315 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A20 192.168.34.96 3000
2025-01-29 13:23:05.381 | DEBUG | tend | com.test.AerospikeDataStore | Update partition map for node A10 192.168.34.95 3000
2025-01-29 13:23:38.914 | DEBUG | tend | com.test.AerospikeDataStore | Failed to create connection: Error -8: Connection failed

The next log is from client v8.1.4 with exception at the end:

2025-01-29 13:12:00.351 | INFO  | main | com.test.AerospikeDataStore | Initializing aerospike data store...
2025-01-29 13:12:00.772 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node A10 192.168.34.95 3000
2025-01-29 13:12:05.078 | DEBUG | main | com.test.AerospikeDataStore | Failed to create connection: Error -8: Connection failed
2025-01-29 13:12:07.052 | INFO  | main | com.test.AerospikeDataStore | Add node A10 192.168.34.95 3000
2025-01-29 13:12:07.053 | INFO  | main | com.test.AerospikeDataStore | Add node A21 192.168.35.96 4000
2025-01-29 13:12:07.053 | INFO  | main | com.test.AerospikeDataStore | Add node B20 192.168.34.98 3000
2025-01-29 13:12:07.054 | INFO  | main | com.test.AerospikeDataStore | Add node B10 192.168.34.97 3000
2025-01-29 13:12:07.054 | INFO  | main | com.test.AerospikeDataStore | Add node B21 192.168.35.98 4000
2025-01-29 13:12:07.054 | INFO  | main | com.test.AerospikeDataStore | Add node B11 192.168.35.97 4000
2025-01-29 13:12:07.054 | INFO  | main | com.test.AerospikeDataStore | Add node A11 192.168.35.95 4000
2025-01-29 13:12:07.055 | INFO  | main | com.test.AerospikeDataStore | Add node A20 192.168.34.96 3000
2025-01-29 13:12:07.055 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node B20 192.168.34.98 3000
2025-01-29 13:12:07.120 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node A21 192.168.35.96 4000
2025-01-29 13:12:07.186 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node B10 192.168.34.97 3000
2025-01-29 13:12:07.256 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node A11 192.168.35.95 4000
2025-01-29 13:12:07.322 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node B21 192.168.35.98 4000
2025-01-29 13:12:07.388 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node B11 192.168.35.97 4000
2025-01-29 13:12:07.453 | DEBUG | main | com.test.AerospikeDataStore | Update peers for node A20 192.168.34.96 3000
2025-01-29 13:12:07.519 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node A10 192.168.34.95 3000
2025-01-29 13:12:07.607 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node B20 192.168.34.98 3000
2025-01-29 13:12:07.673 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node A21 192.168.35.96 4000
2025-01-29 13:12:07.740 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node B10 192.168.34.97 3000
2025-01-29 13:12:07.806 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node A11 192.168.35.95 4000
2025-01-29 13:12:07.872 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node B21 192.168.35.98 4000
2025-01-29 13:12:07.938 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node B11 192.168.35.97 4000
2025-01-29 13:12:08.003 | DEBUG | main | com.test.AerospikeDataStore | Update partition map for node A20 192.168.34.96 3000
2025-01-29 13:12:08.069 | DEBUG | main | com.test.AerospikeDataStore | Add seed la2-as1m0.lan.us 3000
2025-01-29 13:12:08.069 | DEBUG | main | com.test.AerospikeDataStore | Add seed la1-as2m1.lan.us 4000
2025-01-29 13:12:08.069 | DEBUG | main | com.test.AerospikeDataStore | Add seed la1-as2m0.lan.us 3000
2025-01-29 13:12:08.069 | DEBUG | main | com.test.AerospikeDataStore | Add seed la1-as1m1.lan.us 4000
2025-01-29 13:12:08.069 | DEBUG | main | com.test.AerospikeDataStore | Add seed la1-as1m0.lan.us 3000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.35.98 4000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.35.97 4000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.34.96 3000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed la2-as2m0.lan.us 3000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.34.97 3000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed la2-as1m1.lan.us 4000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed la2-as2m1.lan.us 4000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.34.95 3000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.34.98 3000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.35.95 4000
2025-01-29 13:12:08.070 | DEBUG | main | com.test.AerospikeDataStore | Add seed 192.168.35.96 4000
.
.
.
2025-01-29 13:12:15.458 | WARN  | tend | com.test.AerospikeDataStore | Node B10 192.168.34.97 3000 refresh failed: Error -8: Connection failed
2025-01-29 13:12:16.459 | WARN  | tend | com.test.AerospikeDataStore | Node A11 192.168.35.95 4000 refresh failed: Error -8: Connection failed
2025-01-29 13:12:16.655 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A10 192.168.34.95 3000
2025-01-29 13:12:18.768 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B20 192.168.34.98 3000
2025-01-29 13:12:18.768 | DEBUG | tend | com.test.AerospikeDataStore | Failed to create connection: Error -8: Connection failed
2025-01-29 13:12:18.834 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A21 192.168.35.96 4000
2025-01-29 13:12:18.900 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B21 192.168.35.98 4000
2025-01-29 13:12:18.967 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B11 192.168.35.97 4000
2025-01-29 13:12:19.033 | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A20 192.168.34.96 3000
2025-01-29 13:12:19.099 | INFO  | tend | com.test.AerospikeDataStore | Remove node B10 192.168.34.97 3000
2025-01-29 13:12:19.100 | INFO  | tend | com.test.AerospikeDataStore | Remove node A11 192.168.35.95 4000
2025-01-29 13:12:19.101 | WARN  | tend | com.test.AerospikeDataStore | Cluster tend failed: java.lang.ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 4
at com.aerospike.client.cluster.Cluster.removeNodesCopy(Cluster.java:920)
at com.aerospike.client.cluster.Cluster.removeNodes(Cluster.java:898)
at com.aerospike.client.cluster.Cluster.tend(Cluster.java:564)
at com.aerospike.client.cluster.Cluster.run(Cluster.java:508)
at java.base/java.lang.Thread.run(Unknown Source)

This is later causing exceptions like this when querying

com.aerospike.client.AerospikeException$InvalidNode: Error -3,1,0,30000,1000,2: Node not found for partition test:268

The same exception is also present in client v9.0.2.

[BrianNichols]

ArrayIndexOutOfBoundsException will occur if the same node is added twice to the "removeList". In this case, B10 and A11 were probably both added twice causing the new node count be incorrectly calculated as 4 (8 nodes - 4 nodes in removeList) instead of the correct 6 (8 - 2).

I can fix this, but I would first like to determine exactly how that happened. If I created a debug build branch off 8.1.4, would it be possible for you to compile and run this debug build with your application?

The debug build would be 8.1.4 plus additional log messages.

[vanjadardic]

We would be happy to do so. Let us know the branch name when you prepare it.

[BrianNichols]

The "CLIENT-3308" branch is ready.

[vanjadardic]

Looks like the issue might be related to the fact that nodes are being used as both hostname and IP address. I will investigate further on my side to see why that might be happening.

| Thu, 30 Jan 2025 08:59:17.317 UTC | WARN  | tend | com.test.AerospikeDataStore | Node B21 192.168.35.98 4000 refresh failed: Error -8: Connection failed
| Thu, 30 Jan 2025 08:59:17.448 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A10 192.168.34.95 3000
| Thu, 30 Jan 2025 08:59:17.514 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: B21 192.168.35.98 4000
| Thu, 30 Jan 2025 08:59:17.514 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la2-as2m1.lan.di4.us 4000' to node host '192.168.35.98 4000'
| Thu, 30 Jan 2025 08:59:18.235 UTC | INFO  | tend | com.test.AerospikeDataStore | Replace node: B21 192.168.35.98 4000
| Thu, 30 Jan 2025 08:59:18.235 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B20 192.168.34.98 3000
| Thu, 30 Jan 2025 08:59:18.300 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: B21 192.168.35.98 4000
| Thu, 30 Jan 2025 08:59:18.300 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la2-as2m1.lan.di4.us 4000' to node host '192.168.35.98 4000'
| Thu, 30 Jan 2025 08:59:18.300 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A21 192.168.35.96 4000
| Thu, 30 Jan 2025 08:59:18.366 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: B21 192.168.35.98 4000
| Thu, 30 Jan 2025 08:59:18.366 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la2-as2m1.lan.di4.us 4000' to node host '192.168.35.98 4000'
| Thu, 30 Jan 2025 08:59:18.366 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B10 192.168.34.97 3000
| Thu, 30 Jan 2025 08:59:18.432 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: B21 192.168.35.98 4000
| Thu, 30 Jan 2025 08:59:18.432 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la2-as2m1.lan.di4.us 4000' to node host '192.168.35.98 4000'
| Thu, 30 Jan 2025 08:59:18.432 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A11 192.168.35.95 4000
| Thu, 30 Jan 2025 08:59:18.498 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: B21 192.168.35.98 4000
| Thu, 30 Jan 2025 08:59:18.498 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la2-as2m1.lan.di4.us 4000' to node host '192.168.35.98 4000'
| Thu, 30 Jan 2025 08:59:18.498 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B11 192.168.35.97 4000
| Thu, 30 Jan 2025 08:59:18.563 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: B21 192.168.35.98 4000
| Thu, 30 Jan 2025 08:59:18.563 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la2-as2m1.lan.di4.us 4000' to node host '192.168.35.98 4000'
| Thu, 30 Jan 2025 08:59:18.563 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A20 192.168.34.96 3000
| Thu, 30 Jan 2025 08:59:18.629 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: B21 192.168.35.98 4000
| Thu, 30 Jan 2025 08:59:18.629 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la2-as2m1.lan.di4.us 4000' to node host '192.168.35.98 4000'
| Thu, 30 Jan 2025 08:59:18.629 UTC | INFO  | tend | com.test.AerospikeDataStore | Remove non-responding node: B21 192.168.35.98 4000
| Thu, 30 Jan 2025 08:59:18.629 UTC | INFO  | tend | com.test.AerospikeDataStore | Remove nodes. nodes.length: 8 nodesToRemove: 2
| Thu, 30 Jan 2025 08:59:18.629 UTC | INFO  | tend | com.test.AerospikeDataStore | Retain node: A10 192.168.34.95 3000
| Thu, 30 Jan 2025 08:59:18.629 UTC | INFO  | tend | com.test.AerospikeDataStore | Retain node: B20 192.168.34.98 3000
| Thu, 30 Jan 2025 08:59:18.629 UTC | INFO  | tend | com.test.AerospikeDataStore | Retain node: A21 192.168.35.96 4000
| Thu, 30 Jan 2025 08:59:18.629 UTC | INFO  | tend | com.test.AerospikeDataStore | Retain node: B10 192.168.34.97 3000
| Thu, 30 Jan 2025 08:59:18.630 UTC | INFO  | tend | com.test.AerospikeDataStore | Retain node: A11 192.168.35.95 4000
| Thu, 30 Jan 2025 08:59:18.630 UTC | INFO  | tend | com.test.AerospikeDataStore | Remove node B21 192.168.35.98 4000
| Thu, 30 Jan 2025 08:59:18.630 UTC | INFO  | tend | com.test.AerospikeDataStore | Retain node: B11 192.168.35.97 4000
| Thu, 30 Jan 2025 08:59:18.630 UTC | INFO  | tend | com.test.AerospikeDataStore | Retain node: A20 192.168.34.96 3000
| Thu, 30 Jan 2025 08:59:18.630 UTC | WARN  | tend | com.test.AerospikeDataStore | Cluster tend failed: java.lang.ArrayIndexOutOfBoundsException: Index 6 out of bounds for length 6
	at com.aerospike.client.cluster.Cluster.removeNodesCopy(Cluster.java:926)
	at com.aerospike.client.cluster.Cluster.removeNodes(Cluster.java:902)
	at com.aerospike.client.cluster.Cluster.tend(Cluster.java:564)
	at com.aerospike.client.cluster.Cluster.run(Cluster.java:508)
	at java.base/java.lang.Thread.run(Unknown Source)

[BrianNichols]

Yes, this will be fixed in 9.0.4 and a 8.1.5 backport release.

[BrianNichols]

The fix was more complicated than I thought, so I checked-in the fix to the same "CLIENT-3308" branch. Can you try your application with the updated branch?

Also, I'm curious how the server returned a hostname instead of an IP address. What are your server config values for "address", "access-address" and "alternate-access-address" for the B21 node?

[vanjadardic]

I have deployed the service with the patched version of aerospike client but "unfortunately" we didn't have any failed connections yet. We will leave it running over the weekend.

For the server config we just have "address", it's set to the same hostname we use as one of the seeds in client, e.g. la2-as2m1.lan.di4.us. I don't know if this makes any difference, but please note that we run two aerospike nodes on the same server machine (we have 4 servers, so 8 aerospike nodes) that has two network interface cards, so each aerospike node gets their own IP address (and hostname), one is running on port 3000 and the other on port 4000.

[BrianNichols]

I have confirmed that it's expected for the hostname to be returned for peer nodes when a node "address" is also a hostname. Your configuration is fine. The client needs to handle this case.

[vanjadardic]

Here are the newest logs:

| Sun, 02 Feb 2025 05:34:35.382 UTC | WARN  | tend | com.test.AerospikeDataStore | Node A11 192.168.35.95 4000 refresh failed: Error -8: Connection failed
| Sun, 02 Feb 2025 05:34:35.849 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A10 192.168.34.95 3000
| Sun, 02 Feb 2025 05:34:35.914 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A11 192.168.35.95 4000
| Sun, 02 Feb 2025 05:34:35.915 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m1.lan.di4.us 4000' to node host '192.168.35.95 4000'
| Sun, 02 Feb 2025 05:34:35.918 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B20 192.168.34.98 3000
| Sun, 02 Feb 2025 05:34:35.983 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A11 192.168.35.95 4000
| Sun, 02 Feb 2025 05:34:35.983 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m1.lan.di4.us 4000' to node host '192.168.35.95 4000'
| Sun, 02 Feb 2025 05:34:35.983 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A21 192.168.35.96 4000
| Sun, 02 Feb 2025 05:34:36.049 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A11 192.168.35.95 4000
| Sun, 02 Feb 2025 05:34:36.049 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m1.lan.di4.us 4000' to node host '192.168.35.95 4000'
| Sun, 02 Feb 2025 05:34:36.049 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B10 192.168.34.97 3000
| Sun, 02 Feb 2025 05:34:36.115 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A11 192.168.35.95 4000
| Sun, 02 Feb 2025 05:34:36.115 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m1.lan.di4.us 4000' to node host '192.168.35.95 4000'
| Sun, 02 Feb 2025 05:34:36.115 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B21 192.168.35.98 4000
| Sun, 02 Feb 2025 05:34:36.180 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A11 192.168.35.95 4000
| Sun, 02 Feb 2025 05:34:36.180 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m1.lan.di4.us 4000' to node host '192.168.35.95 4000'
| Sun, 02 Feb 2025 05:34:36.181 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B11 192.168.35.97 4000
| Sun, 02 Feb 2025 05:34:36.245 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A11 192.168.35.95 4000
| Sun, 02 Feb 2025 05:34:36.247 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m1.lan.di4.us 4000' to node host '192.168.35.95 4000'
| Sun, 02 Feb 2025 05:34:36.247 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A20 192.168.34.96 3000
| Sun, 02 Feb 2025 05:34:36.312 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A11 192.168.35.95 4000
| Sun, 02 Feb 2025 05:34:36.312 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m1.lan.di4.us 4000' to node host '192.168.35.95 4000'
| Sun, 02 Feb 2025 05:34:37.960 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A10 192.168.34.95 3000
| Sun, 02 Feb 2025 05:34:38.025 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B20 192.168.34.98 3000
| Sun, 02 Feb 2025 05:34:38.090 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A21 192.168.35.96 4000
| Sun, 02 Feb 2025 05:34:38.155 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B10 192.168.34.97 3000
| Sun, 02 Feb 2025 05:34:38.220 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A11 192.168.35.95 4000
| Sun, 02 Feb 2025 05:34:38.285 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B21 192.168.35.98 4000
| Sun, 02 Feb 2025 05:34:38.350 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B11 192.168.35.97 4000
| Sun, 02 Feb 2025 05:34:38.415 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A20 192.168.34.96 3000
| Sun, 02 Feb 2025 05:34:38.480 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update partition map for node A11 192.168.35.95 4000
.
.
.
| Sun, 02 Feb 2025 05:52:20.796 UTC | WARN  | tend | com.test.AerospikeDataStore | Node A10 192.168.34.95 3000 refresh failed: Error -8: Connection failed
| Sun, 02 Feb 2025 05:52:21.253 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B20 192.168.34.98 3000
| Sun, 02 Feb 2025 05:52:21.318 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A10 192.168.34.95 3000
| Sun, 02 Feb 2025 05:52:21.319 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m0.lan.di4.us 3000' to node host '192.168.34.95 3000'
| Sun, 02 Feb 2025 05:52:21.321 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A21 192.168.35.96 4000
| Sun, 02 Feb 2025 05:52:21.386 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A10 192.168.34.95 3000
| Sun, 02 Feb 2025 05:52:21.386 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m0.lan.di4.us 3000' to node host '192.168.34.95 3000'
| Sun, 02 Feb 2025 05:52:21.387 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B10 192.168.34.97 3000
| Sun, 02 Feb 2025 05:52:21.452 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A10 192.168.34.95 3000
| Sun, 02 Feb 2025 05:52:21.452 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m0.lan.di4.us 3000' to node host '192.168.34.95 3000'
| Sun, 02 Feb 2025 05:52:21.452 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A11 192.168.35.95 4000
| Sun, 02 Feb 2025 05:52:21.518 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A10 192.168.34.95 3000
| Sun, 02 Feb 2025 05:52:21.518 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m0.lan.di4.us 3000' to node host '192.168.34.95 3000'
| Sun, 02 Feb 2025 05:52:21.518 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B21 192.168.35.98 4000
| Sun, 02 Feb 2025 05:52:21.583 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A10 192.168.34.95 3000
| Sun, 02 Feb 2025 05:52:21.583 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m0.lan.di4.us 3000' to node host '192.168.34.95 3000'
| Sun, 02 Feb 2025 05:52:21.583 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B11 192.168.35.97 4000
| Sun, 02 Feb 2025 05:52:21.649 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A10 192.168.34.95 3000
| Sun, 02 Feb 2025 05:52:21.649 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m0.lan.di4.us 3000' to node host '192.168.34.95 3000'
| Sun, 02 Feb 2025 05:52:21.649 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A20 192.168.34.96 3000
| Sun, 02 Feb 2025 05:52:21.714 UTC | INFO  | tend | com.test.AerospikeDataStore | Peer node has failures: A10 192.168.34.95 3000
| Sun, 02 Feb 2025 05:52:21.714 UTC | INFO  | tend | com.test.AerospikeDataStore | Compare peer host 'la1-as1m0.lan.di4.us 3000' to node host '192.168.34.95 3000'
| Sun, 02 Feb 2025 05:52:23.369 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A10 192.168.34.95 3000
| Sun, 02 Feb 2025 05:52:23.435 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B20 192.168.34.98 3000
| Sun, 02 Feb 2025 05:52:23.501 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A21 192.168.35.96 4000
| Sun, 02 Feb 2025 05:52:23.566 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B10 192.168.34.97 3000
| Sun, 02 Feb 2025 05:52:23.632 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A11 192.168.35.95 4000
| Sun, 02 Feb 2025 05:52:23.698 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B21 192.168.35.98 4000
| Sun, 02 Feb 2025 05:52:23.764 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node B11 192.168.35.97 4000
| Sun, 02 Feb 2025 05:52:23.829 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update peers for node A20 192.168.34.96 3000
| Sun, 02 Feb 2025 05:52:23.895 UTC | DEBUG | tend | com.test.AerospikeDataStore | Update partition map for node A10 192.168.34.95 3000

Looks like using Set instead of List helped.

[BrianNichols]

Yes, the exception is not generated and the "Replace node" message no longer occurs (la1-as1m0.lan.di4.us was converted to an IP address that matched). The fix will be in the 9.0.4 and 8.1.5 client releases.

[BrianNichols]

Java clients 8.1.5 and 9.0.4 have been released with the fix:

https://download.aerospike.com/download/client/java/notes.html#8.1.5
https://download.aerospike.com/download/client/java/notes.html#9.0.4