Connecting to a machine with multiple nodes fails.

[paulchandler]

When a cluster is created with multiple nodes on one machine ecchronos cannot connect to all the nodes

The nodes are set up so they use different loopback interfaces 127.0.0.1, 127.0.0.2, 127.0.0.3

However the JMX port is always bound to 127.0.0.1 so the JMX port is changed to 7199,7198,7197

The ports and ip addresses can be seen here:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.2:9042          0.0.0.0:*               LISTEN      9560/java
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.3:9042          0.0.0.0:*               LISTEN      575664/java
tcp        0      0 127.0.0.1:9042          0.0.0.0:*               LISTEN      3701/java
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:33461         0.0.0.0:*               LISTEN      575664/java
tcp        0      0 127.0.0.1:63342         0.0.0.0:*               LISTEN      5802/ld-linux-x86-6
tcp        0      0 127.0.0.1:5990          0.0.0.0:*               LISTEN      5802/ld-linux-x86-6
tcp        0      0 127.0.0.1:46481         0.0.0.0:*               LISTEN      5802/ld-linux-x86-6
tcp        0      0 127.0.0.1:45855         0.0.0.0:*               LISTEN      5802/ld-linux-x86-6
tcp        0      0 127.0.0.1:45585         0.0.0.0:*               LISTEN      9560/java
tcp        0      0 127.0.0.2:7000          0.0.0.0:*               LISTEN      9560/java
tcp        0      0 10.255.255.254:53       0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:7199          0.0.0.0:*               LISTEN      3701/java
tcp        0      0 127.0.0.1:7198          0.0.0.0:*               LISTEN      9560/java
tcp        0      0 127.0.0.1:7197          0.0.0.0:*               LISTEN      575664/java
tcp        0      0 127.0.0.3:7000          0.0.0.0:*               LISTEN      575664/java
tcp        0      0 127.0.0.1:39707         0.0.0.0:*               LISTEN      3701/java
tcp        0      0 127.0.0.1:7000          0.0.0.0:*               LISTEN      3701/java
tcp        0      0 127.0.0.1:63561         0.0.0.0:*               LISTEN      5802/ld-linux-x86-6
tcp6       0      0 :::3390                 :::*                    LISTEN      -
tcp6       0      0 127.0.0.1:42023         :::*                    LISTEN      448181/ld-linux-x86
tcp6       0      0 ::1:631                 :::*                    LISTEN      -
tcp6       0      0 127.0.0.1:36579         :::*                    LISTEN      447799/java
tcp6       0      0 ::1:3350                :::*                    LISTEN      -
tcp6       0      0 :::33921                :::*                    LISTEN      574137/java
tcp6       0      0 127.0.0.1:13208         :::*                    LISTEN      447799/java
tcp6       0      0 127.0.0.1:8080          :::*                    LISTEN      574137/java
tcp6       0      0 127.0.0.1:15149         :::*                    LISTEN      448181/ld-linux-x86

So for JMX the connection tries to bind to the other loopback address and fails.

Hard coding 127.0.0.1 in this line fixes the problem, but obviously that is not a solution

https://github.com/Ericsson/ecchronos/blob/agent/master/connection.impl/src/main/java/com/ericsson/bss/cassandra/ecchronos/connection/impl/builders/DistributedJmxBuilder.java#L202

[VictorCavichioli]

Related with #652

[VictorCavichioli]

Hi @paulchandler, after adding this to each node in cassandra-env.sh it started to work:

JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<loopback_address>"

Logs from ecchronos startup:

2025-09-18T15:11:50.296-03:00  INFO 189657 --- [           main] c.e.b.c.e.a.p.AgentJmxConnectionProvider : Creating DistributedJmxConnectionConfig
2025-09-18T15:11:50.299-03:00  INFO 189657 --- [           main] c.e.b.c.e.c.i.b.DistributedJmxBuilder    : Creating connection with node 4c6ac9b8-80d4-4705-8a02-d22d292e4586
2025-09-18T15:11:50.315-03:00  INFO 189657 --- [           main] c.e.b.c.e.c.i.b.DistributedJmxBuilder    : Starting to instantiate JMXService with host: 127.0.0.1 and port: 7199
2025-09-18T15:11:50.344-03:00  INFO 189657 --- [           main] c.e.b.c.e.c.i.b.DistributedJmxBuilder    : Connected JMX for service:jmx:rmi:///jndi/rmi://127.0.0.1:7199/jmxrmi
2025-09-18T15:11:50.360-03:00  INFO 189657 --- [           main] c.e.b.c.e.data.sync.EccNodesSync         : Node 4c6ac9b8-80d4-4705-8a02-d22d292e4586 successfully updated
2025-09-18T15:11:50.361-03:00  INFO 189657 --- [           main] c.e.b.c.e.c.i.b.DistributedJmxBuilder    : Connection created with success
2025-09-18T15:11:50.361-03:00  INFO 189657 --- [           main] c.e.b.c.e.c.i.b.DistributedJmxBuilder    : Creating connection with node 381a58a8-58ca-41f5-b88a-5195d5f6f7e1
2025-09-18T15:11:50.367-03:00  INFO 189657 --- [           main] c.e.b.c.e.c.i.b.DistributedJmxBuilder    : Starting to instantiate JMXService with host: 127.0.0.3 and port: 7399
2025-09-18T15:11:50.399-03:00  INFO 189657 --- [           main] c.e.b.c.e.c.i.b.DistributedJmxBuilder    : Connected JMX for service:jmx:rmi:///jndi/rmi://127.0.0.3:7399/jmxrmi
2025-09-18T15:11:50.404-03:00  INFO 189657 --- [           main] c.e.b.c.e.data.sync.EccNodesSync         : Node 381a58a8-58ca-41f5-b88a-5195d5f6f7e1 successfully updated
2025-09-18T15:11:50.404-03:00  INFO 189657 --- [           main] c.e.b.c.e.c.i.b.DistributedJmxBuilder    : Connection created with success

status of ecchronos.nodes_sync table:

Connected to Test Cluster at 127.0.0.1:9042
[cqlsh 6.2.0 | Cassandra 5.0.5 | CQL spec 3.4.7 | Native protocol v5]
Use HELP for help.
cqlsh> use ecchronos ;
cqlsh:ecchronos> SELECT * from nodes_sync ;

 ecchronos_id      | datacenter_name | node_id                              | last_connection                 | next_connection                 | node_endpoint   | node_status
-------------------+-----------------+--------------------------------------+---------------------------------+---------------------------------+-----------------+-------------
 unique_identifier |     datacenter1 | 4c6ac9b8-80d4-4705-8a02-d22d292e4586 | 2025-09-18 18:11:50.344000+0000 | 2025-09-18 18:41:50.344000+0000 | /127.0.0.1:9042 |   AVAILABLE
 unique_identifier |     datacenter1 | 381a58a8-58ca-41f5-b88a-5195d5f6f7e1 | 2025-09-18 18:11:50.399000+0000 | 2025-09-18 18:41:50.399000+0000 | /127.0.0.3:9242 |   AVAILABLE

(2 rows)
cqlsh:ecchronos> 

As the default config is 127.0.0.1 it would always bound to 127.0.0.1, I propose to add this info in our documentation instead of trying to handle this case in jmx connection builder, what do you think @paulchandler ?