Multi-node setup with Hazelcast and Artemis in container not working

[b-fein]

I have a question regarding Hazelcast in a multi-instance setup: How can I debug if the Hazelcast nodes do not connect to each other?

Artemis setup

I want to try out the LocalCI setup with the build-agent running in a separate VM. We deploy our Artemis as a container instead of directly via the WAR file. Since Artemis is running in a container, it cannot bind directly to the public address of the machine, but Hazelcast allows for different bind-interfaces and public addresses with the addition of a configuration parameter in Artemis [1].

[1] develop...feature/general/hazelcast-public-address, built container image ghcr.io/uni-passau-artemis/artemis:8.2.2-hazelcast

Artemis main instance:

• running in Artemis VM
• hazelcast.interface: 0.0.0.0
• hazelcast.publicAddress: 172.16.1.10:5701

Artemis build agent:

• running in Build Agent VM
• hazelcast.interface: 0.0.0.0
• hazelcast.publicAddress: 172.16.1.20:5701

In the log and in the Eureka web interface I can now see that both instances correcly connect to the registry:

Current Registry members: [172.16.1.10, 172.16.1.20]

However, for example in the log of artemis-container I then get

Current Hazelcast members: [172.16.1.10]
Adding Hazelcast cluster member 172.16.1.20:5701

and every two minutes the same (on the build agent the same with swapped IPs). So it looks like the Hazelcast cluster does not form, but there is no error log explaining what might go wrong. It looks like it correctly uses the public addresses as defined in the config to try to connect.

Using nmap I verified that the port 5701 is open on both machines when checking from the respective other one (i.e. there should not be a firewall that blocks the communication). I also get an error log in Artemis that the Hazelcast REST API is not enabled when I try to curl http://172.16.0.{10,20}:5701/hazelcast/rest/, so it seems the public address/port is successfully routed to Artemis itself and Artemis listens on it.

Artemis config

spring:
  hazelcast:
    interface: 0.0.0.0  # use all interfaces inside the container (10.89.0.*)
    publicAddress: 172.16.1.10:5701  # or 172.16.1.20 on the other host

eureka:
  client:
    enabled: true
    service-url:
      defaultZone: https://admin:${jhipster.registry.password}@172.16.1.10:10443/eureka
  instance:
    prefer-ip-address: true
    ip-address: "172.16.1.10"  # or 172.16.1.20 on the other host
    appname: Artemis
    instanceId: Artemis:artemis-test  # or Artemis:artemis-buildagent on the other host

---

Sanity-check using plain Hazelcast container

To check if this is an issue with Hazelcast itself, or the way it is used in Artemis, I used the plain Hazelcast containers on both hosts to check if they can connect using the public/private IP mapping.

hazelcast.yml identical on both hosts

hazelcast:
  network:
    join:
      multicast:
        enabled: false
      tcp-ip:
        enabled: true
        member-list:
          - 172.16.1.10:5702
          - 172.16.1.20:5702

Then I start the containers on the two hosts:

# on host 1 with IP 172.16.1.10
podman run --rm --name member1 -e HZ_NETWORK_PUBLICADDRESS=172.16.1.10:5702 -p 5702:5701 -v "$(pwd)"/hazelcast.yml:/opt/hazelcast/hazelcast.yml -e HAZELCAST_CONFIG=hazelcast.yml docker.io/hazelcast/hazelcast:5

# on host 2 with IP 172.16.1.20
podman run --rm --name member2 -e HZ_NETWORK_PUBLICADDRESS=172.16.1.20:5702 -p 5702:5701 -v "$(pwd)"/hazelcast.yml:/opt/hazelcast/hazelcast.yml -e HAZELCAST_CONFIG=hazelcast.yml docker.io/hazelcast/hazelcast:5

In the log I can see that the two instances successfully connect:

Members {size:2, ver:2} [
        Member [172.16.1.10]:5702 - 0ed987bc-6342-4b63-acbc-2e74f80b9440
        Member [172.16.1.20]:5702 - da27ec4b-39ee-4561-b18f-2428dc1cd623 this
]

---

Network traffic

Using the Hazelcast containers, I can see network traffic to the destination port 5702 on the other host as expected.

When I tcpdump the traffic to 5701 to check for potential connections Artemis (tries) to open, there is nothing.
Also using strace -f -t -e network -p $ARTEMIS_PID, I can see nothing at the time when Artemis logs Adding Hazelcast cluster member.

$\Rightarrow$ I’m not even sure any more if Artemis even tries to connect to the other node.

[b-fein]

/cc @theblobinthesky

Maybe you have an idea where to start debugging, @Hialus or @bensofficial?

[Hialus]

In our own setup we have a config like this.
Note that we use IPv6 like this:

• a:x is an Artemis node with numerical id x
• b:2 is the registry

Core Node Config:

spring:
  hazelcast:
    interface: "[fcfe:0:0:0:0:0:a:1]"
    
eureka:
  client:
    enabled: true
    service-url:
      defaultZone: http://admin:${jhipster.registry.password}@[fcfe:0:0:0:0:0:b:2]:8761/eureka/
  instance:
    prefer-ip-address: true
    ip-address: "[fcfe:0:0:0:0:0:a:1]"
    appname: Artemis
    instanceId: Artemis:1

So the main difference is the hazelcast.interface value. You could try changing this and then test again.

[bensofficial]

Yes, exactly. We use the IPv6 Wireguard addresses. This shouldn't make much difference.
Something that you might check out is the JHipster Registry web interface. It should be exposed on port 8761 and provide insights about all the connection nodes' health status. Could this offer more insights?

[b-fein]

Both core node (artemis-test) and build agent (artemis.se2) show up in the registry:

[bensofficial]

@Mtze, @Hialus, and I discussed the problem in a meeting yesterday evening. We were not entirely sure what the problem might be, but we had a few ideas:

• The instance ID shown in the registry is artemis-test_fim_uni-passau_de and not artemis-test (eureka.instance.instanceId in the Artemis configuration). This might be the problem of the two nodes being unable to connect because of this mismatch.
• You tried to execute curl http://172.16.0.{10,20}:5701/hazelcast/rest/ to reach the other Artemis node. Have you tried to execute this within the Artemis Docker container? This would ensure that it is not a network issue.

[b-fein]

Thanks for looking into it!

• The instance IDs in the screenshots match what is actually configured in the application.yml. (looks like I just shortened it in the original post since the full domain was not relevant information)

• curl … Have you tried to execute this within the Artemis Docker container?

  Tried again to make sure: curl in both directions to the respective other container works and I get a log in Artemis that the REST API is not enabled.

[bensofficial]

From my understanding, the network between the nodes itself is fine. In contrast to our setup, Artemis runs within containers (which makes binding the IP addresses, etc., more complex). Have you tried the network mode host? I know this is not optimal for a production setup, but right now, I don't know what else should be wrong, other than that the nodes cannot establish a connection.

[b-fein]

Thanks everyone for looking into the issue.

Since the planned alternative setup with Redis (#11449) was a lot easier to configure, I do not think we will consider Hazelcast any further.