Helm memcached deployment via bitnami chart with AWS ELB to headless services

[zenbones]

The title says what our setup is, but, to reiterate, we're deploying memcached in k8s with the bitnami helm chart. We're adding an elb load balancer via k8s annotation to headless services, one fronting each replica from the chart. The chart creates a stateful set, which makes sense. To get the benefit of the cluster via ketama consistent hashing, and to get failover in case we lose a replica, we put a headless service in front of each replica, and to use the memcached setup with machines outside of k8s, we front each headless service with an elb endpoint. Because all clients need to see the same hosts, because they operate on the same keys/vaues, we actually use the elb endpoints both inside and outside of k8s. This was still failing until we updated to 2.4.7 ad used the AddressMemcachedSessionComparator, to guarantee that all clients were ordering the sessions the same way. However, this alone was still failing because each of the elb endpoints responds with multiple IP addresses, and InetAddress.getByName(host)[0] will return the ip addresses in different order. So we needed to construct the address map for XMemcachedClientBuilder out of consistent ip addresses using...

public class InetAddressComparator implements Comparator {

@OverRide
public int compare (InetAddress a, InetAddress b) {

byte[] aOctets = a.getAddress();
byte[] bOctets = b.getAddress();
int len = Math.max(aOctets.length, bOctets.length);

for (int i = 0; i < len; i++) {
  byte aOctet = (i >= len - aOctets.length) ? aOctets[i - (len - aOctets.length)] : 0;
  byte bOctet = (i >= len - bOctets.length) ? bOctets[i - (len - bOctets.length)] : 0;

  if (aOctet != bOctet) return (0xff & aOctet) - (0xff & bOctet);
}

return 0;

}
}

...to first sort the ip addresses returned by InetAddress.getAllByName(host), to make sure the addresses used for each replica were consistent for each client. We used just the address comparator at first, but that failed. Then just the session comparator from 2.4.7, but that also failed. But using both worked.

I don't know if you want to document this, or add an InetAddress comparator into the code and make it available, but I thought our experience might be helpful to others.