Node selector for name node listener PVC causing issue post cluster upgrade/recycle

[rajarshp]

Hi Team,

We have faced an issue with Stackable HDFS name node, which is using a listener pvc but when there is cluster upgrade/recycle and it has all new nodes. Then listener PVC is not automatically getting created with updated node value.

So when we checked the YAML for this PVC we can below annotation and node affinity value and it is pointing to a particular node, which will be terminated during any cluster upgrade or recycle. So the name node is stuck is pending state as it is not able to use the listener PVC.

I am not sure if this a bug or already there is a fix available, but can someone help us to fix this so that in next cluster upgrade/recycle we should not face this issue ?

As a workaround we have deleted the PVC and restarted the service.

Stackable Version: 24.7
Platform: AWS Rosa HCP

POD YAML:

volumeAttributes:
      listeners.stackable.tech/listener-class: external-stable
      storage.kubernetes.io/csiProvisionerIdentity: 1730131773929-3396-listeners.stackable.tech
      volume.beta.kubernetes.io/storage-provisioner: listeners.stackable.tech
      volume.kubernetes.io/selected-node: ip-x-x-x-x.yy-xx-1.compute.internal

nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: listeners.stackable.tech/hostname
              operator: In
              values:
                - ip-x-x-x-x.yy-xx-1.compute.internal

[nightkr]

This is a red herring, I'm afraid, see https://github.com/orgs/stackabletech/discussions/71#discussioncomment-12337054.

[maltesander]

Thanks for the clarification :)

[nightkr]

I'm afraid that issue is actually unrelated. stackabletech/listener-operator#262 happens when trying to reuse the same Node across Node reboots, not when moving services between Nodes.

This is happening because you're using the (default) stable-nodes preset.

Since NameNodes' addresses are baked into clients' configuration files we need to ensure that their addresses don't change too often, which is what the PVC tracks. When using a cloud environment we can use a LoadBalancer service instead as our stable ingress point, decoupling us from any one node. To do this, set the Helm value preset=ephemeral-nodes. We don't currently support doing this with stackablectl (stackabletech/stackable-cockpit#356), so you'll need to do it with Helm instead. For example:

# if you don't already have the repo added
helm repo add stackable https://repo.stackable.tech/repository/helm-stable
helm repo update 

helm -n stackable-operators upgrade listener-operator stackable/listener-operator \
  # or whatever version you're running
  --version 24.11.1 \
  --set preset=ephemeral-nodes

After doing this, deleting the PVC again, and deleting the namenode Pod, you should see that the recreated PV no longer sets any nodeAffinity, and that the namenode Pod is free to reschedule to any Node when deleted and recreated. The datanodes will still have the nodeAffinity set, but that's not a problem since datanodes recreate their PVCs whenever they are restarted anyway.

Alternatively, you can create a new ListenerClass of your own, set its ListenerClass.spec.serviceType=LoadBalancer, and set your HdfsCluster.dataNodes.config.listenerClass to the ListenerClass you created. In that case, you'll also want to use it for anything else that defaults to the external-stable ListenerClass, such as the Kafka bootstrap listener (if you use Kafka). Due to Kubernetes limitations, after modifying this setting you'll need to delete the namenode StatefulSet after modifying the listenerClass option (in addition to deleting the PVCs).

[nightkr]

I dont think we can do it because we are installing the operator from Rosa Marketplace

Ah, fair. OpenShift does make that a bit harder...

so if we want to create our own listener class only I need to set the service type as LoadBalancer or anything else ?

That should be enough, you can see our defaults at https://github.com/stackabletech/listener-operator/blob/24.11.1/deploy/helm/listener-operator/templates/listener-classes.yaml#L17-L23.

[rajarshp]

Thanks will give it a try and see how it is going but not immediate I can confirm, maybe in next cluster upgrade event we can see how it is going.

[rajarshp]

Hi @nightkr

I applied the new changes as you suggested but it did mess up. I can see now name nodes are getting created. I am creating 2 names nodes using the helm.
I can seethe below error

MountVolume.SetUp failed for volume "pvc-cf786c61-3da6-XXXXX-XXXXXXXXX" : rpc error: code = Unavailable desc = listener has no addresses yet

I can see only one PVC is created and still has the annotation volume.kubernetes.io/selected-node:

Listener Class:

apiVersion: listeners.stackable.tech/v1alpha1
kind: ListenerClass
metadata:
  name: external-stable-test
spec:
  serviceAnnotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  serviceType: LoadBalancer

When I change the listener class again to external-stable they pods came up

[nightkr]

Unavailable desc = listener has no addresses yet

Hm, looks like ROSA might not actually ship a LoadBalancer controller by default. See https://docs.openshift.com/rosa/networking/networking_operators/aws-load-balancer-operator.html for how to install that.

After that, it is normal to see the error message for a few minutes while the LoadBalancer is provisioning, but it should eventually resolve itself. You can also look at the corresponding Service to see whether the problem seems to be on the listener-operator or LoadBalancer controller side.

I can see only one PVC is created and still has the annotation volume.kubernetes.io/selected-node:

That's normal (and managed by K8s), you'll have to look at the corresponding PersistentVolume's nodeAffinity to see whether it's still locked to a certain node.

[rajarshp]

@nightkr thanks for the response.
Rosa already have this installed and earlier we created listener class with same config which did work. But now not sure why it is not getting created and seeing that error.