[Bug] availabilityZones setting not being respected for managedNodeGroups

[mgalgs]

What were you trying to accomplish?

Trying to create a managed node group which is restricted to a single availability zone within my cluster's region.

What happened?

The resulting launch template has an empty availability zone field (screenshot below), meaning instances can be launched into any AZ in the region. Since I have zone-specific EBS volumes for some of my pods, those pods are unable to be scheduled on nodes in the managed nodegroup that don't happen to land in the correct region, resulting in a bunch of cluster-autoscaler scale-ups while the scheduler tries over and over again to find a suitable node.

I need to be able to restrict the AZ for the nodegroup so that zone-specific workloads can be scheduled reliably.

How to reproduce it?

Create a new cluster with no nodegroups, like so:

eksctl create cluster --config-file=cluster-config.yaml

cluster-config.yaml:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: yodle
  region: us-west-2
  version: "1.32"
  tags:
    cluster-name: yodle

vpc:
  id: "..."
  cidr: "..."
  subnets:
    public:
      us-west-2a:
        id: "..."
        cidr: "..."
      us-west-2b:
        id: "..."
        cidr: "..."
      us-west-2c:
        id: "..."
        cidr: "..."

And create a nodegroup from a separate config file:

eksctl create nodegroup --config-file=nodegroup-on-demand.yaml --verbose 4

nodegroup-on-demand.yaml:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: streks5
  region: us-west-2
managedNodeGroups:
- name: on-demand-4
  amiFamily: AmazonLinux2023
  instanceType: "m7i-flex.2xlarge"
  availabilityZones: ["us-west-2b"]
  ssh:
    allow: true
    publicKeyName: ...
  minSize: 1
  maxSize: 15
  updateConfig:
    maxUnavailable: 1
  iam:
    withAddonPolicies:
      autoScaler: true
  taints:
  - key: spotty
    value: "false"
    effect: PreferNoSchedule

The launch template for the resulting nodegroup does not have the expected AZ restriction:

Logs

nodegroup create logs

2025-02-11 13:56:16 [▶]  Setting credentials expiry window to 30 minutes
2025-02-11 13:56:17 [▶]  role ARN for the current session is "arn:aws:iam::*****:user/odin"
2025-02-11 13:56:17 [▶]  cluster = &types.Cluster{AccessConfig:(*types.AccessConfigResponse)(0xc001108340), Arn:(*string)(0xc0011046d0), CertificateAuthority:(*types.Certificate)(0xc001104660), ClientRequestToken:(*string)(nil), ComputeConfig:(*types.ComputeConfigResponse)(nil), ConnectorConfig:(*types.ConnectorConfigResponse)(nil), CreatedAt:time.Date(2025, time.February, 7, 22, 39, 48, 527000000, time.UTC), EncryptionConfig:[]types.EncryptionConfig(nil), Endpoint:(*string)(0xc0011046a0), Health:(*types.ClusterHealth)(0xc0011082e0), Id:(*string)(nil), Identity:(*types.Identity)(0xc001104760), KubernetesNetworkConfig:(*types.KubernetesNetworkConfigResponse)(0xc001102fc0), Logging:(*types.Logging)(0xc001108300), Name:(*string)(0xc0011046b0), OutpostConfig:(*types.OutpostConfigResponse)(nil), PlatformVersion:(*string)(0xc001104740), RemoteNetworkConfig:(*types.RemoteNetworkConfigResponse)(nil), ResourcesVpcConfig:(*types.VpcConfigResponse)(0xc00024aa80), RoleArn:(*string)(0xc001104750), Status:"ACTIVE", StorageConfig:(*types.StorageConfigResponse)(nil), Tags:map[string]string{"Name":"eksctl-yodle-cluster/ControlPlane", "alpha.eksctl.io/cluster-name":"yodle", "alpha.eksctl.io/cluster-oidc-enabled":"false", "alpha.eksctl.io/eksctl-version":"0.203.0", "aws:cloudformation:logical-id":"ControlPlane", "aws:cloudformation:stack-id":"arn:aws:cloudformation:us-west-2:******:stack/eksctl-yodle-cluster/61bbf580-****-****-****-********", "aws:cloudformation:stack-name":"eksctl-yodle-cluster", "cluster-name":"yodle", "eksctl.cluster.k8s.io/v1alpha1/cluster-name":"yodle"}, UpgradePolicy:(*types.UpgradePolicyResponse)(0xc00110c4e0), Version:(*string)(0xc0011046c0), ZonalShiftConfig:(*types.ZonalShiftConfigResponse)(nil), noSmithyDocumentSerde:document.NoSerde{}}
2025-02-11 13:56:17 [ℹ]  will use version 1.32 for new nodegroup(s) based on control plane version
2025-02-11 13:56:19 [▶]  cluster = &types.Cluster{AccessConfig:(*types.AccessConfigResponse)(0xc00119fc60), Arn:(*string)(0xc000fa9060), CertificateAuthority:(*types.Certificate)(0xc000fa9000), ClientRequestToken:(*string)(nil), ComputeConfig:(*types.ComputeConfigResponse)(nil), ConnectorConfig:(*types.ConnectorConfigResponse)(nil), CreatedAt:time.Date(2025, time.February, 7, 22, 39, 48, 527000000, time.UTC), EncryptionConfig:[]types.EncryptionConfig(nil), Endpoint:(*string)(0xc000fa9050), Health:(*types.ClusterHealth)(0xc00119fc40), Id:(*string)(nil), Identity:(*types.Identity)(0xc000fa8fd0), KubernetesNetworkConfig:(*types.KubernetesNetworkConfigResponse)(0xc00140a2d0), Logging:(*types.Logging)(0xc00119fc80), Name:(*string)(0xc000fa9020), OutpostConfig:(*types.OutpostConfigResponse)(nil), PlatformVersion:(*string)(0xc000fa9080), RemoteNetworkConfig:(*types.RemoteNetworkConfigResponse)(nil), ResourcesVpcConfig:(*types.VpcConfigResponse)(0xc000167340), RoleArn:(*string)(0xc000fa9030), Status:"ACTIVE", StorageConfig:(*types.StorageConfigResponse)(nil), Tags:map[string]string{"Name":"eksctl-yodle-cluster/ControlPlane", "alpha.eksctl.io/cluster-name":"yodle", "alpha.eksctl.io/cluster-oidc-enabled":"false", "alpha.eksctl.io/eksctl-version":"0.203.0", "aws:cloudformation:logical-id":"ControlPlane", "aws:cloudformation:stack-id":"arn:aws:cloudformation:us-west-2:******:stack/eksctl-yodle-cluster/61bbf580-****-****-****-********", "aws:cloudformation:stack-name":"eksctl-yodle-cluster", "cluster-name":"yodle", "eksctl.cluster.k8s.io/v1alpha1/cluster-name":"yodle"}, UpgradePolicy:(*types.UpgradePolicyResponse)(0xc0011ab638), Version:(*string)(0xc000fa9070), ZonalShiftConfig:(*types.ZonalShiftConfigResponse)(nil), noSmithyDocumentSerde:document.NoSerde{}}
2025-02-11 13:56:19 [ℹ]  nodegroup "on-demand-4" will use "" [AmazonLinux2023/1.32]
2025-02-11 13:56:19 [ℹ]  using EC2 key pair "*******"
2025-02-11 13:56:19 [▶]  cfg.json = \
{
    "kind": "ClusterConfig",
    "apiVersion": "eksctl.io/v1alpha5",
    "metadata": {
        "name": "yodle",
        "region": "us-west-2",
        "version": "1.32"
    },
    "kubernetesNetworkConfig": {
        "ipFamily": "IPv4",
        "serviceIPv4CIDR": "10.100.0.0/16"
    },
    "iam": {
        "withOIDC": false,
        "vpcResourceControllerPolicy": true
    },
    "accessConfig": {
        "authenticationMode": "API_AND_CONFIG_MAP"
    },
    "vpc": {
        "id": "vpc-*****",
        "cidr": "172.40.0.0/16",
        "securityGroup": "sg-*****",
        "subnets": {
            "public": {
                "us-west-2a": {
                    "id": "subnet-*****",
                    "az": "us-west-2a",
                    "cidr": "172.40.32.0/19"
                },
                "us-west-2b": {
                    "id": "subnet-*****",
                    "az": "us-west-2b",
                    "cidr": "172.40.64.0/19"
                },
                "us-west-2c": {
                    "id": "subnet-*****",
                    "az": "us-west-2c",
                    "cidr": "172.40.96.0/19"
                }
            }
        },
        "sharedNodeSecurityGroup": "sg-02f6******",
        "manageSharedNodeSecurityGroupRules": true,
        "autoAllocateIPv6": false,
        "nat": {
            "gateway": "Single"
        },
        "clusterEndpoints": {
            "privateAccess": false,
            "publicAccess": true
        }
    },
    "addonsConfig": {},
    "privateCluster": {
        "enabled": false,
        "skipEndpointCreation": false
    },
    "managedNodeGroups": [
        {
            "name": "on-demand-4",
            "amiFamily": "AmazonLinux2023",
            "instanceType": "m7i-flex.2xlarge",
            "availabilityZones": [
                "us-west-2b"
            ],
            "desiredCapacity": 1,
            "minSize": 1,
            "maxSize": 15,
            "volumeSize": 80,
            "ssh": {
                "allow": true,
                "publicKeyName": "*******"
            },
            "labels": {
                "alpha.eksctl.io/cluster-name": "yodle",
                "alpha.eksctl.io/nodegroup-name": "on-demand-4"
            },
            "privateNetworking": false,
            "tags": {
                "alpha.eksctl.io/nodegroup-name": "on-demand-4",
                "alpha.eksctl.io/nodegroup-type": "managed"
            },
            "iam": {
                "withAddonPolicies": {
                    "imageBuilder": false,
                    "autoScaler": true,
                    "externalDNS": false,
                    "certManager": false,
                    "appMesh": null,
                    "appMeshPreview": null,
                    "ebs": false,
                    "fsx": false,
                    "efs": false,
                    "awsLoadBalancerController": false,
                    "albIngress": false,
                    "xRay": false,
                    "cloudWatch": false
                }
            },
            "securityGroups": {
                "withShared": null,
                "withLocal": null
            },
            "volumeType": "gp3",
            "volumeIOPS": 3000,
            "volumeThroughput": 125,
            "disableIMDSv1": true,
            "disablePodIMDS": false,
            "instanceSelector": {},
            "taints": [
                {
                    "key": "spotty",
                    "value": "false",
                    "effect": "PreferNoSchedule"
                }
            ],
            "updateConfig": {
                "maxUnavailable": 1
            },
            "releaseVersion": ""
        }
    ],
    "availabilityZones": [
        "us-west-2b",
        "us-west-2c",
        "us-west-2a"
    ]
}

2025-02-11 13:56:20 [▶]  ClusterSharedNodeSecurityGroup = sg-02f6******
2025-02-11 13:56:21 [▶]  nodegroups = [0xc001122f00 0xc000fe7cc0]
2025-02-11 13:56:21 [▶]  nodegroup "on-demand-4" present in the given config, but missing in the cluster
2025-02-11 13:56:21 [▶]  nodegroup "on-demand-3" present in the cluster, but missing from the given config
2025-02-11 13:56:21 [▶]  nodegroup "spotty-2" present in the cluster, but missing from the given config
2025-02-11 13:56:21 [ℹ]  2 existing nodegroup(s) (on-demand-3,spotty-2) will be excluded
2025-02-11 13:56:21 [ℹ]  1 nodegroup (on-demand-4) was included (based on the include/exclude rules)
2025-02-11 13:56:21 [ℹ]  will create a CloudFormation stack for each of 1 managed nodegroups in cluster "yodle"
2025-02-11 13:56:21 [ℹ]  
2 sequential tasks: { fix cluster compatibility, 1 task: { 1 task: { create managed nodegroup "on-demand-4" } } 
}
2025-02-11 13:56:21 [▶]  started task: fix cluster compatibility
2025-02-11 13:56:21 [ℹ]  checking cluster stack for missing resources
2025-02-11 13:56:22 [ℹ]  cluster stack has all required resources
2025-02-11 13:56:22 [▶]  completed task: fix cluster compatibility
2025-02-11 13:56:22 [▶]  started task: 1 task: { 1 task: { create managed nodegroup "on-demand-4" } }
2025-02-11 13:56:22 [▶]  waiting for 1 parallel tasks to complete
2025-02-11 13:56:22 [▶]  started task: 1 task: { create managed nodegroup "on-demand-4" }
2025-02-11 13:56:22 [▶]  started task: create managed nodegroup "on-demand-4"
2025-02-11 13:56:22 [▶]  started task: create managed nodegroup "on-demand-4"
2025-02-11 13:56:23 [ℹ]  building managed nodegroup stack "eksctl-yodle-nodegroup-on-demand-4"
2025-02-11 13:56:23 [▶]  CreateStackInput = &cloudformation.CreateStackInput{StackName:(*string)(0xc0012808a0), Capabilities:[]types.Capability{"CAPABILITY_IAM"}, ClientRequestToken:(*string)(nil), DisableRollback:(*bool)(0xc00095a668), EnableTerminationProtection:(*bool)(nil), NotificationARNs:[]string(nil), OnFailure:"", Parameters:[]types.Parameter(nil), ResourceTypes:[]string(nil), RetainExceptOnCreate:(*bool)(nil), RoleARN:(*string)(nil), RollbackConfiguration:(*types.RollbackConfiguration)(nil), StackPolicyBody:(*string)(nil), StackPolicyURL:(*string)(nil), Tags:[]types.Tag{types.Tag{Key:(*string)(0xc001105ac0), Value:(*string)(0xc001105ad0), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc001105ae0), Value:(*string)(0xc001105af0), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc001105b00), Value:(*string)(0xc001105b10), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000fa8360), Value:(*string)(0xc000fa8370), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000fa8380), Value:(*string)(0xc000fa8390), noSmithyDocumentSerde:document.NoSerde{}}}, TemplateBody:(*string)(0xc000fa83a0), TemplateURL:(*string)(nil), TimeoutInMinutes:(*int32)(nil), noSmithyDocumentSerde:document.NoSerde{}}
2025-02-11 13:56:24 [ℹ]  deploying stack "eksctl-yodle-nodegroup-on-demand-4"
2025-02-11 13:56:24 [ℹ]  waiting for CloudFormation stack "eksctl-yodle-nodegroup-on-demand-4"
2025-02-11 13:56:54 [ℹ]  waiting for CloudFormation stack "eksctl-yodle-nodegroup-on-demand-4"
2025-02-11 13:57:50 [ℹ]  waiting for CloudFormation stack "eksctl-yodle-nodegroup-on-demand-4"
2025-02-11 13:58:44 [ℹ]  waiting for CloudFormation stack "eksctl-yodle-nodegroup-on-demand-4"
2025-02-11 13:58:44 [▶]  processing stack outputs
2025-02-11 13:58:44 [▶]  completed task: create managed nodegroup "on-demand-4"
2025-02-11 13:58:44 [▶]  completed task: create managed nodegroup "on-demand-4"
2025-02-11 13:58:44 [▶]  completed task: 1 task: { create managed nodegroup "on-demand-4" }
2025-02-11 13:58:44 [▶]  completed task: 1 task: { 1 task: { create managed nodegroup "on-demand-4" } }
2025-02-11 13:58:44 [ℹ]  no tasks
2025-02-11 13:58:44 [▶]  no actual tasks
2025-02-11 13:58:44 [✔]  created 0 nodegroup(s) in cluster "yodle"
2025-02-11 13:58:45 [ℹ]  nodegroup "on-demand-4" has 1 node(s)
2025-02-11 13:58:45 [ℹ]  node "ip-172-40-69-121.us-west-2.compute.internal" is ready
2025-02-11 13:58:45 [ℹ]  waiting for at least 1 node(s) to become ready in "on-demand-4"
2025-02-11 13:58:45 [▶]  event = watch.Event{Type:"ADDED", Object:(*v1.Node)(0xc00087f340)}
2025-02-11 13:58:45 [▶]  node "ip-172-40-69-121.us-west-2.compute.internal" is ready in "on-demand-4"
2025-02-11 13:58:45 [ℹ]  nodegroup "on-demand-4" has 1 node(s)
2025-02-11 13:58:45 [ℹ]  node "ip-172-40-69-121.us-west-2.compute.internal" is ready
2025-02-11 13:58:45 [✔]  created 1 managed nodegroup(s) in cluster "yodle"
2025-02-11 13:58:45 [▶]  nodegroups = [0xc0001c2640 0xc000a33680]
2025-02-11 13:58:45 [ℹ]  checking security group configuration for all nodegroups
2025-02-11 13:58:45 [▶]  nodegroup "on-demand-4" is compatible
2025-02-11 13:58:45 [▶]  nodegroup "spotty-2" is compatible
2025-02-11 13:58:45 [ℹ]  all nodegroups have up-to-date cloudformation templates

Anything else we need to know?

Host OS: Arch Linux, using a downloaded (not self-built) eksctl, default AWS credentials profile.

Versions

> eksctl info
eksctl version: 0.203.0
kubectl version: v1.30.0
OS: linux

[github-actions]

Hello mgalgs 👋 Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

[mgalgs]

Hmm, looking at this a bit more, my zone-specific workload is actually being scheduled on a node in my managed group that did land in the correct zone, so maybe there's something else going on here... I'll continue debugging. But is it expected for the launch template AZs to not match those requested at managedNodeGroup creation time?

The relevant event from kubectl describe node is:

  Warning  FailedScheduling  25s   default-scheduler   0/2 nodes are available: 1 node(s) had untolerated taint {spotty: true}, 1 node(s) had volume node affinity conflict. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

(specifically the bit about 1 node(s) had volume node affinity conflict, the untolerated taint message is expected and is from a node with taints in another managed node group)

[bryantbiggs]

I came across this example recently and it sounds similar to your setup - can you try adding the zones similar to this example to see if that resolves your issue

eksctl/examples/05-advanced-nodegroups.yaml

Line 68 in a8523c9

# cluster AZs must be set explicitly for single AZ nodegroup example to work

[mgalgs]

@bryantbiggs Interesting, that looked insanely promising but unfortunately the problem still remains after creating a new nodegroup with that in place... 😢

HOWEVER, that got me started down a path which serendipitously led me to this SO answer for the error message I was having... It turns out that my problem was simply not having installed the aws-ebs-csi-driver addon 😭 .

I added the following to my cluster config:

addons:
  # eks-pod-identity-agent is required for
  # `useDefaultPodIdentityAssociations` support, which is needed for
  # aws-ebs-csi-driver
  - name: eks-pod-identity-agent
  # aws-ebs-csi-driver is needed in order to get the latest EKS-supported
  # storage classes (like gp3 EBS).
  - name: aws-ebs-csi-driver
    useDefaultPodIdentityAssociations: true

and now I'm cooking!

It looks like gp2 might be supported out of the box with Kubernetes' in-tree storage drivers, but gp3 needs the ebs-csi-driver... The ability to use gp3 seems like a common enough use case that installing ebs-csi-driver should possibly be called out on the getting started page? But it does kind of snowball into a discussion about Pod Identity Associations and whatnot 🤷 ....

I really appreciate you taking the time to look at this.