EKS Auto Mode example for custom NodePool/NodeClass

[cdenneen]

Currently EKS AutoMode doesn't seem to propagate custom tags from the eks module to the created NodeClass/NodeGroup/LaunchTemplate/EC2/EBS/etc.

According to docs that is doable via custom policy [1] which this module already adds to the Cluster IAM Role so it seems that policy isn't enough and needs to be combined with replacing the "default" NodeClass with a custom one with tags and would imagine a custom NodePools to use that new custom NodeClass to replace "system" and "general-purpose".

I find the fact that tags can't be merged() with the managed tags handled by EKS Auto Mode to be frustrating but I know what is an upstream issue which is outside the scope of this module and probably even the AWS provider until AWS allows it.

Because of that I would really appreciate an example like current one [2], that provides example of how to create custom NodePool, NodeClass and have the cluster_compute_config use that in order to populate tags. Sadly since the "tags" is the only value changing we are duplicating the default's so what those defaults are today could change tomorrow (like the default requirements in the built-in node pools).

1 - https://docs.aws.amazon.com/eks/latest/userguide/auto-learn-iam.html#tag-prop
2 - https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/eks-auto-mode

[bryantbiggs]

I don't quite follow - if you are using a custom nodepool, you define the tags in the custom nodepool manifest which is typically just some YAML.

The only thing the module provides is the ability to enable the permissions that allow EKS Auto Mode to tag resources created (enabled by default

terraform-aws-eks/variables.tf

Line 554 in 325c3fe

default = true

)

There is no connection between this module and what gets populated in your custom nodepool manifest

[cdenneen]

@bryantbiggs I am not using a custom nodepool/nodeclass. I'm currently using the default.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "20.36.0"
...
  cluster_compute_config = {
    enabled    = true
    node_pools = ["system", "general-purpose"]
  }
...
  cluster_tags = merge(local.tags, {
    Name                     = local.name
    "karpenter.sh/discovery" = local.name
  })
   tags = local.tags
...
}

But the nodes don't have any of my tags from the module only the ones from AWS:

aws:eks:cluster-name
eks:eks-cluster-name
kubernetes.io/cluster/dev-use1-nlplatf
aws:ec2:managed-launch
aws:ec2launchtemplate:version
aws:ec2launchtemplate:id
eks:kubernetes-node-class-name
aws:ec2:fleet-id
eks:kubernetes-node-pool-name

So these are the only tags added to nodes created from the default NodeClass and subset of these seem to be added to the EBS volumes created for those instances.

I would like to avoid running custom nodepool/nodeclass and the tags just work.
My question is based on that not being possible. If it should be working then I think there is a bug somewhere.
If it's not possible then could there be an updated example to create use a custom nodepool/nodeclass that will leverage the custom tags policy added in v20.31.4?

[bryantbiggs]

the module doesn't control the propagation of tags from the module to nodes created by EKS Auto Mode node pools - you will need to talk to AWS/EKS about that as a feature request

[cdenneen]

@bryantbiggs ok I can see this #3242 did put policy on cluster role and not node role but can you confirm it works? If the point of this is to add custom tags to cluster and

terraform-aws-eks/variables.tf

Lines 551 to 555 in 8a0efdb

	variable "enable_auto_mode_custom_tags" {
	description = "Determines whether to enable permissions for custom tags resources created by EKS Auto Mode"
	type = bool
	default = true
	}

is supposed to enable those custom tags. Any idea why my EKS Auto nodes don't have the tags?