Skip to content

enabling hybrid nodes on existing cluster #3357

@FernandoMiguel

Description

@FernandoMiguel

Description

According to aws docs https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-cluster-update.html
it is possible to enable EKS hybrid nodes on an existing cluster
as such, we created a new EKS v1.31 cluster without hybrid nodes and tried to enable it afterwards, to test if we could enable it on existing production clusters.

here's the example to enable it
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/README.md#eks-hybrid-nodes

this is our code for it

  ## EKS Hybrid Nodes
  cluster_remote_network_config = var.remote_node_network_cidr != "" ? {
    remote_node_networks = {
      cidrs = ["10.48.126.0/23"]
    }
    # Required if running webhooks on Hybrid nodes
    remote_pod_networks = {
      cidrs = ["10.57.126.0/23"]
    }
  } : {}
  • ✋ I have searched the open/closed issues and my issue is not listed.

Versions

  • Module version [Required]:
    version = "~> 20"

  • Terraform version:
    Terraform v1.12.0-rc2
    on darwin_arm64

  • provider registry.terraform.io/hashicorp/aws v5.97.0
  • provider registry.terraform.io/hashicorp/cloudinit v2.3.7
  • provider registry.terraform.io/hashicorp/kubernetes v2.36.0
  • provider registry.terraform.io/hashicorp/null v3.2.4
  • provider registry.terraform.io/hashicorp/random v3.7.2
  • provider registry.terraform.io/hashicorp/time v0.13.1
  • provider registry.terraform.io/hashicorp/tls v4.1.0

Reproduction Code [Required]

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20"

  access_entries                           = local.access_entries_merge
  attach_cluster_encryption_policy         = var.attach_cluster_encryption_policy
  authentication_mode                      = "API"
  cloudwatch_log_group_kms_key_id          = var.cloudwatch_log_group_kms_key_id
  cloudwatch_log_group_retention_in_days   = local.cloudwatch_log_group_retention_in_days
  cluster_enabled_log_types                = var.cluster_enabled_log_types
  cluster_encryption_config                = var.cluster_encryption_config
  cluster_endpoint_private_access          = true
  cluster_endpoint_public_access           = false
  cluster_ip_family                        = "ipv4"
  cluster_name                             = local.cluster_name
  cluster_security_group_additional_rules  = local.cluster_security_group_additional_rules
  cluster_security_group_tags              = local.cluster_security_group_tags
  cluster_service_ipv4_cidr                = var.cluster_service_ipv4_cidr
  cluster_version                          = var.kubernetes_version
  create_iam_role                          = true
  create_node_security_group               = true
  enable_cluster_creator_admin_permissions = true
  iam_role_additional_policies             = local.iam_role_additional_policies
  iam_role_name                            = local.iam_role_name
  iam_role_use_name_prefix                 = true
  kms_key_enable_default_policy            = true
  kms_key_owners                           = local.kms_key_owners
  node_security_group_tags                 = local.node_security_group_tags
  node_security_group_use_name_prefix      = true
  subnet_ids                               = data.aws_subnets.private_subnets.ids
  vpc_id                                   = data.aws_vpc.vpc_id.id

  ## EKS Hybrid Nodes
  cluster_remote_network_config = var.remote_node_network_cidr != "" ? {
    remote_node_networks = {
      cidrs = [var.remote_node_network_cidr]
    }
    # Required if running webhooks on Hybrid nodes
    remote_pod_networks = {
      cidrs = [var.remote_pod_network_cidr]
    }
  } : {}

  cloudwatch_log_group_tags = {
    BusinessUnit = "security"
  }

  tags = {
    cluster_name_tag = local.cluster_name
  }
}

Steps to reproduce the behavior:
create a eks cluster without hybrid nodes IPs
enable the variables of hybrid nodes CIDRs

Expected behavior

for the cluster to be updated in-place

Actual behavior

  # module.eks.module.eks.aws_eks_cluster.this[0] must be replaced
+/- resource "aws_eks_cluster" "this" {
      ~ arn                           = "arn:aws:eks:us-east-1:XXXXX:cluster/cts-hybrid-ue1-lw" -> (known after apply)
        name                          = "cts-hybrid-ue1-lw"
      ~ platform_version              = "eks.25" -> (known after apply)
      ~ status                        = "ACTIVE" -> (known after apply)

      ~ kubernetes_network_config {
          ~ service_ipv4_cidr = "172.20.0.0/16" -> (known after apply)
          + service_ipv6_cidr = (known after apply)
            # (1 unchanged attribute hidden)

          ~ elastic_load_balancing (known after apply)
          - elastic_load_balancing {
              - enabled = false -> null
            }
        }

      + remote_network_config { # forces replacement
          + remote_node_networks {
              + cidrs = [ # forces replacement
                  + "10.48.126.0/23",
                ]
            }
          + remote_pod_networks {
              + cidrs = [ # forces replacement
                  + "10.57.126.0/23",
                ]
            }
        }

      ~ upgrade_policy (known after apply)
      - upgrade_policy {
          - support_type = "EXTENDED" -> null
        }

        # (3 unchanged blocks hidden)
    }

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions