Using AMI AL2003 (Amazon Linux 2023) in AWS EKS

[sw360cab]

Using this AMI is currently not compatible with AWS EBS CSI plugin for dynamic PVC provisioning.

PROBLEM

When using the plugin for dynamic provisioning there is an error

Events:
  Type     Reason                Age                 From                                                                                     Message
  ----     ------                ----                ----                                                                                     -------
  Normal   WaitForFirstConsumer  102s                persistentvolume-controller                                                              waiting for first consumer to be created before binding
  Normal   Provisioning          36s (x7 over 102s)  ebs.csi.aws.com_ebs-csi-controller-c4f444f56-55z5t_458c77bb-a406-4c43-a679-db85ec72e49a  External provisioner is provisioning volume for claim "gno/indexer-pv-claim"
  Warning  ProvisioningFailed    35s (x7 over 101s)  ebs.csi.aws.com_ebs-csi-controller-c4f444f56-55z5t_458c77bb-a406-4c43-a679-db85ec72e49a  failed to provision volume with StorageClass "eks-ebs-storage": rpc error: code = Internal desc = Could not create volume "pvc-d51fc932-aa17-4833-9a23-0336e93a07ee": could not create volume in EC2: operation error EC2: CreateVolume, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested
  Normal   ExternalProvisioning  1s (x9 over 102s)   persistentvolume-controller                                                              Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

This is due to the fact that the plugin has some limitation with IMDS on Kubernetes in AWS:

You will need to provide the EBS CSI Node pods with access to either IMDS (for example, by raising the hop limit, see our FAQ) or the Kubernetes API server (by finding and configuring what is blocking its access to enable communication between the pod and the Kubernetes API) for it to function.

This has also a vast literature in the plugin repository itself

• Better observability into driver auth/networking errors · Issue #2263 · kubernetes-sigs/aws-ebs-csi-driver
• Occasional retrieving IMDS metadata failed on AL2023 · Issue #2262 · kubernetes-sigs/aws-ebs-csi-driver
• Failed to retrieve instance data from ec2 metadata · Issue #2031 · kubernetes-sigs/aws-ebs-csi-driver

SOLUTION

Node Groups should be provisioned using a Launch template, which overrides the HttpPutResponseHopLimit increasing it to 2, as also suggested in the Terraform AWS Provider issue

AL2023 requires IMDSv2 by default. IMDSv2 has several benefits that help improve security posture. It uses a session-oriented authentication method that requires the creation of a secret token in a simple HTTP PUT request to start the session.
....
For IMDSv2, the default hop count for managed node groups is set to 1. This means that containers won't have access to the node's credentials using IMDS. If you require container access to the node's credentials, you can still do so by manually overriding the HttpPutResponseHopLimit in a custom EC2 launch template, increasing it to 2, and by using EKS Pod Identity.

it can be defined in a Terraform aws_launch_template resource by adding:

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
    instance_metadata_tags      = "enabled"
  }

EXTRA

A correct dynamic provisioning of the volume with the plugin has the following output:

Events:
  Type    Reason                 Age                From                                                                                     Message
  ----    ------                 ----               ----                                                                                     -------
  Normal  WaitForFirstConsumer   10s                persistentvolume-controller                                                              waiting for first consumer to be created before binding
  Normal  Provisioning           10s                ebs.csi.aws.com_ebs-csi-controller-c4f444f56-qgw78_38309d71-fe4c-49ec-9c7a-0b09120ae077  External provisioner is provisioning volume for claim "gno/indexer-pv-claim"
  Normal  ExternalProvisioning   10s (x2 over 10s)  persistentvolume-controller                                                              Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  ProvisioningSucceeded  8s                 ebs.csi.aws.com_ebs-csi-controller-c4f444f56-qgw78_38309d71-fe4c-49ec-9c7a-0b09120ae077  Successfully provisioned volume pvc-c48e63a4-3b0c-4e48-88ac-79356fda7dac