Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Install AWS EKS Add-on #167

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

feat: Install AWS EKS Add-on #167

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f05b333
get upgrade working
Jan 9, 2024
bb5eaa6
install EKS VPC-CNI Add-on
Jan 11, 2024
803acbd
cleanup
Jan 11, 2024
82127cd
Merge branch 'main' into george-test
Jan 22, 2024
6898ebb
remove extra spaces
Feb 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions modules/app_eks/add-ons.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#########################################
# OIDC stuff for VPC CNI
#########################################
data "tls_certificate" "vpc_cni" {
url = module.eks.cluster_oidc_issuer_url
}

resource "aws_eks_addon" "vpc_cni" {
depends_on = [
module.eks,
aws_iam_openid_connect_provider.eks,
aws_iam_role_policy_attachment.vpc_cni
]

addon_name = "vpc-cni"
addon_version = "v1.13.0-eksbuild.1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This version is different than the deleted one, just wanna make sure this is the right one

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. One of the things that can be used to differentiate the self-managed plugin vs. the EKS add-on are the version numbers. This particular version is the next "in line" -- most of our clusters, if not all, are running v.1.12 of the plugin, and I'm gunning for keeping the AWS recommendation of upgrading the CNI one minor version at a time.

cluster_name = var.namespace
preserve = false
resolve_conflicts = "OVERWRITE"
service_account_role_arn = aws_iam_role.node.arn
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this change have any impact on existing workloads?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should not have any impact at all.

28 changes: 16 additions & 12 deletions modules/app_eks/iam-role-attachments.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ resource "aws_iam_role_policy_attachment" "node_kms" {
policy_arn = aws_iam_policy.node_kms.arn
}

resource "aws_iam_role_policy_attachment" "node_secrets_manager" {
role = aws_iam_role.node.name
policy_arn = aws_iam_policy.secrets_manager.arn
}

resource "aws_iam_role_policy_attachment" "node_sqs" {
role = aws_iam_role.node.name
policy_arn = aws_iam_policy.node_sqs.arn
Expand All @@ -23,6 +28,16 @@ resource "aws_iam_role_policy_attachment" "node_s3" {
policy_arn = aws_iam_policy.node_s3.arn
}

resource "aws_iam_role_policy_attachment" "ebs_csi" {
role = aws_iam_role.node.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry" {
role = aws_iam_role.node.name
policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}

resource "aws_iam_role_policy_attachment" "eks_cni" {
role = aws_iam_role.node.name
policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
Expand All @@ -38,17 +53,6 @@ resource "aws_iam_role_policy_attachment" "eks_worker_node" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry" {
role = aws_iam_role.node.name
policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}

resource "aws_iam_role_policy_attachment" "ebs_csi" {
role = aws_iam_role.node.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

resource "aws_iam_role_policy_attachment" "node_secrets_manager" {
role = aws_iam_role.node.name
policy_arn = aws_iam_policy.secrets_manager.arn
}

16 changes: 16 additions & 0 deletions modules/app_eks/iam-role-policies.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,21 @@ data "aws_iam_policy_document" "node_assume" {
identifiers = ["ec2.amazonaws.com"]
}
}

statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
effect = "Allow"

condition {
test = "StringEquals"
variable = "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub"
values = ["system:serviceaccount:kube-system:aws-node"]
}

principals {
identifiers = [aws_iam_openid_connect_provider.eks.arn]
type = "Federated"
}
}
}

1 change: 0 additions & 1 deletion modules/app_eks/iam-roles.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
resource "aws_iam_role" "node" {
name = "${var.namespace}-node"
assume_role_policy = data.aws_iam_policy_document.node_assume.json

}


32 changes: 5 additions & 27 deletions modules/app_eks/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,36 +4,14 @@ locals {
mysql_port = 3306
redis_port = 6379
encrypt_ebs_volume = true
}


resource "aws_eks_addon" "eks" {
cluster_name = var.namespace
addon_name = "aws-ebs-csi-driver"
depends_on = [
module.eks
]
}

resource "aws_eks_addon" "efs" {
cluster_name = module.eks.cluster_id
addon_name = "aws-efs-csi-driver"
addon_version = "v1.7.1-eksbuild.1" # Ensure this version is compatible
resolve_conflicts = "OVERWRITE"
depends_on = [
module.eks
]
managed_policy_arns = concat([
"arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
"arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
], var.eks_policy_arns)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where does this get used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remember when we did all the work on the IAM policy documents? (https://github.com/wandb/terraform-aws-wandb/pull/101/files). I think we determined we have to pass these here as well or the policies aren't attached to the node or the cluster. I'm not sure which.

But if you're asking about the policy specific to the CNI driver, that gets used for anything involving networking in EKS, so grabbing elastic ips, setting up container networking, etc.

}

# removed due to conflict with
# AWS Load Balancer Controller
# being installed with Helm.
# See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.6/
#resource "aws_eks_addon" "vpc_cni" {
# cluster_name = var.namespace
# addon_name = "vpc-cni"
# depends_on = [module.eks]
#}

module "eks" {
source = "terraform-aws-modules/eks/aws"
Expand Down
Loading