LimitExceeded: Cannot exceed quota for ACLSizePerRole: 2048

[j4zzcat]

Hi,

I'm finalizing the identity layer following https://docs.cloudposse.com/layers/identity, running atmos workflow deploy/tfstate -f baseline but getting LimitExceeded: Cannot exceed quota for ACLSizePerRole: 2048.

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

OpenTofu will perform the following actions:

  # aws_iam_role.default["qh-core-gbl-root-tfstate"] will be updated in-place
  ~ resource "aws_iam_role" "default" {
      ~ assume_role_policy    = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Condition = {
                          ~ ArnLike      = {
                              ~ "aws:PrincipalArn" = "arn:aws:iam::324037278566:user/SuperAdmin" -> [
                                  + "arn:aws:iam::218289152745:role/qh-core-gbl-identity-developers",
                                  + "arn:aws:iam::218289152745:role/qh-core-gbl-identity-devops",
                                  + "arn:aws:iam::218289152745:role/qh-core-gbl-identity-gitops",
                                  + "arn:aws:iam::218289152745:role/qh-core-gbl-identity-managers",
                                  + "arn:aws:iam::218289152745:role/qh-core-gbl-identity-planners",
                                  + "arn:aws:iam::324037278566:role/qh-core-gbl-root-admin",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityDevelopersTeamAccess_*",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityDevopsTeamAccess_*",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityManagersTeamAccess_*",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_AdministratorAccess_*",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityPlannersTeamAccess_*",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityGitopsTeamAccess_*",
                                  + "arn:aws:iam::324037278566:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_TerraformUpdateAccess_*",
                                ]
                            }
                          + StringEquals = {
                              + "aws:PrincipalType" = "AssumedRole"
                            }
                        }
                      ~ Principal = {
                          ~ AWS = "arn:aws:iam::324037278566:root" -> [
                              + "arn:aws:iam::324037278566:root",
                              + "arn:aws:iam::218289152745:root",
                            ]
                        }
                      ~ Sid       = "ColdStartRoleAssumeRole" -> "RoleAssumeRole"
                        # (2 unchanged attributes hidden)
                    },
                  + {
                      + Action    = [
                          + "sts:TagSession",
                          + "sts:SetSourceIdentity",
                          + "sts:AssumeRole",
                        ]
                      + Condition = {
                          + ArnLike = {
                              + "aws:PrincipalArn" = "arn:aws:iam::324037278566:user/SuperAdmin"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = {
                          + AWS = "arn:aws:iam::324037278566:root"
                        }
                      + Sid       = "PrincipalAssumeRole"
                    },
                  + {
                      + Action    = [
                          + "sts:TagSession",
                          + "sts:SetSourceIdentity",
                          + "sts:AssumeRole",
                        ]
                      + Condition = {
                          + ArnLike      = {
                              + "aws:PrincipalArn" = "arn:aws:iam::*:user/*"
                            }
                          + ArnNotEquals = {
                              + "aws:PrincipalArn" = "arn:aws:iam::324037278566:user/SuperAdmin"
                            }
                        }
                      + Effect    = "Deny"
                      + Principal = {
                          + AWS = [
                              + "arn:aws:iam::324037278566:root",
                              + "arn:aws:iam::218289152745:root",
                            ]
                        }
                      + Sid       = "RoleDenyAssumeRole"
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        id                    = "qh-core-gbl-root-tfstate"
        name                  = "qh-core-gbl-root-tfstate"
        tags                  = {
            "Environment" = "use1"
            "Name"        = "qh-core-gbl-root-tfstate"
            "Namespace"   = "qh"
            "Stage"       = "root"
            "Tenant"      = "core"
        }
        # (9 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
aws_iam_role.default["qh-core-gbl-root-tfstate"]: Modifying... [id=qh-core-gbl-root-tfstate]
╷
│ Error: updating IAM Role (qh-core-gbl-root-tfstate) assume role policy: operation error IAM: UpdateAssumeRolePolicy, https response error StatusCode: 409, RequestID: 423b95b7-2cf1-4d2e-8dd2-537cb88b2022, LimitExceeded: Cannot exceed quota for ACLSizePerRole: 2048
│
│   with aws_iam_role.default["qh-core-gbl-root-tfstate"],
│   on iam.tf line 81, in resource "aws_iam_role" "default":
│   81: resource "aws_iam_role" "default" {
│
╵
 Error

exit status 1
 Error

[milldr]

This is due to a default ACL size limit for AWS. You could put in a request for a larger size limit, or comment out (or remove) the following. The additional allowed roles are to handle edge use cases, that you may not need at this time.

Here are all places that may exceed the default quota for ACLSizeLimit

aws-sso

# stacks/catalog/aws-sso.yaml
...
         # If you add a new team through `aws-team` component, add it here. this will create the `Identity%sTeamAccess` Permission Set
         # you can then assign that permission set to a group - usually that team group. Take care to note that the set will uppercase
         # the first letter in the team name.
         aws_teams_accessible:
           - devops
           - developers
           - managers
-          - helpdesk
+          # Optionally grant the helpdesk team programmatic access to the support AWS Team role across all accounts.
+          # This creates a new Permission Set and requires adding the team to the `aws-teams` component as well.
+          # - helpdesk
         account_assignments:
           core-root:
             groups:

aws-teams

# stacks/catalog/aws-teams.yaml
...
          # The "planners" team is intended to be used grant a group of users permission to plan Terraform.
          # Planning Terraform requires at least read access to TFState Backend, and read access in the target account.
          # In particular, Cloud Posse uses this Team with the Atmos Terraform Plan GitHub Action to plan affected components in a given pull request without required an overly permissive role
          planners:
            <<: *user-template
            role_description: "Team to run `terraform plan` in all accounts"
            aws_saml_login_enabled: true
            role_policy_arns:
              - "arn:aws:iam::aws:policy/job-function/ViewOnlyAccess"
              - team_role_access
-            trusted_teams:
-              - devops
-              - managers
-              - developers
-            trusted_permission_sets:
-              - IdentityPlannersTeamAccess
-              - IdentityDevopsTeamAccess
-              - IdentityManagersTeamAccess
+            # You can optionally give human users access to the planners role.
+            # Human users typically would use their given team, whereas the planners role is used for machine users.
+            # In some cases we prefer to reuse the planners team or grant a team access for the sake of debugging.
+            #
+            # However, enabling the following will likely put the ACL size beyond the default size limit and will require a quota increase.
+            #
+            # trusted_teams:
+            #   - devops
+            #   - managers
+            #   - developers
+            # trusted_permission_sets:
+            #   - IdentityPlannersTeamAccess
+            #   - IdentityDevopsTeamAccess
+            #   - IdentityManagersTeamAccess

...


           gitops:
             <<: *user-template
             role_description: Team for our privileged GitHub GitOps workflows to run `terraform apply` in most accounts
             role_policy_arns:
             - team_role_access
             aws_saml_login_enabled: false
-            trusted_teams:
-              - devops
-              - managers
-            trusted_role_arns: []
+            # The gitops team is intended to be used by machines to apply Terraform. We do not recommend using the gitops team for human users, since it creates confusion for intended usage. Human users should use their given team.
+            # However, you may want to enable the following for the sake of debugging.
+            # trusted_teams:
+            #   - devops
+            #   - managers

...

-          helpdesk:
-            <<: *user-template
-            role_description: "Team with permissions for accessing the AWS Support Service"
-            role_policy_arns:
-            - "arn:aws:iam::aws:policy/AWSSupportAccess"
-            - "arn:aws:iam::aws:policy/AWSTrustedAdvisorPriorityReadOnlyAccess"
-            - "team_role_access"
-            aws_saml_login_enabled: true
-            trusted_teams:
-              - devops
-              - managers
-              - helpdesk
-            trusted_permission_sets:
-              - IdentityDevopsTeamAccess
-              - IdentityManagersTeamAccess
-              - IdentityHelpdeskTeamAccess
+          # You can optionally add the helpdesk team.
+          # This is useful if you want to give a group of human users access to programmatically assume the support AWS Team role across all accounts.
+          # Make sure to also add the helpdesk team to `var.aws_teams_accessible` in the `aws-sso` component.
+          # 
+          # helpdesk:
+          #   <<: *user-template
+          #   role_description: "Team with permissions for accessing the AWS Support Service"
+          #   role_policy_arns:
+          #   - "arn:aws:iam::aws:policy/AWSSupportAccess"
+          #   - "arn:aws:iam::aws:policy/AWSTrustedAdvisorPriorityReadOnlyAccess"
+          #   - "team_role_access"
+          #   aws_saml_login_enabled: true
+          #   trusted_teams:
+          #     - devops
+          #     - managers
+          #     - helpdesk
+          #   trusted_permission_sets:
+          #     - IdentityDevopsTeamAccess
+          #     - IdentityManagersTeamAccess
+          #     - IdentityHelpdeskTeamAccess

aws-team-roles

# stacks/catalog/aws-team-roles.yaml
...
           support:
             <<: *user-template
             enabled: true
             role_policy_arns:
               - "arn:aws:iam::aws:policy/AWSSupportAccess"
               - "arn:aws:iam::aws:policy/AWSTrustedAdvisorPriorityReadOnlyAccess"
             role_description: "Role with permissions for accessing the AWS Support Service"
             trusted_teams:
               - "devops"
               - "managers"
-              - "helpdesk"
+              # You can optionally add the helpdesk team here. This would allow the helpdesk team to assume this role across all accounts.
+              # Make sure to also add the helpdesk team to the `aws-teams` component.
+              # - helpdesk

[petabook]

Thanks.
I believe that inline policies cannot exceed 2KiB, and there's no way to increase this limit.

[petabook]

Chopped as suggested, but still the same error:

  # aws_iam_role.default["qh-core-gbl-root-tfstate"] will be updated in-place
  ~ resource "aws_iam_role" "default" {
      ~ assume_role_policy    = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Condition = {
                          ~ ArnLike      = {
                              ~ "aws:PrincipalArn" = "arn:aws:iam::324037278566:user/SuperAdmin" -> [
                                  + "arn:aws:iam::218289152745:role/qh-core-gbl-identity-developers",
                                  + "arn:aws:iam::218289152745:role/qh-core-gbl-identity-devops",
                                  + "arn:aws:iam::218289152745:role/qh-core-gbl-identity-gitops",
                                  + "arn:aws:iam::218289152745:role/qh-core-gbl-identity-managers",
                                  + "arn:aws:iam::218289152745:role/qh-core-gbl-identity-planners",
                                  + "arn:aws:iam::324037278566:role/qh-core-gbl-root-admin",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityDevelopersTeamAccess_*",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityDevopsTeamAccess_*",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityManagersTeamAccess_*",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_AdministratorAccess_*",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityPlannersTeamAccess_*",
                                  + "arn:aws:iam::218289152745:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityGitopsTeamAccess_*",
                                  + "arn:aws:iam::324037278566:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_TerraformUpdateAccess_*",
                                ]
                            }
                          + StringEquals = {
                              + "aws:PrincipalType" = "AssumedRole"
                            }
                        }
                      ~ Principal = {
                          ~ AWS = "arn:aws:iam::324037278566:root" -> [
                              + "arn:aws:iam::324037278566:root",
                              + "arn:aws:iam::218289152745:root",
                            ]
                        }
                      ~ Sid       = "ColdStartRoleAssumeRole" -> "RoleAssumeRole"
                        # (2 unchanged attributes hidden)
                    },
                  + {
                      + Action    = [
                          + "sts:TagSession",
                          + "sts:SetSourceIdentity",
                          + "sts:AssumeRole",
                        ]
                      + Condition = {
                          + ArnLike = {
                              + "aws:PrincipalArn" = "arn:aws:iam::324037278566:user/SuperAdmin"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = {
                          + AWS = "arn:aws:iam::324037278566:root"
                        }
                      + Sid       = "PrincipalAssumeRole"
                    },
                  + {
                      + Action    = [
                          + "sts:TagSession",
                          + "sts:SetSourceIdentity",
                          + "sts:AssumeRole",
                        ]
                      + Condition = {
                          + ArnLike      = {
                              + "aws:PrincipalArn" = "arn:aws:iam::*:user/*"
                            }
                          + ArnNotEquals = {
                              + "aws:PrincipalArn" = "arn:aws:iam::324037278566:user/SuperAdmin"
                            }
                        }
                      + Effect    = "Deny"
                      + Principal = {
                          + AWS = [
                              + "arn:aws:iam::324037278566:root",
                              + "arn:aws:iam::218289152745:root",
                            ]
                        }
                      + Sid       = "RoleDenyAssumeRole"
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        id                    = "qh-core-gbl-root-tfstate"
        name                  = "qh-core-gbl-root-tfstate"
        tags                  = {
            "Environment" = "use1"
            "Name"        = "qh-core-gbl-root-tfstate"
            "Namespace"   = "qh"
            "Stage"       = "root"
            "Tenant"      = "core"
        }
        # (9 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
aws_iam_role.default["qh-core-gbl-root-tfstate"]: Modifying... [id=qh-core-gbl-root-tfstate]
╷
│ Error: updating IAM Role (qh-core-gbl-root-tfstate) assume role policy: operation error IAM: UpdateAssumeRolePolicy, https response error StatusCode: 409, RequestID: 476e705b-fc78-4573-b9c1-3a3e37a6d81e, LimitExceeded: Cannot exceed quota for ACLSizePerRole: 2048
│
│   with aws_iam_role.default["qh-core-gbl-root-tfstate"],
│   on iam.tf line 81, in resource "aws_iam_role" "default":
│   81: resource "aws_iam_role" "default" {
│
╵
 Error

exit status 1
 Error

[milldr]

ah we need tfstate-backend as well

# stacks/catalog/tfstate-backend.yaml
...
             denied_roles: {}
-            allowed_permission_sets:
-              core-root:
-              - TerraformUpdateAccess
-              core-identity:
-              - IdentityDevelopersTeamAccess
-              - IdentityDevopsTeamAccess
-              - IdentityManagersTeamAccess
-              - AdministratorAccess
-            denied_permission_sets: {}
+            # Allow the SuperAdmin user access to the Terraform state access roles.
+            # This is necessary to allow SuperAdmin to use the same pattern as an AWS Team to access the backend.
            allowed_principal_arns:
              - arn:aws:iam::YOUR_ROOT_ACCOUNT:user/SuperAdmin
+            denied_principal_arns: []
+            #
+            # You can optionally give Permission Sets direct access to the Terraform State access roles.
+            # This would allow Permission Sets to bypass the role-chaining mechanism required for applying Terraform across multiple accounts.
+            # However, enabling these permission sets will likely put the ACL size beyond the default size limit and will require a quota increase.
+            #
+            allowed_permission_sets: {}
+            # allowed_permission_sets:
+            #   core-root:
+            #   - TerraformUpdateAccess
+            #   core-identity:
+            #   - IdentityDevelopersTeamAccess
+            #   - IdentityDevopsTeamAccess
+            #   - IdentityManagersTeamAccess
+            #   - AdministratorAccess
+            denied_permission_sets: {}

[GabisCampana]

Hi @j4zzcat

Has your question been answered?

[GabisCampana]

@j4zzcat, just checking in—do Dan’s replies above resolve your question? If so, feel free to mark this as answered. Let us know if there’s anything still unclear!