Allow impersonating service accounts in arbitrary namespace on remote clusters

[schrej]

Context

We are using flux to deploy Kustomizations and HelmReleases to remote clusters by specifying a KubeConfig. Since the remote clusters are deployed using Cluster API, we use the cluster-admin Kubeconfig secret created by CAPI to do so.
With spec.serviceAccountName we're able to impersonate ServiceAccounts in the remote cluster as long as the SA resides in a namespace with the same name as the namespace of the Kustomization/HelmRelease on the local cluster.

In our case, the ServiceAccount exists in a different namespace on the remote cluster than the Kust/HR on the local cluster. Therefore we need to specify the namespace.

I assume only allowing to specify a name, but not a namespace, was done due to security considerations, which I totally agree with when impersonating on the local cluster. We could create a custom Kubeconfig with the SA credentials, but that would be quite a lot of effort.

Request

Allow specifying a ServiceAccount namespace for Kustomizations and HelmReleases.

Things to consider:

• maybe allow using spec.targetNamespace by setting a property
• only allow it when a custom Kubeconfig is specified, since that allows circumventing any restrictions anyway
• add a flag to controllers to enable this feature
• allow to specify the full SA name (system:serviceaccount:<namespace>:<name>) to avoid a new property

[matheuscscp]

In a RoleBinding you can specify a ServiceAccount from a different namespace :)

[schrej]

In a RoleBinding you can specify a ServiceAccount from a different namespace :)

Are you talking about a Kubernetes RBAC RoleBinding or did I miss something?

I'm not looking for a way to give a ServiceAccount appropriate permissions, I need to use a ServiceAccount in a specific namespace on a remote cluster.

For context: we're building a k8s platform and are implementing support for addons that are managed by 3rd-party teams. As part of the definition of an addon, a service account with appropriate permissions is specified. On the local cluster we're using one namespace per cluster we manage from it, e.g. cluster-dev1.
When deploying an addon, we're creating an addon-<name> namespace on the remote cluster. Then we're creating the specified ServiceAccount there. Now we need a way to use that ServiceAccount when deploying to the remote cluster, even though the Kust/HR resides in the cluster-dev1 namespace on the local cluster.

[matheuscscp]

We will most likely not implement support for impersonating a ServiceAccount in a different namespace, so I'll lay out here all the alternatives. (We will not add this change to the Flux APIs as it would seriously deviate from what Kubernetes does, sorry!)

Alternative using impersonation (.spec.serviceAccountName)

In this case, it's possible to grant all the RBAC permissions to the ServiceAccount on the remote cluster by creating RoleBinding objects and binding them to the ServiceAccount. The namespace of the RoleBinding object and the namespace of the ServiceAccount object do not need to be the same, Kubernetes allows granting cross-namespace namespaced RBAC.

Alternative 1 not using impersonation (not very recommended, as it involves a long-lived credential)

You can create a long-lived token for the ServiceAccount on the remote cluster like this:

apiVersion: v1
kind: Secret
metadata:
  name: long-lived-token
  namespace: the-ns-of-the-serviceaccount
  annotations:
    kubernetes.io/service-account.name: the-name-of-the-serviceaccount
type: kubernetes.io/service-account-token

This token does not expire. You can then create a kubeconfig with this token and use it with .spec.kubeConfig.secretRef.

Alternative 2 also not using impersonation (recommended)

You can use the secret-less authentication feature for remote clusters that we (literally) just released in Flux 2.7:

• kustomize-controller: https://fluxcd.io/flux/components/kustomize/kustomizations/#secret-less-authentication
• helm-controller: https://fluxcd.io/flux/components/helm/helmreleases/#secret-less-authentication

This feature supports 4 providers:

• aws for EKS
• azure for AKS
• gcp for GKE
• generic for https://kubernetes.io/docs/reference/access-authn-authz/authentication/#configuring-the-api-server

With the generic provider you can map the claims of the token issued for the ServiceAccount on the local/hub cluster to a username on the remote/spoke cluster. This username can be the ServiceAccount you want on the remote/spoke cluster, e.g. system:serviceaccount:<namespace>:<name>.

[schrej]

Thanks for the detailed response. The second alternative looks interesting, and I wasn't even aware that was possible. I'd improve the documentation around it a bit though. If you're unfamiliar with the topic, it's not obvious how this is supposed to work with the generic provider and cross-cluster SA token validation (it took me a while at least). It does require a substantial configuration (& architecture) change, so I'm not sure whether this will be a viable approach for us.

I would still prefer impersonation. I do understand the concerns about impersonating a SA in a different namespace, as namespaces are commonly used for access management. Not being able to use SAs in different namespaces absolutely makes sense within a single cluster. But when interacting with a remote cluster using a Kubeconfig, impersonation should be limited by the permissions of the user in the Kubeconfig, and not be restricted to a namespace with the same name on a completely different cluster.

There are other options for us to work around this limitation, like creating matching namespaces on the local/hub cluster and creating the Kustomizations there. But that requires syncing Kubeconfig secrets and scatters resources, which isn't great.

We will probably go with long-lived credentials for now, since that is the easiest option.

I would be happy to help with implementation of this feature, if you accept it.

[adberger]

I'm also in favor of having the option to specify the namespace or even use the notation system:serviceaccount:<namespace>:<service account name>.

A use-case would be for tenancy with capsule.
All serviceAccounts for tenant users are managed in the same namspace (e.g. tenant-system) and added as owners in the corresponding Tenant.
Additionally all serviceAccounts can be added as tenant userGroups: https://projectcapsule.dev/docs/tenants/permissions/#group-scope

[matheuscscp]

We could do it like this: we introduce .spec.serviceAccountNamespace for Kustomization and HelmRelease. This field can only be used when .spec.kubeConfig is also specified. This feature should also be placed behind the gate EnableServiceAccountNamespace, which should be opt-in (forever, never opt-out).

I would accept this feature (a bit reluctantly), but this is a very sensitive matter and other maintainers (@stefanprodan, @stealthybox) need to be convinced as well.

[matheuscscp]

I'd also consider .spec.impersonation.username and .spec.impersonation.groups.

[adberger]

@matheuscscp You mind explaining why only when .spec.kubeConfig is set (not wanting to sound accusatory)?
I really like the fact that new features are well considered in Flux tbh.

[matheuscscp]

@matheuscscp You mind explaining why only when .spec.kubeConfig is set (not wanting to sound accusatory)?

The use case I'm seeing here that (kinda) makes sense to me is when you're applying things in a remote cluster. I can see how mirroring namespaces across the hub and spoke clusters may sound a bit unneeded/verbose.

Same cluster should remain like Pod.spec.serviceAccountName. Pods can't use a ServiceAccount from a different namespace, why should Kustomizations and HelmReleases be able to do so?

[adberger]

@matheuscscp You mind explaining why only when .spec.kubeConfig is set (not wanting to sound accusatory)?

The use case I'm seeing here that (kinda) makes sense to me is when you're applying things in a remote cluster. I can see how mirroring namespaces across the hub and spoke clusters may sound a bit unneeded/verbose.

Same cluster should remain like Pod.spec.serviceAccountName. Pods can't use a ServiceAccount from a different namespace, why should Kustomizations and HelmReleases be able to do so?

I see, so this is more like a design decision rather than a technical impediment.

[stefanprodan]

One of the rules in Flux API design is that we don't go against Kubernetes namespaced object design. In Kubernetes a Pod can't run with an SA in different namespace nor can it mount a Secret/ConfigMap from a different namespace. I'm not for breaking this rule.

impersonation should be limited by the permissions of the user in the Kubeconfig

Really curious how are you achieving this, are you generating a Kubeconfig for each user and a dedicated role binding for verb: impersonate in the target cluster per user with the SA hardcoded? Can you please post here the RBAC you are using?

[schrej]

impersonation should be limited by the permissions of the user in the Kubeconfig

Really curious how are you achieving this, are you generating a Kubeconfig for each user and a dedicated role binding for verb: impersonate in the target cluster per user with the SA hardcoded? Can you please post here the RBAC you are using?

We aren't. For us it's the admin Kubeconfig created by CAPI.
In my opinion the namespace restrictions make less sense for cross-cluster interaction. It's not the same namespace, as it's on two separate clusters. If you want to impose restrictions on what can be done on the remote cluster, those restrictions should be enforced by the Kubeconfig user's permission, not by namespaces. The Kubeconfig should be (and is) required to be in the same namespace. But If you can impersonate SAs in different namespaces with that Kubeconfig, that should be allowed in my opinion. After all, if you don't specify a SA, you'll have all of the Kubeconfigs permissions as well.
Users that still want to rely on the current behavior can just keep the feature disabled.

More context:
In our management (local/hub) clusters we primarily use namespaces for organization, not to restrict access. Our tenant clusters (remote/spoke) have per-cluster tenancy. Still, namespaces are partly used for restrictions to separate platform and tenant components running there. We now want to introduce components that are provided as a service by separate teams, but are deployed to the tenant's cluster from our management clusters. Since we don't trust those teams with full cluster-admin access, we allow them to deploy a service account with their necessary rbac rules (which we review), and then impersonate that service account when applying their HelmReleases and Kustomizations. On the tenant cluster, that service account and the deployments should live in a addon-<name> namespace, while everything related to one cluster (CAPI, flux, our own CRs) is in c-<cluster name> on the management cluster.

We're running on-prem, not in one of the big clouds, so Workload Identity seems like quite a bit of extra effort. And unnecessary, considering we already have the Kubeconfig available.

[matheuscscp]

We're running on-prem, not in one of the big clouds, so Workload Identity seems like quite a bit of extra effort. And unnecessary, considering we already have the Kubeconfig available.

It's a security improvement to reduce the amount of long-lived credentials. Sure, long-lived credentials are always easier to set up, usually in any environment. Yes, security hardening often has extra costs.

[schrej]

It's a security improvement to reduce the amount of long-lived credentials. Sure, long-lived credentials are always easier to set up, usually in any environment. Yes, security hardening often has extra costs.

I agree. But as long as Cluster API doesn't fully adopt it, we'll have the Kubeconfig anyway.