Kargo 1.6.0 requests new Namespace kargo-cluster-secrets, is it possible to consolidate these into the Kargo namespace?

[avo-sepp]

Howdy,

I noticed we have a new namespace being requested on install of Kargo 1.6.0. It's a good amount of trouble from an RBAC and management perspective to have installs specify namespaces, additionally spreading an application across multiple namespaces is a challenge from an admin perspective.

I'm hoping to get some insight as to why this new namespace exists, and voice an opinion that I'm not a fan of this approach. Ideally hoping for a work-around.

Thanks,

[krancour]

Great question. It exists to account for a feature that is conspicuously absent from Kubernetes itself.

It's not at all uncommon for resources of different types to reference a Secret. Your Deployments and Pods do it. Certificates do it. Ingresses do it. Some Kargo-specific resource types do it as well. Notably, as of v1.6.0, ProjectConfig resources may include the definition of webhook receivers, and those all must reference Secrets.

As a general rule, namespaced resources only reference Secrets in the same namespace. This is true for ProjectConfig resources, but it's equally true for core Kubernetes resource types like Pods, Deployments, etc. By design, a Pod can neither mount a Secret from a different namespace nor define an environment variable by referencing a Secret in another namespace. (This is why you never need to specify a namespace when referencing a Secret. It's understood that the reference is to a Secret in the same namespace.)

Where things get very tricky is where cluster-scoped resources are concerned. The ProjectConfig resource type has a cluster-scoped analog -- ClusterConfig. ClusterConfig may contain the definition of cluster-scoped webhook receivers which, like their namespaced counterparts, must reference Secrets. Since a ClusterConfig resource is not, itself, namespaced, and because Kubernetes lacks any concept of a "ClusterSecret" (non-namespaced Secret), it raises the dilemma of exactly where the Secrets referenced by a ClusterConfig resource (or any other current or future cluster-scoped Kargo resources) ought to live.

The most reasonable solution to this was to designate a namespace as a repository for Secrets that are, for all intents and purposes, cluster-scoped and intended to be referenced by (actual) cluster-scoped Kargo resources. It's worth noting that our plans for this namespace don't end with ClusterConfig. There are already other features in-development that face the same dilemma.

Can this namespace be consolidated with Kargo's own namespace?

Yes and no.

First, I do not recommend it. Doing so would create the possibility of cluster-scoped resources referencing Secrets that belong to Kargo itself and were very much not intended to be cluster-scoped in any capacity.

Is it technically possible -- yes. Although the thing that stands in your way is that the chart currently lacks the flexibility that would permit you to easily achieve that.

The thing I think we absolutely, positively should do is create an option in the chart for Helm not to manage the designated cluster-scoped namespace. (Similar to how most charts, including ours, give you the option to not have Helm manage CRDs or RBAC-related resources.) By opting out of that, you would be free to "bring your own" namespace, which can be provisioned and managed in any way you choose. (And you could, of course, choose to do nothing special to provision or manage a separate namespace for this purpose and then do the thing I don't recommend by designating Kargo's own namespace as the one where "cluster-scoped" Secrets are kept.)

[avo-sepp]

It seems like ClusterConfig is really a Kargo admin-level resource. For setting up universal webhook support. The risk you're pointing out is that it can be used to access secrets other than what it should be using. That seems valid.

But if I consider the use case that a Kargo admin is the only person allowed to create ClusterConfig resources, and the Kargo admin already has access to the Kargo namespace for creating and updating secrets, why would they need to use this work around to gain access to secrets? These secrets are already in-scope for them.

[krancour]

It seems like ClusterConfig is really a Kargo admin-level resource. For setting up universal webhook support.

Yes. That's right.

But if I consider the use case that a Kargo admin is the only person allowed to create ClusterConfig resources, and the Kargo admin already has access to the Kargo namespace for creating and updating secrets, why would they need to use this work around to gain access to secrets? These secrets are already in-scope for them.

I agree that this is probably true the vast, vast majority of the time, but I don't want to take that for granted. I try not to talk about our SaaS a lot here, but I can cite it as a realistic example where the person (or thing, in this case) responsible for installing Kargo isn't the person responsible for managing Kargo. When Kargo is provisioned through the Akuity Platform, no one is meant to have access to the Secrets in Kargo's own namespace. I'm sure there are other realistic scenarios, but this was a real-life one that was top-of-mind.

But as I said... I'm perfectly happy to make some chart modifications so it's easier for you or anyone else to choose to use one namespace for both purposes if you wish.