Ability to limit the operator's scope to specific Kubernetes namespaces

[jgarcia-sqsp]

I'm currently trying to set up the spicedb-operator in our Kubernetes cluster but I'm running into an issue around permissions. Currently the operator is failing to start in our cluster because it seemingly doesn't have cluster-scoped permissions it expects.

For security reasons, most permissions required by the operator have to be namespace-scoped in our clusters.

Can functionality be added to support scoping the operator's actions to specific Kubernetes namespaces?

This is the current RBAC configuration given to our spicedb-operator service account.

ClusterRole

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: spicedb-operator
rules:
- apiGroups: [authzed.com]
  resources: [spicedbclusters]
  verbs: [create, delete, get, list, patch, update, watch]
- apiGroups: [authzed.com]
  resources: [spicedbclusters/status]
  verbs:  [create, delete, get, list, patch, update, watch]

Role (limited to spicedb-auth namespace)

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: spicedb-auth
  name: spicedb-operator-local
rules:
- apiGroups: [""]
  resources: [endpoints]
  verbs: [get, list, watch]
- apiGroups: [""]
  resources: [events]
  verbs: [create, delete, get, list, patch, watch]
- apiGroups: [""]
  resources: [pods]
  verbs: [delete, get, list, watch]
- apiGroups: [""]
  resources: [secrets]
  verbs: [create, delete, get, list, patch, update, watch]
- apiGroups: [""]
  resources: [serviceaccounts]
  verbs: [create, delete, get, list, patch, update, watch]
- apiGroups: [""]
  resources: [services]
  verbs: [create, delete, get, list, patch, update, watch]
- apiGroups: [apps]
  resources: [deployments]
  verbs: [create, delete, get, list, patch, update, watch]
- apiGroups: [batch]
  resources: [jobs]
  verbs: [create, delete, get, list, patch, update, watch]
- apiGroups: [rbac.authorization.k8s.io]
  resources: [rolebindings]
  verbs: [create, delete, get, list, patch, update, watch]
- apiGroups: [rbac.authorization.k8s.io]
  resources: [roles]
  verbs: [create, delete, get, list, patch, update, watch]

Error logs from the spicedb-operator container

I0216 15:58:45.534146       1 merged_client_builder.go:121] Using in-cluster configuration
I0216 15:58:45.535526       1 reflector.go:289] Starting reflector authzed.com/v1alpha1, Resource=spicedbclusters (0s) from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.535623       1 reflector.go:325] Listing and watching authzed.com/v1alpha1, Resource=spicedbclusters from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.535672       1 reflector.go:289] Starting reflector /v1, Resource=serviceaccounts (0s) from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.535705       1 reflector.go:325] Listing and watching /v1, Resource=serviceaccounts from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.535550       1 reflector.go:289] Starting reflector rbac.authorization.k8s.io/v1, Resource=roles (0s) from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.535637       1 reflector.go:289] Starting reflector /v1, Resource=secrets (0s) from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.536096       1 reflector.go:325] Listing and watching /v1, Resource=secrets from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.536159       1 reflector.go:325] Listing and watching rbac.authorization.k8s.io/v1, Resource=roles from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.536222       1 reflector.go:289] Starting reflector apps/v1, Resource=deployments (0s) from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.536441       1 reflector.go:325] Listing and watching apps/v1, Resource=deployments from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.536034       1 reflector.go:289] Starting reflector /v1, Resource=pods (0s) from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.536491       1 reflector.go:325] Listing and watching /v1, Resource=pods from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.536067       1 reflector.go:289] Starting reflector batch/v1, Resource=jobs (0s) from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.536721       1 reflector.go:325] Listing and watching batch/v1, Resource=jobs from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.535997       1 reflector.go:289] Starting reflector /v1, Resource=services (0s) from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.536863       1 reflector.go:325] Listing and watching /v1, Resource=services from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.535970       1 reflector.go:289] Starting reflector rbac.authorization.k8s.io/v1, Resource=rolebindings (0s) from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
I0216 15:58:45.536964       1 reflector.go:325] Listing and watching rbac.authorization.k8s.io/v1, Resource=rolebindings from pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229
W0216 15:58:45.545856       1 reflector.go:535] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: failed to list authzed.com/v1alpha1, Resource=spicedbclusters: the server could not find the requested resource
E0216 15:58:45.546021       1 reflector.go:147] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: Failed to watch authzed.com/v1alpha1, Resource=spicedbclusters: failed to list authzed.com/v1alpha1, Resource=spicedbclusters: the server could not find the requested resource
W0216 15:58:45.546132       1 reflector.go:535] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: failed to list /v1, Resource=pods: pods is forbidden: User "system:serviceaccount:spicedb-operator:spicedb-operator" cannot list resource "pods" in API group "" at the cluster scope
W0216 15:58:45.546158       1 reflector.go:535] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: failed to list /v1, Resource=services: services is forbidden: User "system:serviceaccount:spicedb-operator:spicedb-operator" cannot list resource "services" in API group "" at the cluster scope
E0216 15:58:45.546176       1 reflector.go:147] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: Failed to watch /v1, Resource=pods: failed to list /v1, Resource=pods: pods is forbidden: User "system:serviceaccount:spicedb-operator:spicedb-operator" cannot list resource "pods" in API group "" at the cluster scope
W0216 15:58:45.546142       1 reflector.go:535] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: failed to list /v1, Resource=serviceaccounts: serviceaccounts is forbidden: User "system:serviceaccount:spicedb-operator:spicedb-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope
W0216 15:58:45.546182       1 reflector.go:535] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: failed to list /v1, Resource=secrets: secrets is forbidden: User "system:serviceaccount:spicedb-operator:spicedb-operator" cannot list resource "secrets" in API group "" at the cluster scope
E0216 15:58:45.546201       1 reflector.go:147] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: Failed to watch /v1, Resource=serviceaccounts: failed to list /v1, Resource=serviceaccounts: serviceaccounts is forbidden: User "system:serviceaccount:spicedb-operator:spicedb-operator" cannot list resource "serviceaccounts" in API group "" at the cluster scope
E0216 15:58:45.546200       1 reflector.go:147] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: Failed to watch /v1, Resource=services: failed to list /v1, Resource=services: services is forbidden: User "system:serviceaccount:spicedb-operator:spicedb-operator" cannot list resource "services" in API group "" at the cluster scope
E0216 15:58:45.546215       1 reflector.go:147] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: Failed to watch /v1, Resource=secrets: failed to list /v1, Resource=secrets: secrets is forbidden: User "system:serviceaccount:spicedb-operator:spicedb-operator" cannot list resource "secrets" in API group "" at the cluster scope
W0216 15:58:45.546508       1 reflector.go:535] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: failed to list rbac.authorization.k8s.io/v1, Resource=rolebindings: rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:spicedb-operator:spicedb-operator" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
E0216 15:58:45.546558       1 reflector.go:147] pkg/mod/github.com/ecordell/[email protected]/tools/cache/reflector.go:229: Failed to watch rbac.authorization.k8s.io/v1, Resource=rolebindings: failed to list rbac.authorization.k8s.io/v1, Resource=rolebindings: rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:spicedb-operator:spicedb-operator" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
...
I0216 15:59:18.010153       1 shared_informer.go:337] stop requested
I0216 15:59:18.010233       1 shared_informer.go:337] stop requested
I0216 15:59:18.010245       1 shared_informer.go:337] stop requested
I0216 15:59:18.010254       1 shared_informer.go:337] stop requested
I0216 15:59:18.010263       1 shared_informer.go:337] stop requested
I0216 15:59:18.010272       1 shared_informer.go:337] stop requested
I0216 15:59:18.010283       1 shared_informer.go:337] stop requested
I0216 15:59:18.010289       1 shared_informer.go:337] stop requested
I0216 15:59:18.010295       1 shared_informer.go:337] stop requested
I0216 15:59:18.010305       1 shared_informer.go:341] caches populated
error: context canceled

[joshrosso]

I'm very interested in this capability, I'm also at a place where providing spicedb-operator cluster-wide access is problematic. In my case, I really only need it tied to a single namespace.

I'd be interested in doing the work, if the following proposal made sense:

Proposal

Introduce --watch-namespaces argument to the spicedb-operator cmd. It can support a list of namespaces, separated by commas.

When present, the operator will only look for relevant CRDs in those namespaces. The SpiceDB cluster itself will be created in the same namespace as the CRD. Rolebindings (namespace scoped) will need to be setup by the human(s) accordingly.

To ensure backwards compatibility, the absence of --watch-namespace (default) will mean the operator continues to work at a cluster-wide level.

[ecordell]

Hi @joshrosso, thanks for the interest! I think your proposal will work fine (it's how most operators do it), and I'm okay with doing that for now if we need to.

But I've always thought the UX would be much better if we did something like this:

• The operator always retains read/write access to the SpiceDBCluster object at the cluster scope (i.e via a ClusterRole/ClusterRoleBinding), we'll call it spicedbcluster-rw.
• We create a ClusterRole for managing SpiceDB-related resources in a namespace; i.e. deployment create, service create. Let's call this manage-spicedb

In the default (all namespace) installation, we'll create a ClusterRoleBinding to manage-spicedb. Then the operator can do its work in any namespace.

If you want to let the operator work in just one namespace, i.e. my-spicedbs, then you could instead create a RoleBinding to manage-spicedb in my-spicedbs for the operator's service account. Now the operator has permission to create deployments only in my-spicedbs.

Similarly, if you have dev-spicedbs, staging-spicedbs, and prod-spicedbs namespaces, you can create one RoleBinding per namespace to grant the operator permission to work.

The nice thing about this model is that, because the operator retains R/W control on the SpiceDBCluster object in the entire cluster, it will be able to do things like write a status message in the object when permission is insufficient: Error: Operator does not have permission to list Deployments in namespace foo, please grant etc etc..., with explicit instructions for users to follow (if they have sufficient permission to perform them).

There's two main issues with this:

• Starting the appropriate watches at the right time. On start, the operator should try to watch all required resources in all namespaces, but shouldn't fail if it can't. Then, it should try and start a watch on any namespaces that contain SpiceDBCluster objects as they are created, and either report an error on the object or start reconciling. We don't start and stop watches dynamically like this in the spicedb-operator right now, though https://github.com/authzed/controller-idioms supports it (and we use it in some internal controllers at authzed).
• Detecting when permission has changed. The operator won't be able to watch for changes to its own permission (or at least, this doesn't seem like a reasonable thing to add). But I think on the whole that's not a big deal; we can exponentially backoff attempts to create new watches, and if you need to re-trigger faster than that you can edit the object or delete/recreate.

We could also think about doing this type of thing for some api calls; i.e. maybe you create an operator that can't create Jobs - so it can run spicedb but will error if it tries to run a migration. There might be some cases where this is useful; if you need to pre-provision serviceaccounts or something and don't want the operator to have permission to create them at all.

With respect to your proposal, this could be a "phase2" and part of the default behavior if the namespace list is omitted?

[joshrosso]

@ecordell All the above makes sense to me and I like the UX you proposed.

If you're cool with the 2 phase approach, as you mentioned:

With respect to your proposal, this could be a "phase2" and part of the default behavior if the namespace list is omitted?

I'm on board. I'll try to work through an implementation and PR this week for phase 1.

[ecordell]

The base support is available in 1.17.0, and i've filed the other idea as a separate issue #341