[Kubernetes Operator] cluster Role / Rolebinding

[poblin-orange]

Expecting the toolhive operator to detect MCPServer CR in any ns (or be able to configure the scanned ns). Seems only Role and RoleBinding are provisionned by the chart, preventing my use case of multiple ns.

Resulting log on my cluster (k3s 1.30, toolhive operator toolhive-operator-0.0.7 / 0.0.38 )

0530 21:35:11.030191       1 leaderelection.go:257] attempting to acquire leader lease 19-toolhive-operator/toolhive-operator-leader-election...                                                                                                                                                                           
I0530 21:35:28.884340       1 leaderelection.go:271] successfully acquired lease 19-toolhive-operator/toolhive-operator-leader-election                                                                                                                                                                                     
9:35PM INF toolhive-operator-5d6c7cbfbd-bzbqq_7b3f1856-ab4f-4dcb-bbbf-84eeb475aab2 became leader component=events type=Normal object="{Kind:Lease Namespace:19-toolhive-operator Name:toolhive-operator-leader-election UID:408c3d59-ce79-465a-9caa-fa3f8b53d229 APIVersion:coordination.k8s.io/v1 ResourceVersion:165805445
9:35PM INF Starting EventSource controller=mcpserver controllerGroup=toolhive.stacklok.dev controllerKind=MCPServer source="kind source: *v1.Service"                                                                                                                                                                       
9:35PM INF Starting EventSource controller=mcpserver controllerGroup=toolhive.stacklok.dev controllerKind=MCPServer source="kind source: *v1alpha1.MCPServer"                                                                                                                                                               
9:35PM INF Starting EventSource controller=mcpserver controllerGroup=toolhive.stacklok.dev controllerKind=MCPServer source="kind source: *v1.Deployment"                                                                                                                                                                    
E0530 21:35:28.923725       1 reflector.go:200] "Failed to watch" err="failed to list *v1.Deployment: deployments.apps is forbidden: User \"system:serviceaccount:19-toolhive-operator:toolhive-operator\" cannot list resource \"deployments\" in API group \"apps\" at the cluster scope" logger="UnhandledError" reflecto
E0530 21:35:28.923875       1 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:19-toolhive-operator:toolhive-operator\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-g
E0530 21:35:28.924081       1 reflector.go:200] "Failed to watch" err="failed to list *v1alpha1.MCPServer: mcpservers.toolhive.stacklok.dev is forbidden: User \"system:serviceaccount:19-toolhive-operator:toolhive-operator\" cannot list resource \"mcpservers\" in API group \"toolhive.stacklok.dev\" at the cluster sc
E0530 21:35:29.811402       1 reflector.go:200] "Failed to watch" err="failed to list *v1alpha1.MCPServer: mcpservers.toolhive.stacklok.dev is forbidden: User \"system:serviceaccount:19-toolhive-operator:toolhive-operator\" cannot list resource \"mcpservers\" in API group \"toolhive.stacklok.dev\" at the cluster sc
E0530 21:35:30.091278       1 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:19-toolhive-operator:toolhive-operator\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-g
E0530 21:35:30.420624       1 reflector.go:200] "Failed to watch" err="failed to list *v1.Deployment: deployments.apps is forbidden: User \"system:serviceaccount:19-toolhive-operator:toolhive-operator\" cannot list resource \"deployments\" in API group \"apps\" at the cluster scope" logger="UnhandledError" reflecto

[ChrisJBurns]

@poblin-orange Do you have the full manifests you used to get the error you got above? I am looking into this now as multi-ns is something we want to immediately support

[ChrisJBurns]

@poblin-orange I've got it to work with the following additional manifests in the test namespace. I should caveat, the below is copied from running a helm template toolhive deploy/charts/operator and just changing the namespace for the toolhive ServiceAccount, Role and RoleBinding.

However, I will have a think about what the best way of doing this automatically is. Whether its the below shoved into the Helm Chart, or whether we create them with the Operator. There's pros and cons for each - shall have a think. For now, you should be able to just copy and paste the below and apply them into your custom namespace.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: toolhive-binding
  namespace: test
  labels:
    app: toolhive
    app.kubernetes.io/name: toolhive
subjects:
- kind: ServiceAccount
  name: toolhive
roleRef:
  kind: Role
  name: toolhive
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: toolhive
  namespace: test
  labels:
    app: toolhive
    app.kubernetes.io/name: toolhive
rules:
  - apiGroups:
    - apps
    resources:
    - statefulsets
    verbs:
    - get
    - list
    - watch
    - create
    - update
    - patch
    - delete
    - apply
  - apiGroups:
    - ""
    resources:
    - services
    verbs:
    - get
    - list
    - watch
    - create
    - update
    - patch
    - delete
    - apply
  - apiGroups:
    - ""
    resources:
    - pods
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - pods/log
    verbs:
    - get
  - apiGroups:
    - ""
    resources:
    - pods/attach
    verbs:
    - create
    - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: toolhive
  namespace: test
  labels:
    app: toolhive
    app.kubernetes.io/name: toolhive

Let me know if it works, and I'll try and get an automated fix up that does all of this for you.

[ChrisJBurns]

Have created http://github.com/stacklok/toolhive/issues/538 to track the implementation of this. I'm leaning more towards the Operator doing this as it allows for us to simplify the Helm Charts and also reduce the requirement for users having to specify information about additional namespaces.

[poblin-orange]

@poblin-orange Do you have the full manifests you used to get the error you got above? I am looking into this now as multi-ns is something we want to immediately support

@ChrisJBurns
Here the manifest. Im using fluxcd, so its just plain chart install. I chose a custom ns

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    paas-templates.orange.com/protected: "true"
  name: 19-toolhive-operator
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  annotations:
    paas-templates-helm-source-url: https://github.com/stacklok/toolhive/tree/main/deploy/charts/operator
  name: toolhive-operator
  namespace: 19-toolhive-operator
spec:
  chart:
    spec:
      chart: toolhive-operator
      interval: 5m
      sourceRef:
        kind: HelmRepository
        name: toolhive-oci-helm-repo
        namespace: fluxcd
      version: 0.0.9
  interval: 10m
  values: {}

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  annotations:
    paas-templates-helm-source-url: https://github.com/coroot/helm-charts/tree/main/charts/coroot
  name: toolhive-operator-crds
  namespace: 19-toolhive-operator
spec:
  chart:
    spec:
      chart: toolhive-operator-crds
      interval: 5m
      sourceRef:
        kind: HelmRepository
        name: toolhive-oci-helm-repo
        namespace: fluxcd
      version: 0.0.3
  install:
    crds: Create
  interval: 10m
  upgrade:
    crds: CreateReplace
  values: {}

[poblin-orange]

@poblin-orange I've got it to work with the following additional manifests in the test namespace. I should caveat, the below is copied from running a helm template toolhive deploy/charts/operator and just changing the namespace for the toolhive ServiceAccount, Role and RoleBinding.

However, I will have a think about what the best way of doing this automatically is. Whether its the below shoved into the Helm Chart, or whether we create them with the Operator. There's pros and cons for each - shall have a think. For now, you should be able to just copy and paste the below and apply them into your custom namespace.

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: toolhive-binding
namespace: test
labels:
app: toolhive
app.kubernetes.io/name: toolhive
subjects:

• kind: ServiceAccount
  name: toolhive
  roleRef:
  kind: Role
  name: toolhive
  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: toolhive
namespace: test
labels:
app: toolhive
app.kubernetes.io/name: toolhive
rules:

• apiGroups:
  • apps
    resources:
  • statefulsets
    verbs:
  • get
  • list
  • watch
  • create
  • update
  • patch
  • delete
  • apply
• apiGroups:
  • ""
    resources:
  • services
    verbs:
  • get
  • list
  • watch
  • create
  • update
  • patch
  • delete
  • apply
• apiGroups:
  • ""
    resources:
  • pods
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • ""
    resources:
  • pods/log
    verbs:
  • get
• apiGroups:
  • ""
    resources:
  • pods/attach
    verbs:
  • create
  • get

---

apiVersion: v1
kind: ServiceAccount
metadata:
name: toolhive
namespace: test
labels:
app: toolhive
app.kubernetes.io/name: toolhive

Let me know if it works, and I'll try and get an automated fix up that does all of this for you.

thks @ChrisJBurns. ill give it a try.
Wrt to RBAC, i think other products do create the RBAC with operator helm charts.
Might avec a look at:

• https://github.com/strimzi/strimzi-kafka-operator/tree/main/helm-charts/helm3/strimzi-kafka-operator/templates