Different Authentication/Authorization for Different Use Cases (multiple listeners?)

[forsberg]

Feature or Problem Description

Authentication/Authorization is always a tricky subject. I'm trying to wrap how Apicurio supports it around the use cases I see.

So let me describe my use cases, and how I currently use Confluent Registry + some nginx trickery to solve it.

Background

We have a Kubernetes cluster that runs, among other things, Kafka, the Confluent Schema Registry and a bunch of streaming applications.

We would like to migrate to Apicurio as it has a UI and supports more API types (specifically, AsyncAPI).

CI/CD API access to registry, from gitlab, verifying and registering schemas during build-time

Gitlab authenticates Vault to via gitlab's built in JWT issuer and gets a short-lived TLS certificate, which it then uses to authenticate to the nginx ingress that sits in front of Confluent Registry. I.e, I have configured in vault that some gitlab jobs, if they match the right set of claims, are allowed to retrieve this certificate, and hence, gain R/W access to the registry.

I guess with Apicurio, the idea is to use Keycloak, but I don't quite get how I would authenticate my gitlab jobs to Keycloak? I'm toying with the idea of using vault to retrieve a certificate that can authenticate to Keycloak's mTLS feature and then get access to an OIDC with the right roles to talk to Apicurio, but it feels... complicated.

Human Interface to Registry, mostly for read-only access.

We use nginx with some rules that forbid POST/PUT (except for one of the endpoints which will only verify schemas, not write them) in front of the registry, and then oauth2proxy for integration with Google Workspace OIDC. This provides developers with a read-only view of the schemas, which is fine for most purposes.

Here I can clearly see how Keycloak with federation can solve the problem.

Read-Only access to schema API from applications inside Kubernetes cluster

For streaming applications that run inside our Kubernetes cluster, the only requirement is read-only access to the Confluent REST API, as all schema registration happens at CI/CD time. For this, we use the same readonly nginx rules as above, and add some NetworkPolicy to restrict which applications is allowed to talk with the registry API. This has been deemed good enough for our use case, even though it could potentially leak some information about which schemas exist, to applications that should not have this information.

I have a hard time figuring out how I would implement this in a Keycloak-enabled Apicurio setup. Perhaps I'm missing some puzzle piece here in what the idea is on how to authenticate applications running Kubernetes to Keycloak so they can get an OIDC token and retrieve their schemas? But I also think this may be overkill and a bit too complicated for many installations.

Proposed Solution

One possible solution is to support multiple listeners for Apicurio Registry, where each listener have its own authentication access, a bit like Kafka Does it. To support my use cases above, I could then configure three different listeners:

• One with Mutual TLS authentication for CI/CD access. How to authorize access to different subjects/groups is left as an exercise for the reader, but could be based on attributes in the certificate. Alternatively, this use case would be supported by via mTLS in Keycloak.
• One protected by Keycloak, for human access.
• One with read-only access to all schemas. For intra-cluster communication this doesn't even necessarily have to be encrypted.

Not saying this is the optimal solution, mostly I'm after some guidance on how to support my use cases in the best possible way. As it is now, I'm pondering doing something similar to what I have now, i.e nginx authentication in front of an unauthenticated keycloak instance.

[EricWittmann]

Hey thanks for the question and sorry for the delay. I feel like I'm perpetually behind in responding to issues. Right at the top I will say that our authorization model is currently pretty naive. So handling more advanced use cases like what you describe is difficult for us at the moment. However, we (Red Hat) have a prototype of using Authorino in front of Registry, which isn't perfect (currently) but offers a ton of interesting options to cover advanced authn/z use cases. Authorino can be found here:

https://github.com/Kuadrant/authorino

I would need to summon @guicassolato to contribute to this conversation. :)

CI/CD API access to registry, from gitlab, verifying and registering schemas during build-time

Gitlab authenticates Vault to via gitlab's built in JWT issuer and gets a short-lived TLS certificate, which it then uses to authenticate to the nginx ingress that sits in front of Confluent Registry. I.e, I have configured in vault that some gitlab jobs, if they match the right set of claims, are allowed to retrieve this certificate, and hence, gain R/W access to the registry.

I guess with Apicurio, the idea is to use Keycloak, but I don't quite get how I would authenticate my gitlab jobs to Keycloak? I'm toying with the idea of using vault to retrieve a certificate that can authenticate to Keycloak's mTLS feature and then get access to an OIDC with the right roles to talk to Apicurio, but it feels... complicated.

Conventionally I think we would expect service account credentials to be stored in vault and then used with (I think) Oauth client credentials flow to obtain an access token. That token (JWT) would then be used in REST API calls to Apicurio Registry. I suspect anything more sophisticated than that would require something like Authorino.

Human Interface to Registry, mostly for read-only access.

We use nginx with some rules that forbid POST/PUT (except for one of the endpoints which will only verify schemas, not write them) in front of the registry, and then oauth2proxy for integration with Google Workspace OIDC. This provides developers with a read-only view of the schemas, which is fine for most purposes.

Here I can clearly see how Keycloak with federation can solve the problem.

This should be reasonably straightforward - users must exist in Keycloak (federated or configured via e.g. ldap or configured via an Identity Provider, perhaps even GitLab). A user authenticates via Keycloak, is issued a JWT, and then uses the UI.

I'm not sure how this would work when using an Authorino based solution.

Read-Only access to schema API from applications inside Kubernetes cluster

For streaming applications that run inside our Kubernetes cluster, the only requirement is read-only access to the Confluent REST API, as all schema registration happens at CI/CD time. For this, we use the same readonly nginx rules as above, and add some NetworkPolicy to restrict which applications is allowed to talk with the registry API. This has been deemed good enough for our use case, even though it could potentially leak some information about which schemas exist, to applications that should not have this information.

We typically expect streaming applications to use a client credentials OAuth flow to access the registry. This is, again, a service account for each application. Our SerDes classes support this approach out of the box, but we realize the Confluent's tools do not. With the standard Keycloak setup I think this is the best we can do. With an Authorino approach there are likely all manner of possibilities.

I have a hard time figuring out how I would implement this in a Keycloak-enabled Apicurio setup. Perhaps I'm missing some puzzle piece here in what the idea is on how to authenticate applications running Kubernetes to Keycloak so they can get an OIDC token and retrieve their schemas? But I also think this may be overkill and a bit too complicated for many installations.

We do also support a BASIC auth flow, which we do not recommend. It is what Confluent tooling often uses, however.

Proposed Solution

One possible solution is to support multiple listeners for Apicurio Registry, where each listener have its own authentication access, a bit like Kafka Does it. To support my use cases above, I could then configure three different listeners:

Can you clarify what you mean by "listener" in this context?

* One with Mutual TLS authentication for CI/CD access. How to authorize access to different subjects/groups is left as an exercise for the reader, but could be based on attributes in the certificate. Alternatively, this use case would be supported by via mTLS in Keycloak.

Side note: Apicurio Registry does not (currently) support fine-grained access based on subjects or groups. However, Authorino certainly could.

* One protected by Keycloak, for human access.

* One with read-only access to all schemas. For intra-cluster communication this doesn't even necessarily have to be encrypted.

Existing Apicurio Registry server configuration supports an optional "read-only" access level for both authenticated an anonymous access. For reference, here are some of the configuration settings that can be enabled in an Apicurio Registry instance:

That is missing the optional "Authenticated Read Access" setting (it's not available in the environment I used to take the screenshot).

Not saying this is the optimal solution, mostly I'm after some guidance on how to support my use cases in the best possible way. As it is now, I'm pondering doing something similar to what I have now, i.e nginx authentication in front of an unauthenticated keycloak instance.

Note: it's definitely always an option to use external authn/z in front of Apicurio Registry. This can work well, but there are idiosyncrasies for sure. We'd be happy to help figure it out.

To get Authorino specifically working in front of the registry might take a small number of relatively minor changes to the Registry itself to optimally support that configuration. We would definitely be happy to do that. We've wanted to work with Gui to go further with that configuration, but haven't had the upstream or downstream pressure to do it yet.

@guicassolato can you link to the Apicurio Registry + Authorino POC if it's available?

[guicassolato]

Hi @EricWittmann.

I would need to summon @guicassolato to contribute to this conversation. :)

Happy to help!

link to the Apicurio Registry + Authorino POC if it's available?

It sure is! https://github.com/guicassolato/apicurio-registry-authorino

@forsberg,

Please let me know if you need any help with the steps or with Authorino in general, and feel free to open issues in https://github.com/kuadrant/authorino as well.

[forsberg]

Thanks, I have no problem waiting a couple of weeks when the answers are this detailed.

Can you clarify what you mean by "listener" in this context?

Something like the Kafka Listeners, i.e the registry could be configured to listen to multiple IPs and/or ports, where each listener has separate security setup. For example, apicurio could listen to e.g. port 8440 which is exposed via ingress and requires auth via Apicurio, and then listen to port 8441 which exposes read-only access for access within cluster.

We typically expect streaming applications to use a client credentials OAuth flow to access the registry. This is, again, a service account for each application. Our SerDes classes support this approach out of the box, but we realize the Confluent's tools do not.

Are there any examples on how to do this? Looking for Python code examples in particular, but Java also of interest.

Will have a look at the authorino approach as well.

[carlesarnal]

Thanks, I have no problem waiting a couple of weeks when the answers are this detailed.

Can you clarify what you mean by "listener" in this context?

Something like the Kafka Listeners, i.e the registry could be configured to listen to multiple IPs and/or ports, where each listener has separate security setup. For example, apicurio could listen to e.g. port 8440 which is exposed via ingress and requires auth via Apicurio, and then listen to port 8441 which exposes read-only access for access within cluster.

Makes sense. That should be doable on the e.g. authorino side using different sets of rules. I would use Authorino as a "proxy" to the application and then apply those rules. Unless @guicassolato corrects me.

We typically expect streaming applications to use a client credentials OAuth flow to access the registry. This is, again, a service account for each application. Our SerDes classes support this approach out of the box, but we realize the Confluent's tools do not.

Are there any examples on how to do this? Looking for Python code examples in particular, but Java also of interest.

I'm sorry no. The closest example we might have is the Authorino PoC Gui linked above but still not exactly the use-case you might be looking for (although it might be extendable to cover it).

Will have a look at the authorino approach as well.

[guicassolato]

That should be doable on the e.g. authorino side using different sets of rules. I would use Authorino as a "proxy" to the application and then apply those rules.

This sounds quite reasonable to do with Authorino. You can enforce different policies differentiated by port number, or you can have it all within a same policy with different rules that are applicable based on some other aspect of the context (e.g. HTTP verb, path, user info, etc). For a reference of the latter, please check out http://kuadrant.io/docs/authorino/features.html#common-feature-conditions-when.

Are there any examples on how to do this? Looking for Python code examples in particular, but Java also of interest.

I'm sorry no. The closest example we might have is the Authorino PoC Gui linked above but still not exactly the use-case you might be looking for (although it might be extendable to cover it).

Will have a look at the authorino approach as well.

I don't know the use case in deep, but one option you may want to consider for the applications consuming the Registry within the cluster is using Kubernetes SA tokens as alternative authentication method. kublet gives some degree of configuration of how tokens are mounted ("projected") within the containers. For some use cases, it can be an alternative to OAuth2.

Validating such tokens with Authorino is also very simple, and you can write read-only access rules for those. Ref: http://kuadrant.io/docs/authorino/user-guides/kubernetes-tokenreview.html

[pecigonzalo]

Maybe to add to the conversation, I think it would be great to support other simpler authorization methods as well. For example:

• Read only and Read-write with basic auth (no RBAC, no Keycloak)
  • Use-case: Have systems be able to pull schemas on the fly, but only allow pushing/admin API actions with auth from CICD or similar
• JWT (No OIDC)
  • Use-case: Allow trusted 3rd parties to issue trusted JWT tokens, verifying against public keys (static or a link). This will fit the CICD use-case described here as well as other authentication proxies, eg. Teleport
• Trust headers from Auth proxy:
  • Use-case: Similar to the JWT use-case and quite simple to integrate. The Apicurious UI/API does not have to handle any authentication, it relies on the proxy.

I believe these 3 use-cases and implementations should be fairly simple to integrate, as they are part of quarkus, and provide enough flexibility to integrate any authentication source. The current Keycloak implementation is just too restrictive.

[EricWittmann]

The biggest problem with leveraging Quarkus functionality is that much of it is configured at build time rather than runtime. So to provide alternative authn/z configurations for Registry often requires multiple docker images (unless using Reaugmentation which brings in its own difficulties).

I think this is why we're recommending Authorino as a solution for authn/z use-cases that Registry doesn't natively support. I agree that all of the above use-cases are interesting - and I think that disabling authn/z features in Registry and then protecting it using Authorino is a good approach. Authorino should be able to support all of them and more.

It does mean that we have a few changes to make in Registry to support the more general "I want to use Authorino" use-case. I can't remember what the edge cases are that we identified when Gui did the POC, but there were a few minor issues.

[pecigonzalo]

The problem is that Authorino is again one implementation, which is inflexible and its not even a popular project at the moment.

Many of the Quarkus parameters can be set by env-vars at runtime, like you currently do for many of them. I dont see the issue you are mentioning.

[guicassolato]

@pecigonzalo,

Flexibility is one of Authorino’s main goal. If you think there’s something there we could do better, we could really learn more from your use case, provided it fits within our domain of course.

What exactly is the part it falls short for what you are trying to achieve? Perhaps there’s an RFE there we could investigate? (Even if you decide not to use it, of course!)

Thanks!

[pecigonzalo]

Thanks @guicassolato, its not about Authorino, its more about "what if I dont want to use Authorino?" eg. We use Teleport, it already does authentication, auditing, escalation, etc. It provides authentication as a JWT or header to backing services.

In other companies I have used https://www.ory.sh/, personally I now use https://www.authelia.com/ so why should we limit this integration to Authorino given the open standard or even conventions that most applications have?

FWIW, I do like the idea of Authorino and will try it, I also do like Keycloak. I just believe if we are looking to extend here, we shuold look at it beyond a single tool/integration.

[guicassolato]

@pecigonzalo ,

I totally understand. The solutions you named are good ones and you are right to evaluate your options thoroughly.

In case I failed to make it clear before, Authorino too is heavily based on open standards and technologies. To name a few: OpenId Connect, OAuth2, OPA, Kubernetes RBAC are all embedded modules we use, amongst others.

When/if the time comes for you to give Authorino a shot, please feel free to reach out. I'll be happy to help. Our open Slack channel is kuadrant.slack.com.

We also started gathering docs and additional context in our main project's website: kuadrant.io, which I definitely recommend checking out if you've ever heard of new Kubernetes Gateway API and are interested in an easy way of attaching auth policies directly to your HTTP routes and API gateways.

Cheers!