Multitenancy documentation

[darrellme]

Hi

Looking at your code, there appears to be some form of multi-tenancy support. Is this documented anywhere on how this works, and how to configure it

Thanks

Darrell Merryweather

[EricWittmann]

We don't really have formal documentation (yet) for the multi-tenancy feature of Apicurio Registry (I assume you are interested in Registry, not Studio).

I could write a quick getting-started document if you think that would help. But we currently don't make the MT capabilities of Registry available to Red Hat customers of the on-prem registry product, and as a result we don't have strong end user docs on it. (just a bandwidth/priority thing)

From a high level, it's not a complicated feature. Essentially we just have a "tenantId" column on all our DB tables. And all you need to do is make sure that you make the tenantId available to the registry application on each request (this can be done by including it in the path to the REST API, or by including it in the JWT token, or by passing it in via HTTP request header, or via a subdomain name).

However, things get tricky if you want to enforce appropriate authorization controls (e.g. the population of users is different for each tenant).

The real pitfall is the UI. The UI is not currently built to work with a multi-tenancy enabled Registry. When we use multi-tenant registry at Red Hat (for our managed service) we use federated webpack modules to export the UI components into our larger console.redhat.com console.

To get you started, here are some ENV variables you can enable when starting up the registry docker image:

REGISTRY_ENABLE_MULTITENANCY=true
REGISTRY_MULTITENANCY_TYPES_CONTEXT_PATH_ENABLED=true
REGISTRY_MULTITENANCY_TYPES_REQUEST_HEADER_ENABLED=true
REGISTRY_MULTITENANCY_TYPES_SUBDOMAIN_ENABLED=true
REGISTRY_MULTITENANCY_TYPES_SUBDOMAIN_PATTERN=(\w[\w\d\-]*)\.localhost\.local.*
TENANT_MANAGER_AUTH_ENABLED=false

So for example you could start like this:

docker run -it -p 8080:8080 --env REGISTRY_ENABLE_MULTITENANCY=true --env REGISTRY_MULTITENANCY_TYPES_CONTEXT_PATH_ENABLED=true --env REGISTRY_MULTITENANCY_TYPES_REQUEST_HEADER_ENABLED=true  --env REGISTRY_MULTITENANCY_TYPES_SUBDOMAIN_ENABLED=true    --env REGISTRY_MULTITENANCY_TYPES_SUBDOMAIN_PATTERN="(\w[\w\d\-]*)\.localhost\.local.*"   --env TENANT_MANAGER_AUTH_ENABLED=false   apicurio/apicurio-registry-mem:latest-release

That will enable MT mode and allow the tenantId to be sent either in the API path, or in a request header (configurable but defaults to X-Registry-Tenant-Id), or as TENANTID.localhost.local.

So you can try this, for example:

curl http://localhost:8080/t/12345/apis/registry/v2/system/info

That will try to access a tenant with ID "12345" and invoke the system info API endpoint.

However it will fail. The reason is that Registry itself doesn't manage the tenants - it relies on another component called the Tenant Manager to do that. So you would also need to run an instance of the tenant manager. The MT registry application will try to lookup information about a tenant in the tenant manager. And if it doesn't exist, it will fail the request. So you would need to start up the tenant manager and provision tenants using that tenant manager (via its REST API). Then you need to configure the tenant manager in the registry application so that registry can lookup tenants in the TM.

For a full collection of configuration options for multitenancy, you could look here:

https://github.com/Apicurio/apicurio-registry/blob/master/app/src/main/java/io/apicurio/registry/mt/MultitenancyProperties.java

Depending on your use-case, we could also look into streamlining some of this (e.g. maybe you don't need the TM?).

[darrellme]

Thanks for the reply.

You were correct, we were interested in the registry and not studio :-). So, we have managed to get the tenant manager working, and the tenant specific API's working. It took a little digging through the code (prior to your reply), but easy enough.

Now comes the UI part. Obviously the UI that comes with the registry is disable when multi-tenancy is turned on. We did find another UI, apicurio-registry-mt-ui, which appears as though it would work, but I was struggling to get this to work. Is this something that is still actively maintained, or not? Is there any likely hood of developing a multi-tenant aware UI to retrieve the artifacts in the future?

Thanks

Darrell

[EricWittmann]

First of all - nice work finding that multi-tenant UI. That can be considered a good starting point. But the bad news is we really don't YET have a standalone UI for registry in multi-tenant mode. The apicurio-registry-mt-ui repository is more of a demo or POC right now. I'll explain. But first here is an architecture for how the MT version of Apicurio Registry is currently working for Red Hat cloud:

So you'll notice a component in that diagram that we haven't talked about - the fleet manager. The idea here is that you might want multiple deployments of Multi-Tenant Apicurio Registry running in different regions, or perhaps just in different k8s clusters or something. Each deployment can hold N tenants. The fleet manager is responsible for managing the set of "instances" across all deployments.

The fleet manager is optional - only needed if you want the potential of multiple multi-tenant deployments. But without it, you will need to figure out a way to provision tenants in the Tenant Manager component.

About the UI

We don't currently have a good upstream solution for the UI when running in multi-tenant mode. The closest we have is that apicurio-registry-mt-ui repository, which is stale and built as more of a POC or even as a test-bed for the federated modules exported by Apicurio Registry. I would need to find some time to explore how much that repository has drifted to see if it still works. But the idea is that it presents a UI that interacts directly with the Tenant Manager API to manage the collection of tenants. And then it also allows users to click on a given tenant to display the standard Apicurio Registry UI for just that tenant.

In reality, the UI should be use-case driven. In Red Hat managed services (Red Hat OpenShift Service Registry), the above is basically what we do, except our UI interacts with the fleet manager API, which in turn invokes the Tenant Manager component to provision tenants. We do this because our managed service is self-service based. That might not be your use-case.

The bottom line is that we may need a standalone multi-tenant aware UI that has not yet been built.

The good news is that it wouldn't be very hard to build, since it would mostly just be a simple React application that imports the federated modules exported by Apicurio Registry. But we would need to discuss exactly what you're trying to achieve...

[pantaoran]

This discussion was very helpful to me just now. It allowed me to get basic multitenancy to work in a test environment. But I have important questions:

1. This pull request makes it look like multitenancy will be removed in v3 altogether. Am I reading this right? Why is this?
2. Assume I have a productive instance of Apicurio registry without multitenancy, and I wanted to introduce MT. Is there a migration path for existing schemas which were written before MT was turned on? Which tenant would they belong to? I'm looking at owner-based authorization with standalone multitenancy.

[EricWittmann]

1. You are correct that multi-tenancy is being removed in 3.0. The reason is we did not have many community users of this feature. And the Red hat offering that prompted this feature was decommissioned.

When we made this decision we thought that we could re-implement multitenancy better using the lessons we learned from the first round. Assuming it was a feature the community actually ever wanted.

2. My recommendation for migration from non-tenant to tenant, would be to export all data to zip file and then import that back into an empty multi-tenant installation of registry.

[jsenko]

I can add that there is a pretty extensive OpenShift template in our repository, that produces a multi-tenant Registry deployment https://github.com/Apicurio/apicurio-registry/blob/master/distro/openshift-template/mt/apicurio-registry-mt-template.yaml . It has some additional complexity (e.g. it uses Envoy proxies) and is missing some private configuration, but it can be a good place to take a look.

[darrellme]

Thanks, that will help us with a valid helm chart for deploying to K8s (non-open shift)