doc: Add documentation on how to directly interact with the schema registry

[nickchomey]

The schema registry doc says

work directly with the schema registry (for example, in cases where automatic schema extraction isn't enough and a schema needs to be built manually using the information from a source)

But I dont really see much information on how to do that anywhere.

I see that there's a REST API for the Schema Registry

There is a schema.go package in the connector sdk and processor sdk. Is that we should use? It seems to have fewer functions than the REST API...

And the schema registry seems to use the franz sr package under the hood. Should that ever be used?

Some more docs on how to use it and especially with some real-world examples/use cases would be very helpful.

[nickchomey]

I now see that the REST API is largely incomplete (I didnt notice that they were checkboxes and most are unchecked) and probably reflects the Get and Create functions in the sdks.

[lovromazgon]

Yes, as you pointed out, the schema registry in Conduit is still incomplete. It only supports the functionality that's strictly required by Conduit, so that the user doesn't necessarily have to deploy another service to get the benefits that schemas bring.

The schema registry REST API is currently not exposed in Conduit at all, as it's missing a lot of important endpoints expected by a proper schema registry. Currently the only way to access schemas stored in the internal schema registry is through connectors or processors. However, you can configure an external schema registry like the Confluent schema registry or Apicurio and point Conduit to that service, then Conduit will store schemas there and you can use it with any other tool that knows how to work with a schema registry.

For now this means that we don't have much to document about the interaction with the schema registry, since it's not really exposed.