Easier way to include additional built-in plugins in a custom Conduit build

[hariso]

The problem
Currently, to include additional built-in plugins into a custom Conduit build, the source code needs to be changed. That makes it more complicated to pull in new Conduit changes (revert changes to the list of built-in connector, pull new Conduit changes, add new built-in connectors again, build Conduit).

The 'why'

Built-in plugins are faster than standalone plugins, so there's a strong motivation to include a plugin in Conduit's binary. I'm also assuming that the connectors used for a specific Conduit instance are not going to change that frequently. In those cases, where the connectors used are all/most of the time the same, it makes sense to have a custom build with those connectors as built-in plugins.

An example

The list of built-in plugins can be made "external", i.e. in a separate file and then included during the build process, for example:

/path/to/plugins.go

import (
   _ github.com/conduitio-labs/conduit-connector-foo
)

Then, something like make build-with-plugins /path/to/plugins.go would build Conduit with conduit-connector-foo as a built-in plugin.

For now, I think this discussion can focus on the usefulness of such a feature. If we agree that having it is useful, we can discuss the implementation.

[lovromazgon]

I don't entirely understand the problem. Do you mean that it's hard to exclude built-in plugins? Or are we trying to make it easier to include additional plugins? Or is it updating built-in plugins themselves? Essentially, I'm not sure what "pull in new Conduit changes" means and what we're improving.

[lovromazgon]

Ok, so if I understand correctly, that can be done using the ConnectorPlugins field, something like this:

package main

import (
	"github.com/conduitio/conduit/pkg/conduit"
	"github.com/conduitio-labs/conduit-connector-foo"
)

func main() {
	// get the default config including all builtin plugins
	cfg := conduit.DefaultConfig()
	
	// add additional builtin plugins here
	cfg.ConnectorPlugins["foo"] = foo.Connector
	
	conduit.Serve(cfg)
}

Pulling new changes can be done by updating the conduit dependency, no need to change the source code.

[hariso]

For the embedded mode, yes. What about a custom build though?

[lovromazgon]

Define custom build please, I thought that's what I did above 😄 The default Conduit entrypoint doesn't do anything else, so with the above code you get exactly the same behavior as Conduit + one additional builtin connector plugin.

[hariso]

Oh, I see now, I read it too quickly.:) I was looking for flag parsing and similar, I missed that it's in Serve. This looks good, I think I'll something in the docs.

[lovromazgon]

We could definitely add a doc entry for how to build a custom conduit instance with additional builtin plugins.