gRPC Source/Destination

[ahmeroxa]

It would be useful to have generic gRPC Source/Destination pair.

The idea would be to use these connectors to bridge two Conduit server instances:

[hariso]

@ahmeroxa What would be the use case? I guess there's a reason why remote site B cannot access resource A directly.

[ahmeroxa]

Some use cases:

• Aggregation - The remote Conduit server can run a number of connectors talking to a number of resources over many different protocols. The same server will then push the data only via a secure MTLS gRPC connection to some where else (say in the cloud)
• Protocol Translation - It's possible to pull data from sources that cannot be reached directly by a hosted Conduit server. I have a Conduit connector that captures packages from the local NIC or integrates directly with a piece of hardware using proprietary drivers.

[lovromazgon]

I've got some questions and comments:

• What happens if the gRPC connection is unstable? I assume the bridge connector on site A should be fault tolerant to a degree.
• Given that Conduit is stateless and the only points where records are persisted are the standard resources on site A and B, we need to pass acknowledgments back through the bridge connector when a record reaches the standard resource destination. If we don't do this we risk losing data.
• Sounds like the bridge connector can use gRPC streams under the hood, or are there any concerns about that?
• Should one bridge connector on site B be able to receive data from multiple sites (A1, A2, ...)?
• Should one bridge connector on site A be able to transmit data to multiple sites (B1, B2, ...)?
  • If yes, we need to figure out when a record is regarded as "acked".
• Remote site A is the one that initiates the connection with remote site B, correct? In other words - A is the client, B is the server.
• How do we prevent unauthorized users from posting data to the bridge connector? mTLS?
• I'm thinking of the last position stored on remote site B. If that pipeline is restarted, should the last position there play a role? Should site B be able to request a record at a specific position from site A?

[hariso]

Or am I missing something? :D

[lovromazgon]

No, that's how I understand it as well. I wanted to make sure we're all on the same page so I called it out. Given that we expect the gRPC destination to send records to only one gRPC source we won't need the identification part you mentioned (i.e. consumer groups), the pipeline on site A would simply send over any records that have not yet been acked.

The only drawback is that if we want to reprocess records, it would need to be done on site A and not site B. Site B is "dumb" in the sense that it only receives whatever site A sends it, while site A keeps track of the position based on the acks it gets from site B.

[ahmeroxa]

• Remote site A is the one that initiates the connection with remote site B, correct? In other words - A is the client, B is the server.

In this case yes, but necessarily it doesn't matter which side is the client and which side is the destination. There may be cases where I would like the connection to be initiated on one side vs the other, but (I imagine) I would simply swap which side is the "source" and which is the "destination".

Not sure I understand. It won't be as easy as swapping the source and destination, since a Conduit source connector can only read records and a destination can only produce them. In other words, a destination can't really read a record and send it into the Conduit pipeline. So if we want a gRPC source (e.g. site B) to reach out and establish a connection with a gRPC destination (e.g. site A) that isn't really trivial.

What I meant here is that we can swap which site is the Source and which is the Destination:

Site A (Destination) ---> (Source) Site B
Site A (Source)      <--- (Destination) Site B

[lovromazgon]

@ahmeroxa if you swap the sides you reverse the flow of records, site B is then sending records to site A, site A is only sending acknowledgments back to site B.

[hariso]

@ahmeroxa if you swap the sides you reverse the flow of records, site B is then sending records to site A, site A is only sending acknowledgments back to site B.

i.e. if we implement the gRPC connector as a normal connectors, swapping the sides means that we need to change the pipeline structure. Whereas maybe Ali meant that the bridge would somehow be special and would allow for a simple switch to change the sides?

[hariso]

@ahmeroxa Another question: did you imagine this as generic gRPC connectors or would those connectors be specifically for Conduit?

[ahmeroxa]

Specifically for Conduit (at least at first).

We can potentially make it generic later.