Write to multiple delta tables from a single topic

[Kikkon]

Referencing the design doc, I understand that if I want to write to multiple tables, I would need multiple processes and topics. This means I would need to manage the state of multiple topics & process.

However, since our company is relatively small, I think a single process would be sufficient for my needs.

So I would like to inquire about what the best practices are for this part? I think I could fork this repository and make some modifications, for example using a field in the topic as routing to write to different tables. But that feels a bit hack. I'm hoping for an approach more aligned with the current architecture. I'm happy to contribute back to the community if needed.

[ThijsKoot]

We're interested in this as well at our company.

[mightyshazam]

One way to keep it less hacky is to add the ability to filter messages after the transformation. This would still require you to run two processes, but it wouldn't require multiple topics.
The idea of one consumer delivering multiple places is a layer of complexity that probably exists above this project.

[leon-oosterwijk-akunacapital]

I have this working on a private fork. trying to figure out how to share this.

[Yevhenii-Kozlovskyi1]

Hi There! Has anyone had any luck implementing this approach, or encountered any blockers with this solution? I am interested in any implementation.
Our team is thinking of changing our Delta Lake architecture due to scaling, performance, isolation, and delta-sharing things.
One such approach is using a separate delta table per some input data characteristic which means our kafka-delta-ingest application should be able to recognize to which delta lake table the income message should be written based on some message properties.

[jasongoodwin]

Also interested in understanding the feasibility. Generally, it seems there isn't an easy way to sink from one topic into multiple tables based on message contents. Flink dynamic sink won't work as dynamic record isn't supported, this won't work, trino I don't think will work. Tempted to just try to contribute but @leon-oosterwijk-akunacapital says he's got a private fork already so I'd be interested to see what other people did before I take a crack at it.