Skip to content

Make topic to table name mapping configurable #1066

@lipeicarta

Description

@lipeicarta

Hi team, we are having some issue and may need your suggestion, and if there is no workaround I want to suggest some changes.

For context:

  1. We have about ~200 topics which needs to ingest in Snowflake, they are all domain events so we have some existing system to structure the
    topic name by domain, eg com.abc.$domain.$subdomain.$version.$something which guarantee there is no topic name collission.
  2. For every new topic we use Terraform to create Snowflake table.
  3. Given the topic name has . in it, when we do not define topic2table config, Snowflake streaming connector will create table with hash suffix to avoid naming collision, this generated table name apparently is not the same as what we created through TF.
  4. Problem, to define the connector we cannot avoid defining topics and topic2table now since the constraint, and we have to define a lot.. also it also constraint us to use topics.prefix which automatically detect new topic.

I want to learn if there is existing use case like this and how people handle this. If possible, can we have a way to customize tablename generation function through an interface since all we need is replace . with underscore and uppercase all characters

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions