Option to enable CHANGE_TRACKING on automatically created tables

[mrjoycee]

I would like to attach a stream to tables created and owned by the Kafka connector. Doing this requires change tracking to be enabled on the table, however our security model makes enabling this difficult as we would have to allow users to assume the role that is being used by the Kafka connector. It would be great if there was a configurable option in the connector for creating tables with change tracking enabled.

[sfc-gh-mwies]

@sfc-gh-japatel Do you have any thoughts on this?

[adrianisk]

We're finally moving from an S3 connector + lambda that creates snow pipes to the official Snowflake connector, and would also find this useful. We're going to have to add some custom code/security configuration to get this enabled on tables created by the connector, it would be great if it could just optionally do it for us

[sfc-gh-japatel]

Yes, this should be possible to do it.

[mrjoycee]

@sfc-gh-japatel If you're open to community contributions I've got the code for this working in a fork. We've been using it in production for a while now.

[sfc-gh-japatel]

Hey yes sure, that would be awesome! Can you open the PR in this repo? For first time contributors, you will have to sign CLA which I can ask our legal department to send to an email address.

Thanks in advance!

[lloydbranch]

@sfc-gh-japatel If you're open to community contributions I've got the code for this working in a fork. We've been using it in production for a while now.

@mrjoycee - Is there an open PR for the proposed contribution? We're currently leveraging a custom procedure and task to accomplish this and would rather not have that additional overhead if we can avoid it : ).

[sfc-gh-rcheng]

Closing this issue out due to age - please reopen if @mrjoycee has a PR or if further discussion is needed.

[sfc-gh-japatel]

FYI, we will soon have an option to enable change tracking in your landing table. Stay tuned!

[sfc-gh-japatel]

@convoy-lb @mrjoycee feel free to open a PR and we will take a look, FYI, we would want to have it behind a parameter and not enabled by default for all tables.

[momirza]

@convoy-lb @mrjoycee feel free to open a PR and we will take a look, FYI, we would want to have it behind a parameter and not enabled by default for all tables.

@sfc-gh-japatel I've created a PR to add this here: #1212

Please let me know if you require further changes. Happy to sign the CLA if needed.