Add support for change tracking on table creation

[momirza]

Related to #343

Overview

SNOW-XXXXX

Pre-review checklist

• This change should be part of a Behavior Change Release. See go/behavior-change.
• This change has passed Merge gate tests
• Snowpipe Changes
• Snowpipe Streaming Changes
• This change is TEST-ONLY
• This change is README/Javadocs only
• This change is protected by a config parameter snowflake.enable.change.tracking.
  • Yes - Added end to end and Unit Tests.
  • No - Suggest why it is not param protected
• Is his change protected by parameter <PARAMETER_NAME> on the server side?
  • The parameter/feature is not yet active in production (partial rollout or PrPr, see Changes for Unreleased Features and Fixes).
  • If there is an issue, it can be safely mitigated by turning the parameter off. This is also verified by a test (See go/ppp).

---

Email for CLA: mo.mirza@iwoca.co.uk

[momirza]

@sfc-gh-lkucharski could you please give this a review?

[sfc-gh-lkucharski]

@sfc-gh-lkucharski could you please give this a review?

Hi, thank you for the contribution.
Can you resolve conflicts and add tests?

A few things to keep in mind

• master branch has 4.0.0 PrPr version of the connector. Many features have been removed from that version. It supports Snowpipe Streaming High Performance architecture only.
• Release of 4.0.0 rc2 is planned for next week.
• Pre 4.x versions are maintained in 3.2.x branch. If you're using pre 4.0.0 version of the connector, you probably want to change this pr to merge to 3.2.x branch

[momirza]

@sfc-gh-lkucharski could you please give this a review?

Hi, thank you for the contribution. Can you resolve conflicts and add tests?

A few things to keep in mind

• master branch has 4.0.0 PrPr version of the connector. Many features have been removed from that version. It supports Snowpipe Streaming High Performance architecture only.
• Release of 4.0.0 rc2 is planned for next week.
• Pre 4.x versions are maintained in 3.2.x branch. If you're using pre 4.0.0 version of the connector, you probably want to change this pr to merge to 3.2.x branch

Resolved conflicts, applied the suggested change.

We're happy for this to go into 4.X as we're already using snowflake.ingestion.method: SNOWPIPE_STREAMING in our 3.X connector. Can upgrade to 4.X whenever that's ready.

[sfc-gh-lkucharski]

@sfc-gh-lkucharski could you please give this a review?

I'll take a look tomorrow

[momirza]

@sfc-gh-lkucharski did you get the chance to review this? Just resolved the more recent merge conflicts.

I don't have the credentials on my fork for this repo to run all the tests so would be great to get this branch pulled into the main repo to run the tests in a pull request from there.

[sfc-gh-lkucharski]

@sfc-gh-lkucharski did you get the chance to review this? Just resolved the more recent merge conflicts.

I don't have the credentials on my fork for this repo to run all the tests so would be great to get this branch pulled into the main repo to run the tests in a pull request from there.

Thank you again for this effort. Can you explain what you mean by "pulled into the main repo"?. Your pull request looks like a nice to have feature. Currently, I think it's the worst possible time to work on this, as we are undergoing major refactoring on master for version 4 of the connector. We are rearranging classes and deleting content. Currently, the master doesn't even support automatic table creation. Therefore, this feature is not functional temporarily. Do you think we can revisit this in two weeks? I think internally I have like four pull requests waiting to be merged. I wouldn't want you to resolve conflicts on every PR we merge. I would gladly let you know if the code gets stable enough so that you won't waste your time resolving conflicts all the time. What do you think?

[momirza]

Can you explain what you mean by "pulled into the main repo"?

A bunch of Checks failed with the error

Run ./.github/scripts/decrypt_secret.sh AWS
gpg: gcry_kdf_derive failed: Invalid data
gpg: decryption failed: Bad session key

It just means that my fork does not have the credentials needed to run them. This is expected and not a problem on my side. Just allows the checks to run before this can be merged.

I would gladly let you know if the code gets stable enough so that you won't waste your time resolving conflicts all the time. What do you think?

Yes, that's totally fine. Just let me know when your changes are stable, and I'll update this pull request to resolve any conflicts.

[sfc-gh-lkucharski]

Hi @momirza, hope you're doing well. I'm just letting you know that the master branch is now pretty stable. If you still need that feature, please resolve the conflicts and fix the tests. Best, Lukas

[momirza]

Hi @momirza, hope you're doing well. I'm just letting you know that the master branch is now pretty stable. If you still need that feature, please resolve the conflicts and fix the tests. Best, Lukas

Great thanks! I'll have a look later next week :)