Allow configuration of refresh_on_create for Snowflake External Tables on AWS #348
Description
Describe the feature
There's no way to set refresh_on_create to false. This property is useful to set for extremely large number of files in S3 paths that would breach the limits for auto refresh with this error.
Encountered an error while running operation: Database Error
001057 (0A000): Total size (>=1,073,742,112 bytes) for the list of file descriptors returned from the stage exceeded limit (1,073,741,824 bytes); Number of file descriptors returned is >=3,273,604. Please use a prefix in the stage location or pattern option to reduce the number of files.
Describe alternatives you've considered
Not using this package and using DDL. Other alternative is of course to define external tables on more specific prefixes so as not to breach the file listing limits.
Additional context
This feature is Snowflake on AWS specific.
Who will this benefit?
Anyone having large external tables being moved to dbt.