Add additional state-check before loading transaction

[chuwy]

Right now snowflake loader grabs whole state from DynamoDB in the beginning and iterates through all not-yet-loaded folders. Between loader grabbed state and loaded all folders - another simultaneous loader can be launched which will lead to race condition. This should not be dangerous as Snowflake keeps information about loaded files to prevent double-loading them, but just in-case we would like to at least warn user that folder is already processed.

Logic:

for folder in not_yet_loaded:
  if manifest.contains(folder):
    error("Folder is already loaded skipping")
  else:
    load(folder)
    manifest.add(folder)

[alexanderdean]

Sounds sensible!

[alexanderdean]

This should not be dangerous as Snowflake keeps information about loaded files to prevent double-loading them

Didn't we confirm that this safety feature doesn't work as expected given how we stage data?

[chuwy]

Yes, it doesn't work anymore unfortunately. We realized it exactly after I submitted this issue.