feature: clean up batch files

[rafalkrupinski]

Feature scope

Other

Description

Since taps and targets are responsible only for handling data and connections to their respective services, it seems that it should fall on meltano to clean up any batch files produced by the pipelines.

Only local FS and S3 is supported for storage of files, both of which are already supported by meltano.

This feature should probably be opt-in, configurable per environment.

Currently this is handled by some targets, but it's not supported by the SDK.

[rafalkrupinski]

I just realized that when running meltano from kubernetes cron job, it gets a ephemeral disk space, so effectively all files are temporary.

I don't know about other use cases, but on my side, I'm happy with closing as not planned.

[edgarrmondragon]

I think we should actually finally implement this in the SDK