feature request: s3 access via short-lived tokens

[menzenski]

We run Meltano in Kubernetes with AWS EKS and our jobs run with a Kubernetes ServiceAccount mapped to an AWS IAM role with S3 access. Currently, we use tap-spreadsheets-anywhere to extract CSV data from AWS S3 using short-lived tokens via sts:AssumeRole (AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_SESSION_NAME environment variables).

In tap-spreadsheets-anywhere the tap just uses boto3 and this behavior is provided out-of-the-box. I'm interested to try this tap-universal-file extractor but it seems to only allow long-lived access key and secret credentials for S3.

[visch]

@menzenski hmm this does utilize boto3 under the hood. If you set those env variables it might "just work" give it a shot and we could document it!

[visch]

@menzenski

https://github.com/MeltanoLabs/tap-universal-file/blob/main/tap_universal_file/files.py#L186-L200

I think we have a heavy hand here you're probably right. If you dropped https://github.com/MeltanoLabs/tap-universal-file/blob/main/tap_universal_file/files.py#L189-L197 it might "just work" and we could handle s3 auth better so it seems. Fyi @sebastianswms