Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

N5Factory add make s3 ClientConfiguration customizable #22

Open
bogovicj opened this issue May 28, 2024 · 1 comment
Open

N5Factory add make s3 ClientConfiguration customizable #22

bogovicj opened this issue May 28, 2024 · 1 comment

Comments

@bogovicj
Copy link
Collaborator

see saalfeldlab/n5-aws-s3#31
@kgabor
Copy link

kgabor commented Jun 5, 2024 •
edited
Loading

Another issue in an EMR-serverless environment was authentication "token expiration" errors when a spark task runs for several hours. This is related to assumed roles in the EMR-serverless environment. The suggested workaround is to use a custom credentialsProvider in setting up the S3Client. Beside the ClientConfiguration it'd be nice to have flexibility in customizing the credentialsProvider as well. Attached the suggested singleton S3Manager class solution by AWS support for both the token expiration and for the slowdown error issues mentioned in 31. (I am not sure, this may need the upgrade to AWS SDK v2.)

emr-spark-demo.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kgabor @bogovicj