Skip to content

Add option to skip recursive list and delete operation in the S3 exchange #25897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jack-o-hara opened this issue Jun 2, 2025 · 2 comments

Comments

@jack-o-hara
Copy link
Contributor

The recursive list then delete operation adds pressure to the exchange. This can help reduce the S3 volume and likelihood of throttling by the S3 service.

S3 management lifecycle rules can clean up these files after 24h (cost is paid via management fees instead of LIST operation cost) so it's not required.

We rebuilt the trino-exchange-filesystem module with this operation disabled on aborts/cleanup and noticed a significant reduction in throttling in our production environment.

@raunaqmorarka
Copy link
Member

cc: @losipiuk

@losipiuk
Copy link
Member

losipiuk commented Jun 3, 2025

@jack-o-hara makes sense. Happy would you be wiling to send a PR which adds that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants