impl(storage): configurable buffered upload types #2710
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Switch the upload type based on the size of the payload. Small uploads perform better when using single-shot uploads. Large uploads require resumable uploads to keep the buffer size under control, and the extra latency is not that bad in relative terms.
product-auto-label
bot
added
the
api: storage
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2710 +/- ##
==========================================
+ Coverage 95.59% 95.71% +0.11%
==========================================
Files 93 93
Lines 4018 4034 +16
==========================================
+ Hits 3841 3861 +20
+ Misses 177 173 -4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
dbolduc
reviewed
Jul 17, 2025
Comment on lines
36
to
37
| // TODO(#2043) - make the threshold to use resumable uploads and the | ||
| // target size for each chunk configurable. |
| // TODO(#2043) - make the threshold to use resumable uploads and the | ||
| // target size for each chunk configurable. | ||
| if self.payload.lock().await.size_hint().0 > RESUMABLE_UPLOAD_QUANTUM as u64 { | ||
| let hint = self.payload.lock().await.size_hint().1; | ||
| if hint.is_some_and(|max| max >= self.options.resumable_upload_threshold as u64) { |
There was a problem hiding this comment.
Q: should the default for unknown-sized objects be resumable uploads? i.e. s/is_some_and/is_none_or/
| @@ -205,23 +214,84 @@ impl Seek for BytesSource { | |||
| } | |||
| } | |||
|
|
|||
| pub struct IterSource { | |||
| impl Seek for IterSource { | ||
| type Error = std::io::Error; | ||
| async fn seek(&mut self, offset: u64) -> std::result::Result<(), Self::Error> { | ||
| let mut current: VecDeque<_> = self.contents.iter().cloned().collect(); |
There was a problem hiding this comment.
if this is a public type, this implementation should be more efficient. We only need to clone the chunks from contents which have already been consumed from current.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Switch the upload type based on the size of the payload. Small uploads
perform better when using single-shot uploads. Large uploads require
resumable uploads to keep the buffer size under control, and the extra
latency is not that bad in relative terms.
Fixes #2709, fixes #2056