feat(puffin): Add PuffinWriter #959
Conversation
fqaiser94
force-pushed
the
puffin_writer
branch
3 times, most recently
from
af396c4 to
d3749f4
Compare
|
@fqaiser94 Whats the current idea for this? I can offer help if needed. |
Forgot about this, let me try to revive this over the weekend |
fqaiser94
force-pushed
the
puffin_writer
branch
from
d3749f4 to
7252011
Compare
| pub fn bytes_written(&self) -> u64 { | ||
| self.written_size.load(std::sync::atomic::Ordering::SeqCst) | ||
| } |
There was a problem hiding this comment.
This is the only part I'm concerned about and have been struggling to find time to investigate. What memory ordering option should we use here?
In the rest of the codebase, the written_size variable is accessed using Relaxed memory ordering ... I'm a little concerned about that TBH from a consistency perspective but for the moment, let's focus on just this PR's use case. For Puffin, it's important we get accurate written_size as that is included in the BlobMetadata that we write into Puffin file footer for each Blob so that users can efficiently extract blobs later.
Now that I'm thinking about this again, I think I might have a bug here lol I can't just SeqCst for the load side ... I think I need SeqCst for the store side too for this to work correctly ... in which case I can just use Release for load and Acquire for store probably ... What are people's thoughts on this, any memory ordering experts around? XD
Other alternatives:
- Write a custom tracking writer for the puffin use case and avoid async/atomic variables entirely (like the reference Java implementation)
Part of #744
Summary
Context