feat: add bitwise ops for BooleanBufferBuilder
and for MutableBuffer
#8619
+1,401
−2
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
BooleanBufferBuilder
andMutableBuffer
that mutate directly the buffer #8618.Rationale for this change
Allowing to combine BooleanBuffers without a lot of copies and more (see issue)
What changes are included in this PR?
Created most of
Buffer
ops that exists inarrow-buffer/src/buffer/ops.rs
forMutableBuffer
andBooleanBufferBuilder
because we can't create
BitChunksMut
due to the reasons described below I had to port those to the mutable ops codeImplementation notes
Why there is a trait for
MutableOpsBufferSupportedLhs
and not gettingMutableBuffer
like theBuffer
ops getBuffer
Because then we wouldn't be able to do an operation (e.g.
AND
) on a subset (e.g. from bit 10 to bit 100) of aBooleanBufferBuilder
becauseBooleanBufferBuilder
does not exposeMutableBuffer
and I don't want to expose it as the user could then add some values that will affect theBooleanBufferBuilder
length without updating the lengthWhy there is a trait for
BufferSupportedRhs
and not gettingBuffer
like theBuffer
ops getBuffer
Because we want to be able to do
MutableBuffer & Buffer
and alsoMutableBuffer & MutableBuffer
Why not creating
BitChunksMut
forMutableBuffer
and making the code be likeBuffer
which is very simple opsAt first I thought of implementing
BitChunksMut
forMutableBuffer
like and implement the ops the same way that it was implemented for Buffer but saw that it was impossible as:u64
and I can't get a reference for thatu64
and convert them to little endian as bit-packed buffers are stored starting with the least-significant byte first.len % 64
)Are these changes tested?
Yes, although I did not run them on big endian machine
Are there any user-facing changes?
Yes, new functions which are documented
I will later change
BooleanBufferBuilder#append_packed_range
function to usemutable_bitwise_bin_op_helper
as I saw that running theboolean_append_packed
benchmark improved by 57%