-
Notifications
You must be signed in to change notification settings - Fork 1k
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to change the code in datafusion to use a bitmask rather than slice of booleans for tracking equal group by values so I can do bitwise operations.
And to allow me to nudge the compiler to auto-vectorize code.
Also it will allow me to avoid a lot of copies in combining multiple bitmasks
If I wanted to AND
a vector of BooleanBuffer
into 1, I would have to do the code below which creates a lot of intermediate copies (one for each AND
)
fn and_all_before(buffers: Vec<BooleanBuffer>) -> BooleanBuffer {
assert!(!buffers.is_empty());
let mut buffers_iter = buffers.into_iter();
let mut inter = buffers_iter.next().unwrap();
for buffer in buffers_iter {
inter = (&inter) & (&buffer);
}
inter
}
If I could use BooleanBufferBuilder
I could only have single copy:
fn and_all(buffers: Vec<BooleanBuffer>) -> BooleanBuffer {
assert!(!buffers.is_empty());
let mut acc = BooleanBufferBuilder::new(buffers[0].len());
acc.append_buffer(&buffers[0]);
for buffer in &buffers[1..] {
acc &= buffer;
}
acc.finish()
}
Related issue:
Describe the solution you'd like
Adding AND
/NOT
/XOR
/OR
operations to BooleanBufferBuilder
and MutableBuffer
I would like to be able to not only do the operation on an entire BooleanBufferBuilder
and MutableBuffer
but also on a slice of it (only AND between 8 and 64 bit from the mutable buffer)
Describe alternatives you've considered
None
Additional context
None