bloom filters

[raphaellaude]

Still much to do but making progress!

When testing locally, I'm hitting some issues getting matches against the fixtures. I'm pretty sure this is an issue somewhere in the hashing step, such that the 32 least significant digits in the hash (using xxhashjs) are different than what the writer (duckdb in this case) inserted into the block.

Parsing

• Read bloom filter bytes
• Parse bloom filter bytes into header and bitset
• Parse bitset into bloom filter Uint32Array blocks

Bloom filter checks

• filterCheck function which hashes the input value, determines the block index for insertion and provides the 32 least significant digits to be inserted into that block
• XxHash64 algorithm
• Block check
• Block insert not technically needed in the hyparquet reader here but useful for testing and will be needed in writer implementation

Querying

• Integrate into parquetQuery: if a query is made that has an equality operation in a filter, and the relevant column also has a bloom filter, then we should use that to accelerate the query
• Measure effectiveness by adding a benchmark in the hyparquet-perf repo

[platypii]

Added some early feedback. Looking like a good start though thanks @raphaellaude!

[platypii]

would prefer smaller test examples if possible. use small row group size to make the bloom filter useful on a smaller file?

[raphaellaude]

Reduced the file sizes as much as I could–by a factor of 10–while still getting useful bloom filters written by duckdb.

[raphaellaude]

This is just a temporary dev dependency as I try to get a minimal functional implementation. Once I have everything other than the xxhash function implemented, we can vendor slash copy-paste an existing implementation or reimplement the xxhash64 algorithm.