Skip to content

Conversation

@etseidl
Copy link
Contributor

@etseidl etseidl commented Nov 6, 2025

Which issue does this PR close?

Rationale for this change

This builds on #8763 to add options to either skip decoding of the page encoding statistics array, or transform it to a bitmask. This gets rid of the last of the heavy allocations in the metadata decoder.

What changes are included in this PR?

Adds new options to ParquetMetaDataOptions. Also adds more metadata benchmarks.

Are these changes tested?

Yes

Are there any user-facing changes?

No, just adds new field to ColumnChunkMetaData.

@github-actions github-actions bot added the parquet Changes to the parquet crate label Nov 6, 2025
@etseidl
Copy link
Contributor Author

etseidl commented Nov 6, 2025

excerpts from new benchmarks

decode parquet metadata time:   [15.050 µs 15.105 µs 15.164 µs]
decode metadata with schema
                        time:   [7.7038 µs 7.7286 µs 7.7561 µs]
decode metadata with stats mask
                        time:   [14.035 µs 14.100 µs 14.182 µs
decode metadata with skip PES
                        time:   [13.976 µs 14.016 µs 14.060 µs]
decode parquet metadata (wide)
                        time:   [54.013 ms 54.236 ms 54.468 ms]
decode metadata (wide) with schema
                        time:   [48.399 ms 48.562 ms 48.738 ms]
decode metadata (wide) with stats mask
                        time:   [44.912 ms 45.077 ms 45.253 ms]
decode metadata (wide) with skip PES
                        time:   [44.500 ms 44.616 ms 44.739 ms]

Skipping the stats is not any faster than turning them into a mask 😮.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce allocations in ParquetMetaData for improved performance

1 participant