GH-46205: [C++][Parquet][WIP] Read/Write null count statistics for UNKNOWN sort order #46275
Conversation
|
|
github-actions
bot
added
Component: Parquet
Component: C++
awaiting committer review
| page_statistics_ = MakeStatistics<ParquetType>(descr_, allocator_); | ||
| chunk_statistics_ = MakeStatistics<ParquetType>(descr_, allocator_); |
There was a problem hiding this comment.
The actual version of this should maybe modify the if (SortOrder::UNKNOWN != descr_->sort_order()) { check just above. Perhaps there needs to be a descr_->can_write_statistics() to separate the sortedness from whether or not we can write anything?
github-actions
bot
added
awaiting changes
| bool is_geometry = | ||
| descr_->logical_type() != nullptr && descr_->logical_type()->is_geometry(); | ||
| if (!column_metadata_->__isset.statistics || | ||
| descr_->sort_order() == SortOrder::UNKNOWN) { | ||
| (descr_->sort_order() == SortOrder::UNKNOWN && !is_geometry)) { |
There was a problem hiding this comment.
Again, maybe we need a descr_->can_read_statistics? There's a HasCorrectStatistics(), too, and maybe the check needs to be there. I wonder whether the types currently marked as unsorted had null counts written reliably by other implementations or whether we have to ignore those?
| template <typename DType> | ||
| class UnsortedTypedStatisticsImpl : public TypedStatistics<DType> { |
There was a problem hiding this comment.
I'm not sure this is the answer here...I just needed a TypedStatistics<> to make this work in the ColumnWriter
There was a problem hiding this comment.
Perhaps we can reuse the existing typed one and just ignore the stats on write? (Seems inefficient but may be more compact?)
Rationale for this change
Minimum and maximum values are not useful in the context of an unsorted converted or loigcal type; however, null counts are! Geometry is one example of an unsorted type where this type of information might be useful; however, there are others as well!
What changes are included in this PR?
Early work-in-progress explorations to find the relevant pieces of code.
Are these changes tested?
They will be once a general direction is decided on!
Are there any user-facing changes?
Possibly!