Emit and Read min/max statistics for int96 timestamp columns

[rahulketch]

Describe the enhancement requested

Parquet-java does not emit or read stats for int96 timestamp columns. Since int96 is used as the default timestamp in spark, this limits a lot of optimization opportunities. Engines like Photon populate the statistics for the int96 timestamps correctly. So parquet-java coud also emit the statistics, and also allow reading these statistics from known good writers.

Component(s)

No response

[alamb]

Here is a related mailing list discussion on this topic: https://lists.apache.org/thread/6fm50b3pmh6mz659jb5wx5vzmvwccz1n

[alamb]

Pedantically, according to the current spec there is no defined ordering for Int96 types and thus there is no such thing as "correct" statistics for Int96

• relevant portion of the spec:
  https://github.com/apache/parquet-format/blob/cf943c197f4fad826b14ba0c40eb0ffdab585285/src/main/thrift/parquet.thrift#L1079

That being said, what Photon is doing is reasonable (ordering Int96 according to the only logical type that uses Int96)

[alamb]

This issue discusses updating the format

• Define timestamp ordering for int96 timestamp columns parquet-format#502