-
Notifications
You must be signed in to change notification settings - Fork 1k
Open
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The parquet community is considering adding new encodings to the format.
One proposed encoding is:
Describe the solution you'd like
I would like someone(s) to create a branch of arrow-rs 's parquet reader and implement this proposed encoding in the context of parquet to see how it would work.
This would involve likely
- Adding a new encoding to the metadata: source link
- Add the appropriate encoder/decoders in parquet (existing code)
- Add some basic tests showing data could be round tripped with these encodings: examples of similar tets here
- Add some benchmarks: examples of similar benchmarks
There are already several Rust implementations of FSST (both Lance and Vortex) that could probably adapted
Describe alternatives you've considered
Additional context
Metadata
Metadata
Assignees
Labels
No labels