-
Notifications
You must be signed in to change notification settings - Fork 1k
Open
Labels
enhancementAny new improvement worthy of a entry in the changelogAny new improvement worthy of a entry in the changelog
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We currently have two APIs, variant_get and shred_variant, that share code paths in variant_to_arrow.
variant_get: cast aVariantto any valid Arrow data type.shred_variant: convert aVariantonly to valid shredded value types per the Parquet Variant Shredding spec.
Right now these responsibilities blur together. This makes the API confusing and may let shred_variant accept types it shouldn’t.
Describe the solution you'd like
- Add checks in
shred_variantto allow only spec-approved shredded types. - Keep
variant_to_arrowsimple, it should only convert Variant → Arrow. No shredding-specific rules.
Describe alternatives you've considered
2 separate builders for the 2 APIs
Additional context
Found while reviewing #8768
Metadata
Metadata
Assignees
Labels
enhancementAny new improvement worthy of a entry in the changelogAny new improvement worthy of a entry in the changelog