Skip to content

[Variant] Enforce shredded-type validation in shred_variant #8795

@liamzwbao

Description

@liamzwbao

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We currently have two APIs, variant_get and shred_variant, that share code paths in variant_to_arrow.

  • variant_get: cast a Variant to any valid Arrow data type.
  • shred_variant: convert a Variant only to valid shredded value types per the Parquet Variant Shredding spec.

Right now these responsibilities blur together. This makes the API confusing and may let shred_variant accept types it shouldn’t.

Describe the solution you'd like

  • Add checks in shred_variant to allow only spec-approved shredded types.
  • Keep variant_to_arrow simple, it should only convert Variant → Arrow. No shredding-specific rules.

Describe alternatives you've considered

2 separate builders for the 2 APIs

Additional context

Found while reviewing #8768

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelog

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions