[Variant] Support Shredded Lists/Array in variant_get

[liamzwbao]

Which issue does this PR close?

• Closes [Variant] Support Shredded Lists/Array in variant_get #8082.

Rationale for this change

What changes are included in this PR?

• Move ArrayVariantToArrowRowBuilder from shred_variant to variant_to_arrow so it can be shared with variant_get.
• Reorder definitions in variant_to_arrow to clarify the hierarchy: start with the top-level VariantToArrowRowBuilder, then second-level builders such as PrimitiveVariantToArrowRowBuilder and ArrayVariantToArrowRowBuilder, etc.
• Add tests for variant_get with lists.

Are these changes tested?

Yes

Are there any user-facing changes?

variant_get now supports lists.

[scovich]

I only had a few minutes to skim, and I couldn't tell quickly what was code movement vs. actual changes. Would there be a way to split them out into separate commits or something? Or at least describe clearly what moved vs. what changed?

The second commit in the stack was helpful that way, but the first commit still seemed to do an awful lot.

[alamb]

I only had a few minutes to skim, and I couldn't tell quickly what was code movement vs. actual changes. Would there be a way to split them out into separate commits or something? Or at least describe clearly what moved vs. what changed?

The second commit in the stack was helpful that way, but the first commit still seemed to do an awful lot.

I am also happy to help review / quickly merge a PR that is just code movement

[klion26]

LGTM, Maybe we can create an issue to track the VariantPathElement::Index case.

[klion26]

Do we need to add some cases for that with VariantPathElement::Field(We don't support VariantPathElement::Index in the current pr) or some perfectly shredded case

When trying to use shred_variant to generate some perfectly shredded cases, I found that the member variable value is not null for the return value of shred_variant. Is this the expected behavior? According to the parquet-vairant-spec

If the value is not an array, typed_value must be null. If the value is an array, value must be null.

[liamzwbao]

Good idea, I will add some tests for VariantPathElement::Field.

For shred_variant, I think the following test covers the behavior in the spec.

arrow-rs/parquet-variant-compute/src/shred_variant.rs

Lines 1193 to 1242 in 802b890

	fn test_array_shredding_as_list() {
	let input = build_variant_array(vec![
	// Row 0: List of ints should shred entirely into typed_value
	VariantRow::List(vec![
	VariantValue::from(1i64),
	VariantValue::from(2i64),
	VariantValue::from(3i64),
	]),
	// Row 1: Contains incompatible types so values fall back
	VariantRow::List(vec![
	VariantValue::from(1i64),
	VariantValue::from("two"),
	VariantValue::from(Variant::Null),
	]),
	// Row 2: Not a list -> entire row falls back
	VariantRow::Value(VariantValue::from("not a list")),
	// Row 3: Array-level null propagates
	VariantRow::Null,
	// Row 4: Empty list exercises zero-length offsets
	VariantRow::List(vec![]),
	]);
	let list_schema = DataType::List(Arc::new(Field::new("item", DataType::Int64, true)));
	let result = shred_variant(&input, &list_schema).unwrap();
	assert_eq!(result.len(), 5);
	assert_list_structure_and_elements::<Int64Type, i32>(
	&result,
	5,
	&[0, 3, 6, 6, 6, 6],
	&[Some(3), Some(3), None, None, Some(0)],
	&[
	None,
	None,
	Some(Variant::from("not a list")),
	Some(Variant::Null),
	None,
	],
	(
	&[Some(1), Some(2), Some(3), Some(1), None, None],
	&[
	None,
	None,
	None,
	None,
	Some(Variant::from("two")),
	Some(Variant::Null),
	],
	),
	);
	}

We can make some changes to the test to verify that the value would be an array of None if the input is a perfectly shredded list array,

    #[test]
    fn test_array_shredding_as_list() {
        let input = build_variant_array(vec![
            VariantRow::List(vec![
                VariantValue::from(1i64),
                VariantValue::from(2i64),
                VariantValue::from(3i64),
            ]),
            VariantRow::List(vec![
                VariantValue::from(1i64),
                VariantValue::from(2i64),
                VariantValue::from(3i64),
            ]),
            VariantRow::List(vec![
                VariantValue::from(1i64),
                VariantValue::from(2i64),
                VariantValue::from(3i64),
            ]),
            VariantRow::List(vec![
                VariantValue::from(1i64),
                VariantValue::from(2i64),
                VariantValue::from(3i64),
            ]),
        ]);
        let list_schema = DataType::List(Arc::new(Field::new("item", DataType::Int64, true)));
        let result = shred_variant(&input, &list_schema).unwrap();
        assert_eq!(result.len(), 4);

        assert_list_structure_and_elements::<Int64Type, i32>(
            &result,
            4,                                     // expected length
            &[0, 3, 6, 9, 12],                     // expected offsets
            &[Some(3), Some(3), Some(3), Some(3)], // expected sizes
            &[None, None, None, None],             // `value`
            (
                &[
                    Some(1),
                    Some(2),
                    Some(3),
                    Some(1),
                    Some(2),
                    Some(3),
                    Some(1),
                    Some(2),
                    Some(3),
                    Some(1),
                    Some(2),
                    Some(3),
                ], // `typed_values...typed_value`
                &[
                    None, None, None, None, None, None, None, None, None, None, None, None,
                ], // `typed_values...value`
            ), // `typed_value`
        );
    }

[klion26]

Ah, yes, thanks for pointing this(shred_variant) out, please ignore my previous comment.