Add Parquet read pruning configuration for max elements in inList

[16pierre]

This PR adds Parquet read configuration for the previous hard-coded constant MAX_LIST_VALUE_SIZE_REWRITE, which applies an upper-bound to the number of elements in InList exprs when pruning is applied.

Closes #8609

[16pierre]

Went for a struct to ease up possible subsequent refactors if we add extra args; not sure if aligned with standards.

[16pierre]

row_group_predicate_in_list_to_many_values felt like an obvious utest to verify the configuration is applied, I may be missing some extra test coverage though; in particular, this change is missing integration tests that verify the end-to-end integration of the new Parquet configuration. Not sure if extra coverage is required, and if so where to implement it, happy to get guidance.

[adriangb]

I would put it in datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt

[16pierre]

Tried f6624d7, happy to add more coverage if necessary. To some degree the test is testing the formatter logic rather than the internal logic because fmt re-does the predicate pruning logic:

datafusion/datafusion/datasource-parquet/src/source.rs

Lines 650 to 667 in 118dc6f

	if let (Some(pruning_predicate), _) = build_pruning_predicates(
	Some(predicate),
	self.table_schema.table_schema(),
	&predicate_creation_errors,
	&pruning_predicate_config,
	) {
	let mut guarantees = pruning_predicate
	.literal_guarantees()
	.iter()
	.map(|item| format!("{item}"))
	.collect_vec();
	guarantees.sort();
	write!(
	f,
	", pruning_predicate={}, required_guarantees=[{}]",
	pruning_predicate.predicate_expr(),
	guarantees.join(", ")
	)?;

[adriangb]

I wonder if we could just make PruningPredicateConfig a field here instead of polluting with more fields. Also can this be pub(crate) instead of pub?

[16pierre]

Is this what you had in mind ? 0f6cdeb

[adriangb]

Also here could we make PruningPredicateConfig serializable and send that across the wire?

[16pierre]

In general in this PR I hesitated at which level we should starting wiring PruningPredicateConfig vs wiring the individual usize pruning_max_inlist_limit.

On the ParquetOptions Rust struct I tried to follow the existing standards of "flat types", so I mirrored the exact same structure on the protobuf objects.

If I understand correctly, you're suggesting we introduce an explicit protobuf type for PruningPredicateConfig in the ParquetOptions proto format, without changing the current rust-side ParquetOptions struct ?

[adriangb]

@16pierre just in case it's helpful you can update an slt by running cargo test --test sqllogictests -- parquet_filter_pushdown --complete and such

[16pierre]

it's helpful you can update an slt by running cargo test --test sqllogictests -- parquet_filter_pushdown --complete and such

Yeah I discovered this for parquet_filter_pushdown but unfortunately for information_schema.slt I got local errors (which I manually patched)

Thanks for the pointer (and cool tooling btw)