You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The arrow expression evaluator for the default and sync engine does not handle non-nullable fields in a nullable struct. If all the (possibly non-nullable) fields of a struct are null, and the top level struct is nullable, the expression evaluator must allow the output struct to also be null.
Consider the schema
let nested = StructType::new([StructField::new("path", DeltaDataTypes::STRING, false)]);
let schema = StructType::new([StructField::new("add", nested, true)]);
Describe the bug
The arrow expression evaluator for the default and sync engine does not handle non-nullable fields in a nullable struct. If all the (possibly non-nullable) fields of a struct are null, and the top level struct is nullable, the expression evaluator must allow the output struct to also be null.
Consider the schema
For engine data
I expect the following:
To Reproduce
I created a MRE test that can be run in
arrow_expression.rs
. I have a top level nullable structadd
with a non-nullable fieldpath
.Expected behavior
I expect the test to successfully transform the data to something like this:
Additional context
This was caught when working on CDF scan file transformation and schema.
The text was updated successfully, but these errors were encountered: