-
Notifications
You must be signed in to change notification settings - Fork 139
fix: preserve null bitmap in nested transform expressions #1645
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1645 +/- ##
=======================================
Coverage 84.65% 84.65%
=======================================
Files 123 123
Lines 34109 34114 +5
Branches 34109 34114 +5
=======================================
+ Hits 28875 28880 +5
Misses 3905 3905
Partials 1329 1329 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
d37ed82 to
85063cc
Compare
85063cc to
166dd26
Compare
c13395e to
9dd11dd
Compare
nicklan
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice catch! i'm assuming from your PR comment that this is tested by code later in the stack, so lgtm
dengsh12
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks
9dd11dd to
37e7550
Compare
🥞 Stacked PR
Use this link to review incremental changes.
What changes are proposed in this pull request?
Fixes a bug in nested transform expression evaluation where null rows in the source struct were losing their null bitmap, causing null structs to incorrectly appear as non-null structs with null fields.
When evaluating nested transform expressions (transforms with an input_path that operate on a nested struct), the output StructArray was created with None for the null buffer:
let data = StructArray::try_new(output_fields.into(), output_cols, None)?;This meant that if the source struct had null rows (e.g., an add action that is null in a checkpoint batch), the output would lose that null information. The struct would appear as non-null but with all-null fields, which is semantically different.
How was this change tested?
Existing transform tests pass. The stats transform integration tests (in a follow-up PR) exercise this code path.