Skip to content

Conversation

@DrakeLin
Copy link
Collaborator

@DrakeLin DrakeLin commented Jan 20, 2026

🥞 Stacked PR

Use this link to review incremental changes.


What changes are proposed in this pull request?

Change apply_schema to propagate top-level struct nulls to child columns instead of erroring

  • Remove the error check for top-level nulls in apply_schema
  • Document that child columns are expected to already have nulls propagated (Arrow's JSON reader does this automatically, and parquet data goes through fix_nested_null_masks)
  • Add comprehensive test case test_apply_schema_handles_top_level_null

How was this change tested?

Edited unit tests

Added unit test to show new behavior

@DrakeLin DrakeLin changed the title null feat: Propagate top-level struct nulls to child columns in apply_schema Jan 20, 2026
@DrakeLin DrakeLin requested review from dengsh12 and nicklan January 20, 2026 19:15
@github-actions github-actions bot added the breaking-change Change that require a major version bump label Jan 20, 2026
@codecov
Copy link

codecov bot commented Jan 20, 2026

Codecov Report

❌ Patch coverage is 81.39535% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.16%. Comparing base (07083a1) to head (e5beaf9).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
kernel/src/engine/arrow_expression/apply_schema.rs 81.39% 0 Missing and 8 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1635      +/-   ##
==========================================
- Coverage   84.16%   84.16%   -0.01%     
==========================================
  Files         123      123              
  Lines       34179    34214      +35     
  Branches    34179    34214      +35     
==========================================
+ Hits        28767    28796      +29     
  Misses       4021     4021              
- Partials     1391     1397       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@DrakeLin DrakeLin marked this pull request as draft January 20, 2026 20:28
@DrakeLin DrakeLin force-pushed the stack/propagate-nulls branch from f126545 to b3b511b Compare January 20, 2026 20:29
@DrakeLin DrakeLin changed the title feat: Propagate top-level struct nulls to child columns in apply_schema feat: Enable Arrow to convert nullable StructArray to RecordBatch Jan 20, 2026
@DrakeLin DrakeLin marked this pull request as ready for review January 20, 2026 21:26
Copy link
Collaborator

@dengsh12 dengsh12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, amazing to learn the Arrow's JSON reader does this automatically

@github-actions github-actions bot removed the breaking-change Change that require a major version bump label Jan 21, 2026
Copy link
Member

@nicklan nicklan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks for looking into this!

@DrakeLin DrakeLin merged commit 2f74fd3 into delta-io:main Jan 21, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants