Commit 25fa27f
authored
fix: preserve null bitmap in nested transform expressions (#1645)
## 🥞 Stacked PR
Use this
[link](https://github.com/delta-io/delta-kernel-rs/pull/1645/files) to
review incremental changes.
-
[**stack/null-propagation**](#1645)
[[Files
changed](https://github.com/delta-io/delta-kernel-rs/pull/1645/files)]
-
[stack/coalesce](#1648)
[[Files
changed](https://github.com/delta-io/delta-kernel-rs/pull/1648/files/37e755009566511bf7c2f00e014c1647e77e4533..d64042f7908844ef2d8a1c68312dc3ff936d60dc)]
-
[stack/checkpoint-transforms](#1646)
[[Files
changed](https://github.com/delta-io/delta-kernel-rs/pull/1646/files/d64042f7908844ef2d8a1c68312dc3ff936d60dc..4e66ca004f89b23431a96ac106a9c0d400718b10)]
-
[stack/write-stats](#1643)
[[Files
changed](https://github.com/delta-io/delta-kernel-rs/pull/1643/files/4e66ca004f89b23431a96ac106a9c0d400718b10..cd64f79fd3b40ebfa811cb333369cb17aa1a2a74)]
---------
## What changes are proposed in this pull request?
Fixes a bug in nested transform expression evaluation where null rows in
the source struct were losing their null bitmap, causing null structs to
incorrectly appear as non-null structs with null fields.
When evaluating nested transform expressions (transforms with an
input_path that operate on a nested struct), the output StructArray was
created with None for the null buffer:
`let data = StructArray::try_new(output_fields.into(), output_cols,
None)?;`
This meant that if the source struct had null rows (e.g., an add action
that is null in a checkpoint batch), the output would lose that null
information. The struct would appear as non-null but with all-null
fields, which is semantically different.
## How was this change tested?
Existing transform tests pass. The stats transform integration tests (in
a follow-up PR) exercise this code path.1 parent daa9a1e commit 25fa27f
1 file changed
+11
-4
lines changedLines changed: 11 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
159 | 159 | | |
160 | 160 | | |
161 | 161 | | |
162 | | - | |
| 162 | + | |
163 | 163 | | |
164 | 164 | | |
165 | 165 | | |
166 | 166 | | |
167 | | - | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
168 | 175 | | |
169 | 176 | | |
170 | 177 | | |
| |||
204 | 211 | | |
205 | 212 | | |
206 | 213 | | |
207 | | - | |
| 214 | + | |
208 | 215 | | |
209 | 216 | | |
210 | 217 | | |
| |||
216 | 223 | | |
217 | 224 | | |
218 | 225 | | |
219 | | - | |
| 226 | + | |
220 | 227 | | |
221 | 228 | | |
222 | 229 | | |
| |||
0 commit comments