Change some panics to errors in parquet decoder #8602
Conversation
rambleraptor
force-pushed
the
fix-some-panics
branch
from
17d5287 to
4d193b3
Compare
|
I guess we can lump this in with #7806 |
|
I'm happy to split this up into some separate PRs. I know it's a lot of random things as-is. |
The pedant in me wants to take you up on your offer, but there's not so much going on here that I think that's necessary. Maybe just change the title to something that sounds better 😉. ("Address panics found in external testing"). |
There was a problem hiding this comment.
Thanks @rambleraptor, these all look sensible to me.
Would it be possible to gin up some tests for at least some of them?
| return Ok((end, buf.slice(i32_size..end))); | ||
| } | ||
| } | ||
| Err(general_err!("not enough data to read levels")) |
There was a problem hiding this comment.
This is definitely an improvement over the existing code, but it opens a question:
Given that we're reading bytes from a byte buffer, it seems like we must expect to hit this situation at least occasionally? And the correct response is to fetch more bytes, not fail? Is there some mechanism for handling that higher up in the call stack? Or is there some reason it should be impossible for this code to run off the end of the buffer?
There was a problem hiding this comment.
Also -- it seems like read_num_bytes should do bounds checking internally and return Option<T>, so buffer overrun is obvious at the call site instead of a hidden panic footgun? The method has a half dozen other callers, and they all need to do manual bounds checking, in various ways and with varying degrees of safety. In particular, parquet/src/data_type.rs has two call sites that lack any visible bounds checks.
There was a problem hiding this comment.
In this particular instance we're reading a buffer that should contain an entire page of data. If it doesn't, that likely points to a problem with the metadata.
Changes to read_num_bytes would likely need more careful consideration as I suspect it might be used in some performance critical sections.
| } else if !is_root_node { | ||
| return Err(general_err!("Repetition level must be defined for non-root types")); | ||
| } | ||
| Ok((next_index, Arc::new(builder.build().unwrap()))) |
There was a problem hiding this comment.
How do we know the unwrap is safe?
There was a problem hiding this comment.
build never returns an Err 😉. But good point, could replace unwrap with ?.
rambleraptor
changed the title
Co-authored-by: Ed Seidl <[email protected]>
Co-authored-by: Ryan Johnson <[email protected]>
alamb
changed the title
3 participants
Rationale for this change
We've caused some unexpected panics from our internal testing. We've put in error checks for all of these so that they don't affect other users.
What changes are included in this PR?
Various error checks to ensure panics don't occur.
Are these changes tested?
Tests should continue to pass.
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
Existing tests should cover these changes.
Are there any user-facing changes?
None.