2025-05-01 kernel meeting notes

[zachschuermann]

Summary

released 0.10.0 with arrow 55 support! delta-rs upgraded to kernel 0.10.0 (on arrow-54 - making arrow 55 a later problem).

ongoing threads

• IN/NOT IN PR to merge @roeap
• predicate/expressions split @scovich - merging lots of PRs
• merge opaque expressions @scovich
• merge IN/NOT IN deprecation @zachschuermann
• scan incremental API from @roeap
• @hntd187 targeting CDF reads, need timestamp work
• two PRs for checkpoint write support from @sebastiantia
• #[derive(IntoEngineData)] (zach)
• CREATE TABLE (zach)
• stats for writes (blocked)
• domain metadata (zach)
• need row id's (re-introduce predicate pushdown in
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• robert: working on support for multiple object stores

Attendees

@zachschuermann @nicklan @mightyshazam @hntd187 @scovich @rtyler

notes stream

• release: 0.10.0 for arrow-55
• ryan tells a story
• upcoming: 0.11.0 for checkpoint writes
• Error framework refactor: error codes (database engines, rustc, etc. do this)
  • user-facing vs internal error (internal error could have more info)
  • internal strongly-typed error classes, expose via type erasure
  • tyler: #[non_exhaustive]?
  • what do we expect clients to do with errors?
  • tyler: code + msg might be unfriendly to some consumers
  • kyjah: translating to dotnet exception
  • ryan: error codes are defined/stable
  • delta_kernel::Error non_exhautive enum (that implemens rust Error). we do tostring with error codes.
  • how do non-exhautive enums work with breaking changes? match requires _ case.
  • con: we can never be sure that you get all error cases. for expressions this is bad, for errors this is okay.
  • PR for non_exhaustive with future plan
• possibility: splitting crates - delta_kernel_engine crate (includes Engine impl) and delta_kernel crate
  • todo zach: exhaustive list of how these interactions look, what pieces live where (e.g. need arrow without everything else)
  • kernel needs dev dep on engine crate for arrow
  • early on FFI split out and ryan likes
  • we get versioning differences
  • concern: does DefaultEngine do special things? need to split out to find out
• coming: writing file stats
• robert: incremental scan API