2025-03-20 kernel meeting notes

[zachschuermann]

Summary

null_row PR to unblock lots of work (idempotent writes, CREATE TABLE, metadata updates)

ongoing threads

• Change the discussion format, particularly the pull requests/ongoing threads section.
• CI to build nightly with external integrations and make sure FFI works: CI to build nightly with external integrations and make sure FFI works #754
• Upcoming priorities for kernel
• null_row/create_one API: in final review feat!: new null_row ExpressionHandler API #662
• col [NOT] IN API: @roeap
  • does default evaluator need this? delta spec doesn't use this expression - delta-rs uses for partition skipping (going opaque kernel expression route)
  • IN and NOT IN could be default-engine provided opaque expression types (i.e. if you use default engine then you can leverage them but the kernel isn't aware of it)
  • kernel expression type for IN/not IN exists - but no support for evaluation
  • should we revert in/not in? we have partial support right now..
  • related: can we get rid of Array data scalar type.
• opaque expressions @scovich
• snapshot incremental API: Snapshot::new_from(Arc<Snapshot>) -> Arc<Snapshot>
• @hntd187 targeting CDF reads, need timestamp work
• @sebastiantia to target metadata updates?
• future: can we get rid of Array data scalar type (can we represent as EngineData)

opaque expressions and robert's IN/notIN:

• merge roberts and mark deprecated (different PR but just before release)
• oprioritize opaque expressions from @scovich (@zachschuermann to take a quick pass) some issue in rustc?
• THEN remove in/not in and migrate to opaque expression + default engine implementation.
• related/future work: can we get rid of Array data scalar type.

Attendees

@nicklan @sebastiantia @hntd187 @zachschuermann @mightyshazam @roeap @scovich

notes stream

• @roeap: ObjectStoreRegistry (hashmap lookup prior to hitting object store)
• new contributions: refactor!: make snapshot.schema() return a SchemaRef #751 and refactor: split up arrow_expression module #750
• @roeap list_from fix: fix!: get prefix from offset path #699
• object_store FFI: https://github.com/apache/arrow-rs/issues/7075
• @roeap vended credentials?
• object_store_ffi crate
• v2checkpoint writes: single-file checkpoints for now
• zach to size effort for object_store FFI (1) sync then (2) async.
• kyjah to help with object store FFI!
• opaque expressions and robert's IN/notIN:
  1. merge roberts and mark deprecated (different PR but just before release)
  2. prioritize opaque expressions from @scovich (@zachschuermann to take a quick pass) some issue in rustc?
  3. THEN remove in/not in and migrate to opaque expression + default engine implementation.
  4. related/future work: can we get rid of Array data scalar type.
     (copied above)
• Scalar::null(Map) has unimplemented!() in it - this should just return Error::unsupported()
• zach to do a couple PR for that and deprecated to IN/notIN