Tracking read/write support for new catalogManaged tables. first targeting reads, items tracked below.
Ownership
Status: ✅ = Done, 🛠️ = WIP/Needs Review, 🟠 = TODO, ⚠️ = Delayed/Paused
Milestone 1: Catalog-managed reads in Kernel
Snapshot APIs
| Owner |
Description |
Issue/PR |
Status |
| zachschuermann |
txnId in commit info: required for catalog-managed tables |
#1148 |
✅ |
| zachschuermann |
prefactor: move out listed log files |
#1150 |
✅ |
| zachschuermann |
prefactor: move out last checkpoint hint |
#1154 |
✅ |
| zachschuermann |
add experimental UC client |
#1164 |
✅ |
| zachschuermann |
add catalogManaged table features and catalog-managed feature flag |
#1165 |
✅ |
| zachschuermann |
prefactor: push down _last_checkpoint read into LogSegment |
#1204 |
✅ |
| zachschuermann |
prefactor: SnapshotBuilder |
#1189 |
✅ |
| zachschuermann |
Add log_tail to listing |
#1194 |
✅ |
| zachschuermann |
prefactor: move Snapshot::try_new_from into SnapshotBuilder |
#1289 |
✅ |
| zachschuermann |
Add staged commits to ParsedLogPath |
#1305 |
✅ |
| zachschuermann |
SnapshotBuilder.with_log_tail |
#1290 |
✅ |
| zachschuermann |
prefactor: LogSegmentBuilder |
#1159 |
⚠️ |
Paused: TableChanges APIs
| Owner |
Description |
Issue/PR |
Status |
| zachschuermann |
Plumb through log_tail to TableChanges APIs |
#1292 |
🛠️ |
Milestone 2: Catalog-managed reads with UnityCatalog
| Owner |
Description |
Issue/PR |
Status |
| zachschuermann |
uc-catalog crate with load_table API |
#1324 |
✅ |
| scottsand-db |
uc-catalog in-mem tests |
xx |
🟠 |
| xx |
uc-catalog FFI |
xx |
⚠️ |
Milestone 3: Catalog-managed writes in kernel
| Owner |
Description |
Issue/PR |
Status |
Posted |
Merged |
| zachschuermann |
prefactor: CommitResult |
#1343 |
✅ |
|
|
| zachschuermann |
new Committer API with FileSystemCommitter |
#1349 |
✅ |
|
|
| zachschuermann |
Engine copy_atomic API |
#1400 |
✅ |
|
|
| zachschuermann |
Engine head API |
#1465 |
✅ |
|
|
| zachschuermann |
No staged commits in checkpoint/compaction |
#1374 |
✅ |
|
|
| scottsand-db |
max_pub_ver: refactor log_segment tests |
#1581 |
✅ |
|
|
| scottsand-db |
max_pub_ver: refactor ListedLogFiles builder pattern |
#1585 |
✅ |
01/06/26 |
01/06/26 |
| scottsand-db |
max_pub_ver: add to LogSegment |
#1587 |
✅ |
01/09/26 |
01/13/26 |
| scottsand-db |
max_pub_ver: add to CommitMetadata |
#1609 |
✅ |
01/13/26 |
01/15/26 |
| scottsand-db |
post-commit snapshot: add FileMeta to CommitResult |
#1599 |
✅ |
01/08/26 |
01/14/26 |
| scottsand-db |
post-commit snapshot: LogSegment::new_with_commit_appended |
#1602 |
✅ |
01/09/26 |
01/16/26 |
| scottsand-db |
post-commit snapshot: Snapshot::new_post_commit |
#1604 |
✅ |
01/15/26 |
01/20/26 |
| scottsand-db |
post-commit snapshot: return postCommitSnapshot from txn |
#1633 |
🛠️ |
01/20/26 |
xx |
| scottsand-db |
Publish: refactor committer.rs |
#1622 |
✅ |
01/16/26 |
01/21/26 |
| scottsand-db |
Publish: Committer::publish API |
#1625 |
🛠️ |
01/16/26 |
xx |
| scottsand-db |
Publish: UC-Committer impl |
#1626 |
🛠️ |
01/17/26 |
xx |
| scottsand-db |
Publish API |
xx |
🟠 |
|
|
Milestone 4: Catalog-managed writes with UnityCatalog
| Owner |
Description |
Issue/PR |
Status |
Posted |
Merged |
| zachschuermann |
uc-client commit API |
#1399 |
✅ |
|
|
| zachschuermann |
UCCommitter implementation |
#1418 |
✅ |
|
|
| zachschuermann |
rustls for uc-client |
#1533 |
✅ |
|
|
| scottsand-db |
UCCommitsClient: extract shared HTTP utilities |
#1590 |
✅ |
|
|
| scottsand-db |
UCCommitsClient: trait for custom client injection |
#1591 |
✅ |
01/07/26 |
01/13/26 |
| scottsand-db |
UCCommitter: use max known published version |
#1609 |
✅ |
01/13/26 |
01/15/26 |
| scottsand-db |
uc-client in-memory implementation for testing |
#1644 |
🛠️ |
01/20/26 |
xx |
| scottsand-db |
CCv2 metrics (commit/publish) |
xx |
🟠 |
|
|
| xx |
Improve UC-Catalog error types? discussion link |
xx |
xx |
|
|
| xx |
Improve LogSegment construction and ListedLogFiles constructor. Currently we are collecting twice. This can be neater. |
xx |
xx |
|
|
