Introduce branching and support for spark.wap.branch #379

cbb330 · 2025-09-29T19:31:12Z

Summary

Implement Iceberg table branching functionality in OpenHouse internal catalog with comprehensive testing infrastructure.

The old code was getting unwieldy with nested for-loops, mutable collections, and scattered state management, repeated logic and overall needed a restructure with branching supported first-class. i feel more confident that this is reliable.

Changes

Branch Operations: Added branch management methods to OpenHouseInternalTableOperations including create, delete, list, and commit reference handling
Multi-branch Commit Support: Implemented logic to handle commits across different branch references with checks to reject ambiguous commits
WAP Branch Integration: Added Write-Audit-Publish (WAP) branch support with proper test coverage
Test Infrastructure: Created comprehensive test suite (BranchTestSpark3_5) covering branch operations, edge cases, and integration scenarios

Changes

Testing Done

Manually Tested on local docker setup. Please include commands ran, and their output.
Added new tests for the changes made.
Updated existing tests to reflect the changes made.
No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
Some other form of testing like staging or soak time in production. Please explain.

Additional Information

Breaking Changes
Deprecations
Large PR broken into smaller PRs, and PR plan linked in the description.

.../src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalTableOperations.java

sumedhsakdeo

PTAL

.../src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalTableOperations.java

jiang95-dev

Thanks Christian for the comprehensive tests to ensure safety! I couldn't finish reading all of the tests, but I read through SnapshotDiffApplier and left some comments. I don't have concerns about correctness, but have some regarding readability and easiness to debug.

...e-spark-itest/src/test/java/com/linkedin/openhouse/spark/catalogtest/BranchTestSpark3_5.java

...ternalcatalog/src/main/java/com/linkedin/openhouse/internal/catalog/SnapshotDiffApplier.java

...e-spark-itest/src/test/java/com/linkedin/openhouse/spark/catalogtest/BranchTestSpark3_5.java

...ternalcatalog/src/main/java/com/linkedin/openhouse/internal/catalog/SnapshotDiffApplier.java

jiang95-dev · 2025-10-19T09:33:11Z

...ternalcatalog/src/main/java/com/linkedin/openhouse/internal/catalog/SnapshotDiffApplier.java

+      int appendedCount =
+          (int)
+              regularSnapshots.stream()
+                  .filter(s -> !existingById.containsKey(s.snapshotId()))
+                  .count();


Related to the above comment. We should have regularSnapshots as appended regular snapshots.

...ternalcatalog/src/main/java/com/linkedin/openhouse/internal/catalog/SnapshotDiffApplier.java

sumedhsakdeo

Can you rebase this PR on top of #392 (review) thanks 🙏

sumedhsakdeo · 2025-11-09T20:27:13Z

...e-spark-itest/src/test/java/com/linkedin/openhouse/spark/catalogtest/BranchTestSpark3_5.java

+      assertTrue(refNames.contains("main"));
+    }
+  }
+


Can we add a test-case here that enables MOR on table, and then updates/deletes rows in the main branch, then test two-scenarios, commit conflict that cannot be retried because main has received new updates and cherry pick that goes through because main has not received new updates.

sumedhsakdeo

We should find a way to reuse some of the tests in iceberg repo to work against our OH catalog implementation for complete confidence. I found the following test cases that may be relevant.

We could either leverage the test modules from linkedin/iceberg in linkedin/openhouse, or use OpenHouse catalog in linkedin/iceberg.

sumedhsakdeo · 2025-11-09T20:30:21Z

...e-spark-itest/src/test/java/com/linkedin/openhouse/spark/catalogtest/BranchTestSpark3_5.java

+              .sql("SELECT * FROM " + tableName + " VERSION AS OF 'feature_a'")
+              .collectAsList()
+              .size()); // feature-a unchanged
+


add a test for select from wap snapshot?

sumedhsakdeo · 2025-11-09T20:33:37Z

...e-spark-itest/src/test/java/com/linkedin/openhouse/spark/catalogtest/BranchTestSpark3_5.java

+          "Main branch should still have 1 row (staged WAP not visible)");
+
+      // ===== SELECTIVE WAP PUBLISHING =====
+


swap the publish order here, wap3 before wap2, then do a time-travel query see if the cherry pick order is maintained.

sumedhsakdeo · 2025-11-09T20:34:58Z

...e-spark-itest/src/test/java/com/linkedin/openhouse/spark/catalogtest/BranchTestSpark3_5.java

+              .collectAsList();
+      String wap2SnapshotId = String.valueOf(wap2Snapshots.get(0).getLong(0));
+      spark.sql(
+          "CALL openhouse.system.cherrypick_snapshot('"


Any particular reason why you picked cherrypick_snapshot over publish_changes? What is the difference? Should we have a test for both?

sumedhsakdeo reviewed Sep 29, 2025

View reviewed changes

.../src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalTableOperations.java Outdated Show resolved Hide resolved

sumedhsakdeo reviewed Sep 29, 2025

View reviewed changes

.../src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalTableOperations.java Outdated Show resolved Hide resolved

sumedhsakdeo reviewed Sep 29, 2025

View reviewed changes

.../src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalTableOperations.java Outdated Show resolved Hide resolved

sumedhsakdeo requested changes Oct 1, 2025

View reviewed changes

cbb330 changed the title ~~WIP: introduce branching and and wap branch~~ WIP: introduce branching and support for spark.wap.branch Oct 2, 2025

cbb330 changed the title ~~WIP: introduce branching and support for spark.wap.branch~~ Introduce branching and support for spark.wap.branch Oct 2, 2025

sumedhsakdeo reviewed Oct 5, 2025

View reviewed changes

.../src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalTableOperations.java Outdated Show resolved Hide resolved

jiang95-dev reviewed Oct 19, 2025

View reviewed changes

cbb330 added 19 commits November 3, 2025 14:48

introducing branching

4b60c70

wap branch green tests

8546d43

accidentally commented line

e206888

remove test with old behavior

ef1e5b4

fixing multi-branch commits and ambiguous references

d0de1da

refactoring for readability

554a3c3

fixed edge case

ea5ff0e

refactoring to make more simple

4fc3792

removing unused function

9d6aec0

workign tests for ambiguous commits

bf5a474

tests for the replication use case

4d9dae0

refactoring pipeline

abdf335

working tests and restructured code

4087462

adding comments

a101d72

working tests

11be438

complete refactor + new tests

c7426b4

fixing broken tests

afe2627

centralizing maps/lists in constructor and reusing in applyTo

6ba98f5

responding to comments

39b6cf1

cbb330 force-pushed the main branch 2 times, most recently from 8748e63 to 39b6cf1 Compare November 4, 2025 23:01

removing guava

ad57067

rebasing from PR1 for the business logic, TODO: test rebasing

aa6f73e

sumedhsakdeo reviewed Nov 9, 2025

View reviewed changes

cbb330 added 4 commits November 10, 2025 23:36

small changes

c9c05ae

fixing comments

99f5636

working multi branch with improved and more reliable logic

2f1cad2

not removing props

28f4185

		"Main branch should still have 1 row (staged WAP not visible)");

		// ===== SELECTIVE WAP PUBLISHING =====

Introduce branching and support for spark.wap.branch #379

Are you sure you want to change the base?

Introduce branching and support for spark.wap.branch #379

Uh oh!

Conversation

cbb330 commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Changes

Testing Done

Additional Information

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sumedhsakdeo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jiang95-dev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jiang95-dev Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sumedhsakdeo left a comment

Choose a reason for hiding this comment

Uh oh!

sumedhsakdeo Nov 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sumedhsakdeo left a comment

Choose a reason for hiding this comment

Uh oh!

sumedhsakdeo Nov 9, 2025

Choose a reason for hiding this comment

Uh oh!

sumedhsakdeo Nov 9, 2025

Choose a reason for hiding this comment

Uh oh!

sumedhsakdeo Nov 9, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cbb330 commented Sep 29, 2025 •

edited

Loading

sumedhsakdeo Nov 9, 2025 •

edited

Loading