OpenHouse <-> Iceberg Compatibiltiy Test Infra #218
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces Service Provider Interface (SPI) support that allows external catalog implementations to plug into Iceberg's test suite without modifying Iceberg source code. New SPI Interfaces: - TestTableProvider: Creates tables for iceberg-core tests - TestCatalogProvider: Provides catalog configurations for Spark tests - TestSparkSessionProvider: Provides custom SparkSession instances Modified Test Infrastructure: - TableTestBase: Discovers and uses external TestTableProvider via SPI - TestBase: Discovers and uses external TestSparkSessionProvider via SPI - TestBaseWithCatalog: Discovers and uses external TestCatalogProvider via SPI - CatalogTestBase: Respects external catalog configurations - ParameterizedTestExtension: Prefers most specific @parameters method New Gradle Configuration: - spark/openhouse.gradle: Shared compatibility test task configuration - build.gradle: openhouseCompatibility aggregate task Usage: ./gradlew openhouseCompatibility -PopenhouseCompatibilityCoordinate=<maven-coordinate> This enables running 860+ Iceberg tests against external catalogs like OpenHouse without forking or patching Iceberg's test classes.
- Remove legacy openHouseFixturesCoordinate approach with exclusions - Use openhouseCompatibilityRuntime with transitive=false (cleaner) - Remove hardcoded SNAPSHOT coordinate from gradle.properties - Simplify TestBase.loadSparkSessionProvider using ServiceLoader.findFirst() - Remove redundant system property handling from spark/v3.5/build.gradle
- TestBaseWithCatalog: centralize SPI loading with loadExternalCatalogProviders() - CatalogTestBase: just override defaultCatalogParameters(), inherit SPI logic - spark/v3.5/build.gradle: remove duplicate openhouseCompatibilityRuntime config - spark/openhouse.gradle: single source of truth for OpenHouse test config - Use consistent single quotes and add descriptive comments
- Converted iceberg submodule to real git repo to fix gradle-git-properties - Simplified TableTestBase to use reflection (no internal SPI dependency) - Fixed ServiceLoader usage in TestBase for Java 8 compatibility - Centralized OpenHouse config in spark/openhouse.gradle - Fixed Hadoop Configuration in OpenHouseTestTableProvider (in tables-test-fixtures) to prevent NPE
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title
Add SPI interfaces for external catalog compatibility testing
Description
This PR introduces Service Provider Interface (SPI) support that allows external catalog implementations (like OpenHouse, Polaris, Unity Catalog, etc.) to plug into Iceberg's test suite without modifying Iceberg source code.
Motivation
External catalog implementations need to validate Iceberg spec compliance by running Iceberg's own tests. Currently, this requires either:
This PR enables a fourth option: inject external catalogs via SPI and run Iceberg's tests unmodified.
New SPI Interfaces
TestTableProvidercore/src/test/java/TestCatalogProviderspark/v3.5/spark/src/test/java/TestSparkSessionProviderspark/v3.5/spark/src/test/java/Modified Test Infrastructure
TableTestBase: Discovers and uses externalTestTableProvidervia SPI or system propertyTestBase: Discovers and uses externalTestSparkSessionProvidervia SPITestBaseWithCatalog: Discovers and uses externalTestCatalogProvidervia SPICatalogTestBase: Respectsiceberg.test.catalog.skip.defaultsto exclude default catalogsNew Gradle Configuration
spark/openhouse.gradle: Shared compatibility test task configuration (reusable template)build.gradle:openhouseCompatibilityaggregate task demonstrating usageUsage Example
External catalogs can implement the SPI interfaces and register them via
META-INF/services/:# Run Iceberg tests against an external catalog ./gradlew :iceberg-spark:iceberg-spark-3.5_2.12:openhouseCompatibilityTest \ :iceberg-core:openhouseCompatibilityTest \ -PopenhouseCompatibilityCoordinate=com.example:my-catalog-fixtures:1.0:uberKey Design Decisions
Test Coverage
This infrastructure has been validated with OpenHouse, running 860+ Iceberg tests including:
TestWapWorkflow(WAP commit workflows)TestFastAppend,TestMergeAppend(append operations)TestOverwrite,TestReplacePartitions(overwrite operations)TestTransaction(multi-operation transactions)TestDeleteFrom,TestPartitionedWrites(Spark SQL tests)Breaking Changes
None. All changes are additive and backward-compatible.