Fix Unity Catalog and Iceberg REST endpoint construction #5904
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix multiple endpoint construction issues for Unity Catalog integration:
Unity Catalog endpoint construction:
- Add catalog name fallback when /v1/config has no prefix
- Fix catalog name resolution for 2-part table names (use session catalog)
- Fix double /v1/ in plan endpoint URL
- Add ?implementation=MATERIALIZED_PARQUET query parameter
These fixes ensure Unity Catalog endpoints are constructed correctly:
{base}/v1/catalogs/{catalog}/namespaces/{db}/tables/{table}/plan?implementation=MATERIALIZED_PARQUET
- Move comment about optional prefix to the fallback case - Fix indentation - Simplify comment about session catalog usage
- Fix fallback test: now expects catalog name in default prefix - Add test to verify no double /v1/ in endpoint URL - Add test for 2-part table name resolution (uses session catalog) - Add test for 3-part table name resolution (uses explicit catalog) These tests ensure the endpoint construction changes work correctly and improve test coverage for Unity Catalog integration.
- Remove duplicate /v1/ count check (already validated by path structure check) - Simplify comment about session catalog usage
Simplify test to use the default spark_catalog instead of trying to create a custom catalog which requires additional setup. The test now verifies that for 2-part identifiers, the code correctly uses the session's current catalog instead of hardcoding "spark_catalog".
Assert on all metadata fields constructed from the identifier: - catalogName (from session for 2-part, from identifier for 3-part) - ucUri (read from config based on catalog name) - ucToken (read from config based on catalog name) - tableProperties (should be empty) This ensures the metadata object is correctly constructed from both 2-part and 3-part identifiers.
Consolidate two tests into one parameterized test that only verifies catalog name resolution: - 2-part identifier (schema.table) → uses session catalog - 3-part identifier (catalog.schema.table) → uses explicit catalog Removed unnecessary assertions on ucUri, ucToken, and tableProperties to focus the test on the core catalog name resolution logic.
Reorder parameters for better readability: - description (first) - input (identifier) - expectedOutput (catalog name) This makes test cases more self-documenting and follows a logical description → input → expected pattern.
The UnityCatalogMetadata test now expects the endpoint to include
/v1/catalogs/{catalogName} when the /v1/config endpoint has no prefix.
This aligns with the implemented catalog name fallback behavior that
ensures Unity Catalog endpoints are constructed correctly even without
explicit prefix configuration.
The test validates that when /v1/config is unreachable, the system
falls back to constructing the endpoint as:
{base}/api/2.1/unity-catalog/iceberg-rest/v1/catalogs/test_catalog
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Changes
IcebergRESTCatalogPlanningClient.scala
Fixed the double
/v1/issue in the plan endpoint URL. TheicebergRestCatalogUriRootalready contains/v1/from UnityCatalogMetadata, so adding another/v1/was causing malformed URLs.UnityCatalogMetadata.scala
Catalog name fallback: When the
/v1/configendpoint has no prefix, construct one from the catalog name following Unity Catalog's required format:{base}/v1/catalogs/{catalog}/namespaces/{db}/tables/{table}/plan2-part table name resolution: For 2-part table names (
schema.table), use the current session catalog instead of hardcoding"spark_catalog", which allows queries to work correctly with Unity Catalog.