-
Notifications
You must be signed in to change notification settings - Fork 35
Open
Description
Iceberg has recently added ETag-based caching support to the REST API, which can be useful to avoid always pulling table metadata if nothing has changed.
We'd like to use this eventually with the (REST) catalog client, so I'm wondering what is the best way to integrate this
- at the very minimum this would involve changing
catalog_api_api::load_tableto support adding additional headers, so thatIf-None_Matchcan be provided, along with - extending the results to include the
etags inLoadTableResponseandCreateTableResponse, and potentially - (optional) making the catalog freshness-aware by
a) making RestCatalog internally freshness-aware, by maintaining its own metadata cache (ideally something that can be shared between different instances) and retainingCatalogtrait as is, or
b) extending theCatalog::load_tabularmethod so that the ETag can be passed at this level, and thus delegating caching to the caller
In principle 1 and 2 are required for someone to build a custom freshness-aware REST catalog, but 3 a or b would be nice to abstract over this and have more flexibility.
JanKaul
Metadata
Metadata
Assignees
Labels
No labels