Skip to content

Add support for REST cache semantics in catalog #268

@gruuya

Description

@gruuya

Iceberg has recently added ETag-based caching support to the REST API, which can be useful to avoid always pulling table metadata if nothing has changed.

We'd like to use this eventually with the (REST) catalog client, so I'm wondering what is the best way to integrate this

  1. at the very minimum this would involve changing catalog_api_api::load_table to support adding additional headers, so that If-None_Match can be provided, along with
  2. extending the results to include the etags in LoadTableResponse and CreateTableResponse, and potentially
  3. (optional) making the catalog freshness-aware by
    a) making RestCatalog internally freshness-aware, by maintaining its own metadata cache (ideally something that can be shared between different instances) and retaining Catalog trait as is, or
    b) extending the Catalog::load_tabular method so that the ETag can be passed at this level, and thus delegating caching to the caller

In principle 1 and 2 are required for someone to build a custom freshness-aware REST catalog, but 3 a or b would be nice to abstract over this and have more flexibility.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions