Discussion: splitting up (or extending) the `Catalog` traits tabular methods for tables/views separately

[gruuya]

The iceberg_rust::catalog::Catalog trait currently treats both tables and views as different flavors of the more general tabular (super) type, e.g.

async fn list_tabulars(&self, namespace: &Namespace) -> Result<Vec<Identifier>, Error>;
async fn load_tabular(self: Arc<Self>, identifier: &Identifier) -> Result<Tabular, Error>;

While I find this to be neat/elegant, I do encounter some problems with it occasionally.

Issues

For one thing, note that this is only the case for the above "read" methods, whereas the "write" methods (create/update/delete) are already split up for tables and views.

In addition, it's not aligned with the REST spec, which defines endpoints separately for tables and views. This is also reflected in the java (and python) clients having separate client methods/interfaces for tables and views too.

While relatively benign, this difference can lead to some more important practical limitations:

• there's no way to know which identifier returned by list_tabulars is a table and which one is a view; another call to load_tabular per identifier would be needed to discern that
• since list_tabulars chains identifiers coming from the list-tables and list-views endpoint, having access control that e.g. forbids a client to query views means it won't be able to list tables either (even though it has permissions for it)
• load_tabular does a trial-and-error approach to fetch the tabular—it first tries loading a view and if it fails with error code 404 it tries to load a proper table. Besides the potential above issue with permissions, note that this also means we duplicate the number of API hits per identifier. This in turn can become a problem in a scenario where we're e.g. trying to load all tables in a namespace/catalog periodically (think IMPORT FOREIGN SCHEMA, but for iceberg catalogs), as one more easily gets rate-limited/throttled by the catalog server.

Proposal

If the above issues resonate with people, I can see two potential resolutions here, either

1. split up the tabular methods into table/view ones, or
2. extend the existing interface with table/view specific ones

where the former is simpler, but the later retains backward compatibility.

[JanKaul]

Let me provide my motivation for the current design:

1. load_tabular
   For statements as SELECT * FROM mytest, the query engine doesn't know whether the reference refers to a table or a view. So you would need to perform some kind of trial-and-error approach anyway. I wanted to isolate that logic somewhere and didn't use it all over the codebase, so I moved the trial-and-error logic to the catalog. I also raised a PR at the time to change the REST spec, but without success.
   The reason why the create/update/delete methods are split up is because in these cases the query engine has prior knowledge and knows which kind of entities it's dealing with.

You raise valuable concerns and we can definitely change the catalog trait in order to solve. Let me think if there is a handy solution.

[gruuya]

Thanks, yeah that makes perfect sense. Also agree that the trial-and-error dance is well suited to the catalog.

Still it would be nice if there was an escape hatch to circumvent some of the above issues.

Ideally we'd retain the tabular methods, which would lead to option 2. The tabular methods could then use the split list/load table/view methods internally (effectively preserving the current behavior), while a user could also use one of the granular methods explicitly when needed. Not sure how this would play along with other catalog types, but I figure since java/python clients have this dichotomy it could work.

Either way this is not an urgent issue for us, just wanted to bring it up—we can take some time to think about it.