Unable to Join Tables from Different Iceberg Catalogs

[koyoboros]

Hi Jan,

I've successfully configured both REST catalog and file-based catalog in my DataFusion context. Both catalogs work properly when used individually or with other non-Iceberg catalogs. However, when executing a query that joins tables from two different Iceberg catalogs, the operation fails without a specific error. (nothing is returned)

Environment

DataFusion: v46.0.0
Iceberg Rust: latest

Steps to Reproduce

1. Register a REST catalog and a file-based catalog with DataFusion
2. Create a query that joins tables from both catalogs
3. Execute the query

Query example:

SELECT
    posts.id as post_id, items.id as item_id
FROM
    warehouse1.public.posts AS posts CROSS JOIN warehouse2.ns.items AS items;

Questions

Is this a known limitation of the current implementation?
Are there any workarounds available?
Is there a specific way to configure the catalogs to enable cross-catalog joins?

Thank you for your help!

[JanKaul]

It's definitely a limitation of the current implementation and is supposed to work. I have to look deeper into this.

The fact that no error occurs but zero rows are returned might indicate that registering the object stores with datafusion somehow doesn't work.

Are you using the same endpoint/bucket for the data from both catalogs?

[koyoboros]

No. The first one is a file catalog (local), and the second one is a REST Catalog using Lakekeeper and MinIO.