Iceberg Catalogs & S3 access

[jonsnowseven]

Hello all.

I am doing a PoC with Gravitino.

I noticed that for Iceberg catalogs, we can define access to S3: https://gravitino.apache.org/docs/1.1.0/lakehouse-iceberg-catalog#s3.

• Are there other options for S3 authentication (web identity token)?
• Do we really need to define those S3 configurations or can we do something like:

  gravitino.bypass.iceberg.rest-catalog.vended-credentials-enabled=true
  gravitino.bypass.s3.region=${ENV:AWS_REGION}
  gravitino.bypass.s3.endpoint=${ENV:S3_ENDPOINT}
  gravitino.bypass.s3.sts.region=${ENV:AWS_REGION}
  gravitino.bypass.s3.sts.endpoint=${ENV:STS_ENDPOINT}
  gravitino.bypass.s3.use-web-identity-token-credentials-provider=true
  

  ? I would assume these would be provided to each Iceberg catalog configuration.
• Does Gravitino itself need access to S3? Or only the Iceberg catalog needs it?

I don't know exactly what Gravitino does behind the scenes so I apologize if my questions are nonsense.

Thank you in advance

[FANNG1]

https://gravitino.apache.org/docs/1.1.0/lakehouse-iceberg-catalog/#s3, this define the catalog properties for Iceberg catalog, for the other properties used by Iceberg S3 file io, you could pass by gravitino.bypass.xxx.

Does Gravitino itself need access to S3? Or only the Iceberg catalog needs it?

Only Iceberg catalogs need it.

[FANNG1]

Gravitino Iceberg catalog doesn't support credential vending, IRC server support it, make sure which do you want.

[jonsnowseven]

I will need to define multiple Iceberg catalogs that point to different buckets and DBs and I wanted to use Gravitino as central point of contact for all the Iceberg catalogs.

Can I do that only using credential vending from Iceberg (through gravitino.bypass.) using Gravitino as metadata lake?

[jonsnowseven]

Besides, how does https://gravitino.apache.org/docs/1.1.0/security/credential-vending#credential-vending-for-iceberg-rest-server fit with my question?

[FANNG1]

You could use Iceberg REST server to manage multiple catalogs , https://gravitino.apache.org/docs/1.0.1/iceberg-rest-service#multiple-catalog-backend-support, then you could use credential vending.

[jonsnowseven]

Yes that is what I am looking for!
My additional questions are:

• Configuring multiple catalog backends do not create the respective catalogs, right? If so, how do I create a catalog that maps to a certain backend?
• How can I ensure each backend initializes the db when using JDBC backend?

[FANNG1]

Configuring multiple catalog backends do not create the respective catalogs, right? If so, how do I create a catalog that maps to a certain backend?

what do you mean about create the respective catalogs, IRC supports static and dynamic catalog providers.

How can I ensure each backend initializes the db when using JDBC backend?

There is a flag to control whether to initialize the JDBC backend

[jonsnowseven]

what do you mean about create the respective catalogs, IRC supports static and dynamic catalog providers.

Let's say I have a dynamic catalog providers enabled. When I create multiple catalogs with different backends (using the Gravitino Catalog API), dynamic catalog providers feature will poll these catalogs and register them in Iceberg REST server side so they will be accessible in downstream processes through Iceberg REST?

Now let's say I have static catalog providers enabled. If I have the following:

gravitino.iceberg-rest.catalog-backend = jdbc
gravitino.iceberg-rest.uri = jdbc:postgresql://127.0.0.1:5432
gravitino.iceberg-rest.warehouse = hdfs://127.0.0.1:9000/user/hive/warehouse-postgresql
...
gravitino.iceberg-rest.catalog.hive_backend.catalog-backend = hive
gravitino.iceberg-rest.catalog.hive_backend.uri = thrift://127.0.0.1:9084
gravitino.iceberg-rest.catalog.hive_backend.warehouse = /user/hive/warehouse-hive/
...
gravitino.iceberg-rest.catalog.jdbc_backend.catalog-backend = jdbc
gravitino.iceberg-rest.catalog.jdbc_backend.uri = jdbc:mysql://127.0.0.1:3306/
gravitino.iceberg-rest.catalog.jdbc_backend.warehouse = hdfs://127.0.0.1:9000/user/hive/warehouse-mysql
...

When I launch Gravitino server, what is the effect of having the previous configuration?

There is a flag to control whether to initialize the JDBC backend

Does this flag apply to all backends?