Setting up Iceberg REST server with Apache Gravitino with JDBC backend and storage GCS

[vishnu-chalil]

Hi Gravitino community,

I was looking to deploy gravitino rest server with JDBC backend. I noticed that there is s3 and HDFS configuration in the Helm chart but the configurations for GCS and other storages is missing. Is it possible to currently use GCS as storage with JDBC rest catalog or the just the documentation is yet to be updated? Thanks in advance.

[FANNG1]

The helm chart for IRC doesn't support GCS storage yet, tracked in #7159

[vishnu-chalil]

@FANNG1 Here there seems to be updated on Helm for support on most cloud storages.

  # warehouse: s3://iceberg-test/iceberg-docker
  ## Name of the Iceberg catalog backend
  ##
  # catalogBackendName: memory
  ## URI for connecting to the catalog backend (e.g., Hive Metastore Thrift URI)
  ##
  # uri: thrift://localhost:9083
  ## JDBC connection configuration for jdbc catalog backend (if applicable)
  ##
  # jdbc:
  #   user: "user name"
  #   password: "password"
  #   ## Whether to initialize the Iceberg meta tables in RDBMS
  #   ##
  #   initialize: True
  #   ## JDBC driver class name
  #   ##
  #   driver: "com.mysql.cj.jdbc.Driver"
  ## Implementation class for Iceberg file I/O operations
  ##
  # ioImpl: "org.apache.iceberg.aws.s3.S3FileIO"
  ## Comma-separated list of credential providers.
  ##
  # credentialProviders: "s3-token"
  ## S3 storage configuration
  ##
  # s3:
  #   accessKeyId: ""
  #   secretAccessKey: ""
  #   ## Whether to use path-style access instead of virtual hosted-style access
  #   ##
  #   pathStyleAccess:
  #   roleArn:
  #   endpoint:
  #   region:
  #   externalId:
  #   tokenServiceEndpoint:
  ## OSS storage configuration
  ##
  # oss:
  #   accessKeyId: ""
  #   secretAccessKey: ""
  #   endpoint: ""
  #   region: ""
  #   roleArn:
  #   externalId:
  ## Azure Blob Storage configuration
  ##
  # azure:
  #   storageAccountName: ""
  #   storageAccountKey: ""
  #   tenantId: ""
  #   clientId: ""
  #   clientSecret: ""
  ## No need to configure extra Google Cloud Storage configuration
  ##
  ## Catalog configuration provider class name
  ##
  # catalogConfigProvider: "static-config-provider"
  ## Dynamic configuration provider settings
  ##
  # dynamicConfigProvider:
  #   ## URI of the Gravitino server
  #   ##
  #   uri: "http://localhost:8090"
  #   ## Name of the metalake
  #   ##
  #   metalake: "test"
  #   ## Default catalog name for operations
  #   ##

Here it is mentioned No need to configure extra Google Cloud Storage configuration. Does this mean GCS is yet to be supported?

[vishnu-chalil]

Thank you @FANNG1 . This works. But this also means that I won't be able to use the same rest server in different metastores without sharing the details? Also is there any option to configure or set up multiple REST catalog server with gravitino IRC itself?

[FANNG1]

You mean using multiple catalog backends? If yes, please refer to https://gravitino.apache.org/docs/0.9.0-incubating/iceberg-rest-service#multi-catalog-support

[vishnu-chalil]

I meant more like configuring same instances of Iceberg REST server. Suppose say I implemented a JDBC Gravitino IRC with postgres and gcs, now I can configure it in one of the catalogs in a metastore. If I want to use the same rest server in another metastore but doesn't want to allow access or visbility of any of the entities I created as part of the inital metastore. Looking at the doc what I realized is I need to have multiple IRC setup and use seperate instances of it. What I wanted to know is is it possible to use same Gravitino IRC across metastores but with segregation of metadata used in different metastores.

[FANNG1]

Could you provide more context about metastore?

[vishnu-chalil]

Sorry. I meant the metalake in gravitino