diff --git a/ingestion/sources/iceberg-config.mdx b/ingestion/sources/iceberg-config.mdx index 913d3bb5..8877814c 100644 --- a/ingestion/sources/iceberg-config.mdx +++ b/ingestion/sources/iceberg-config.mdx @@ -109,6 +109,22 @@ These parameters configure the connection to the underlying GCS storage system w | `warehouse.path` | The base path to your Iceberg warehouse in GCS. Example: `'gs://bucket/path'` | Conditional | | `gcs.credential` | The credential to access GCS. | Conditional | +## Storage configuration (Azure Blob) + +These parameters configure the connection to Azure Blob storage where the Iceberg data files are stored. + + +Added in version 2.4. + + +| Parameter name | Description | Required (Conditional) | +| :------------- | :---------- | :---------- | +| `warehouse.path` | Specifies the Azure Blob Storage path. Example: `'azblob:///...'`. | Conditional | +| `azblob.container_name` | The name of the Azure Blob Storage container.| Conditional | +| `azblob.credentials.account_name` | The Azure Storage account name used to authenticate access to the container. | Conditional | +| `azblob.credentials.account_key` | The Azure Storage account key associated with the account name, used for authentication. | Conditional | +| `azblob.endpoint_url` | The full endpoint URL of the Azure Blob service. | Conditional | + ## Catalog configuration These parameters configure the Iceberg catalog. The catalog is responsible for managing table metadata (schema, partitioning, location). RisingWave supports several catalog types. @@ -118,7 +134,7 @@ These parameters configure the Iceberg catalog. The catalog is responsible for m | `catalog.name` | The name of the Iceberg catalog. This is a user-defined identifier. *Optional* for the `storage` catalog, but *required* for all other catalog types. | Conditional | | `catalog.type` | The type of Iceberg catalog to use. Supported values: If not specified, defaults to `'storage'`. | No (defaults to `storage`) | | `catalog.uri` | The URI of the catalog. The required format depends on the `catalog.type`: | Conditional | -|`warehouse.path`| The path of the Iceberg warehouse. Currently, only S3-compatible object storage systems and GCS, are supported. It is required if the `catalog.type` is not `rest`. | Conditional| +|`warehouse.path`| The path of the Iceberg warehouse. It is required if the `catalog.type` is not `rest`. | Conditional| | `catalog.credential` | Credential for accessing the Iceberg catalog, used to exchange for a token in the OAuth2 client credentials flow. Applicable only in the `rest` catalog. | No | | `catalog.token` | A Bearer token for accessing the Iceberg catalog, used for interaction with the server. Applicable only in the `rest` catalog. | No | | `catalog.oauth2_server_uri` | The `oauth2_server_uri` for accessing the Iceberg catalog, serving as the token endpoint URI to fetch a token if the `rest` catalog is not the authorization server. Applicable only in the `rest` catalog. | No | diff --git a/integrations/destinations/apache-iceberg.mdx b/integrations/destinations/apache-iceberg.mdx index a304d136..af590f0e 100644 --- a/integrations/destinations/apache-iceberg.mdx +++ b/integrations/destinations/apache-iceberg.mdx @@ -26,7 +26,7 @@ WITH ( ### Basic parameters -| Parameter Names | Description | +| Parameter name | Description | | :------------- | :---------- | | type | Required. Allowed values: `append-only` and `upsert`. | | force_append_only | Optional. If true, forces the sink to be append-only, even if it cannot be. | @@ -39,7 +39,7 @@ WITH ( | table.name | Required. The name of the target Iceberg table. | | catalog.name | Conditional. The name of the Iceberg catalog. It can be omitted for storage catalog but required for other catalogs. | | catalog.type | Optional. The catalog type used in this table. Currently, the supported values are storage, rest, hive, jdbc, and glue. If not specified, storage is used. For details, see [Catalogs](#catalogs). | -| warehouse.path | Conditional. The path of the Iceberg warehouse. Currently, only S3-compatible object storage systems, such as AWS S3 and MinIO, are supported. It's required if the `catalog.type` is not rest. | +| warehouse.path | Conditional. The path of the Iceberg warehouse. It's required if the `catalog.type` is not rest. | | catalog.url | Conditional. The URL of the catalog. It is required when `catalog.type` is not storage. | | primary_key | The primary key for an upsert sink. It is only applicable to the upsert mode. | | partition_by | Optional. Specify partitioning using column names or transformations. Supported formats include: `column`, `transform(column)`, `transform(n,column)`, and `transform(n, column)`. Transformations can include functions like `bucket` or `truncate`, where `n` is an optional parameter. Ensure that the specified partition fields exist in the schema. | @@ -50,6 +50,70 @@ WITH ( | catalog.oauth2_server_uri | Optional. The `oauth2_server_uri` for accessing the Iceberg catalog, serving as the token endpoint URI to fetch a token if the `rest` catalog is not the authorization server. Applicable only in the `rest` catalog. | | catalog.scope | Optional. Scope for accessing the Iceberg catalog, providing additional scope for OAuth2. Applicable only in the `rest` catalog. | +### Use GCS as object storage for Iceberg + + +Added in version 2.3. + + +RisingWave supports creating Iceberg sinks on GCS with catalog types `storage` or `rest`. To enable it, configure the following specific parameters: + +| Parameter name | Description | +| :------------- | :---------- | +| warehouse.path | Specifies the Google Cloud Storage path. | +| gcs.credential | Base64-encoded credential key obtained from the GCS service account key JSON file. To get this JSON file, refer to the [guides of GCS documentation](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console). | + +```sql Example +CREATE SINK sink_t FROM t WITH ( + connector = 'iceberg', + type = 'append-only', + force_append_only = 'true', + database.name = 'public', + table.name = 't', + catalog.name = 'demo', + catalog.type = 'rest', + catalog.uri = 'http://127.0.0.1:8181', + warehouse.path = 'gs://bucket/path', + gcs.credential = 'xxxxxxxx' +); +``` +For more details on sinking data to GCS, see [Sink data to Google Cloud Storage](/integrations/destinations/google-cloud-storage). + +### Use Azure Blob as object storage for Iceberg + + +Added in version 2.4. + + +RisingWave supports creating Iceberg sinks on Azure Blob with catalog types `storage` or `rest`. To enable it, configure the following specific parameters: + +| Parameter name | Description | +| :------------- | :---------- | +| warehouse.path | Specifies the Azure Blob Storage path. | +| azblob.container_name | The name of the Azure Blob Storage container.| +| azblob.credentials.account_name | The Azure Storage account name used to authenticate access to the container. | +| azblob.credentials.account_key | The Azure Storage account key associated with the account name, used for authentication. | +| azblob.endpoint_url | The full endpoint URL of the Azure Blob service. | + +```sql Example +CREATE SINK sink1 AS select * from mv1 WITH ( + connector = 'iceberg', + type = 'append-only', + database.name = 'demo_db', + table.name = 'test_azblob_iceberg', + catalog.name = 'demo', + catalog.type = 'storage', + warehouse.path = 'azblob:///demo', + azblob.container_name = 'xxx', + azblob.credentials.account_name = 'xxx', + azblob.credentials.account_key = 'xxx', + azblob.endpoint_url = 'https://.blob.core.windows.net/', + ... +); +``` + +For more details on sinking data to Azure Blob, see [Sink data to Azure Blob](/integrations/destinations/azure-blob). + ### Use Amazon S3 Tables with the Iceberg sink You can configure the RisingWave Iceberg sink connector to use Amazon S3 Tables as its catalog. This setup allows RisingWave to sink data into Iceberg tables managed by the AWS native S3 Tables catalog service. @@ -58,15 +122,13 @@ To achieve this, specify the `rest` catalog type within your `CREATE SINK` state **Required REST Catalog Parameters for S3 Tables:** -| Parameter Name | Description | Value for S3 Tables | +| Parameter name | Description | Value for S3 Tables | | :---------------------------- | :--------------------------------------------------------------- | :------------------ | | `catalog.rest.signing_region` | The AWS region for signing REST catalog requests. | e.g., `us-east-1` | | `catalog.rest.signing_name` | The service name for signing REST catalog requests. | `s3tables` | | `catalog.rest.sigv4_enabled` | Enables SigV4 signing for REST catalog requests. Set to `true`. | `true` | -**Example `CREATE SINK` Statement:** - -```sql +```sql Example CREATE SINK my_s3_tables_sink FROM source_table WITH ( connector = 'iceberg', @@ -388,25 +450,4 @@ WITH ( ); ``` -### Iceberg sink on GCS - - -Added in version 2.3. - - -RisingWave supports creating Iceberg sinks on GCS with catalog types `storage` or `rest`. For more information about `gcs.credential`, see [parameters](/integrations/destinations/google-cloud-storage#parameters). -```sql -CREATE SINK sink_t FROM t WITH ( - connector = 'iceberg', - type = 'append-only', - force_append_only = 'true', - database.name = 'public', - table.name = 't', - catalog.name = 'demo', - catalog.type = 'rest', - catalog.uri = 'http://127.0.0.1:8181', - warehouse.path = 'gs://bucket/path', - gcs.credential = 'xxxxxxxx' -); -```