Skip to content

external storage url in tidb cloud #21058

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 42 additions & 2 deletions external-storage-uri.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ The basic format of the URI is as follows:

## Amazon S3 URI format

<CustomContent platform="tidb">

- `scheme`: `s3`
- `host`: `bucket name`
- `parameters`:
Expand Down Expand Up @@ -48,12 +50,42 @@ tiup cdc:v7.5.0 cli changefeed create \
--config=cdc_csv.toml
```

The following is an example of an Amazon S3 URI for [`IMPORT INTO`](/sql-statements/sql-statement-import-into.md). In this example, you need to specify a specific filename `test.csv`.
</CustomContent>

<CustomContent platform="tidb-cloud">

- `scheme`: `s3`
- `host`: `bucket name`
- `parameters`:

- `access-key`: Specifies the access key.
- `secret-access-key`: Specifies the secret access key.
- `session-token`: Specifies the temporary session token. For backup and restore, this parameter is supported starting from v7.6.0.
- `use-accelerate-endpoint`: Specifies whether to use the accelerate endpoint on Amazon S3 (defaults to `false`).
- `endpoint`: Specifies the URL of custom endpoint for S3-compatible services (for example, `<https://s3.example.com/>`).
- `force-path-style`: Use path style access rather than virtual hosted style access (defaults to `true`).
- `storage-class`: Specifies the storage class of the uploaded objects (for example, `STANDARD` or `STANDARD_IA`).
- `sse`: Specifies the server-side encryption algorithm used to encrypt the uploaded objects (value options: empty, `AES256`, or `aws:kms`).
- `sse-kms-key-id`: Specifies the KMS ID if `sse` is set to `aws:kms`.
- `acl`: Specifies the canned ACL of the uploaded objects (for example, `private` or `authenticated-read`).
- `role-arn`: To allow TiDB Cloud to access Amazon S3 data using a specified [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html), specify the corresponding [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) of the IAM role using the `role-arn` URL query parameter, such as `arn:aws:iam::888888888888:role/my-role`. For backup and restore, this parameter is supported starting from v7.6.0.

> **Note:**
>
> - To automatically create an IAM role, navigate to the **Import Data from Amazon S3** page of your cluster in the [TiDB Cloud console](https://tidbcloud.com/), fill in the **Folder URI** field, click **Click here to create new one with AWS CloudFormation** under the **Role ARN** field, and then follow the on-screen instructions in the **Add New Role ARN** dialog.
> - If you have any trouble creating the IAM role using AWS CloudFormation, click **click Having trouble? Create Role ARN manually** in the **Add New Role ARN** dialog to get the TiDB Cloud Account ID and TiDB Cloud External ID, and then follow the steps in [Configure Amazon S3 access using a Role ARN](/tidb-cloud/config-s3-and-gcs-access.md#configure-amazon-s3-access-using-a-role-arn) to create the role manually. When configuring the IAM role, make sure to enter the TiDB Cloud account ID in the **Account ID** field and select **Require external ID** to protect against [confused deputy attacks](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html).
> - To enhance security, you can reduce the valid duration of the IAM role by configuring a shorter **Max session duration**. For more information, see [Update the maximum session duration for a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_update-role-settings.html#id_roles_update-session-duration) in AWS documentation.

- `external-id`: To allow TiDB Cloud to access Amazon S3 data, you must specify the TiDB Cloud External ID using the `external-id` parameter. You can get the TiDB Cloud External ID from the **Add New Role ARN** dialog in the [TiDB Cloud console](https://tidbcloud.com/). For more information, see [Configure Amazon S3 access using a Role ARN](/tidb-cloud/config-s3-and-gcs-access.md#configure-amazon-s3-access-using-a-role-arn).

The following is an example of an Amazon S3 URI for [`BACKUP`](/sql-statements/sql-statement-backup.md) and [`RESTORE`](/sql-statements/sql-statement-restore.md). In this example, you need to specify a specific file path `testfolder`.

```shell
s3://external/test.csv?access-key=${access-key}&secret-access-key=${secret-access-key}
s3://external/testfolder?access-key=${access-key}&secret-access-key=${secret-access-key}
```

</CustomContent>

## GCS URI format

- `scheme`: `gcs` or `gs`
Expand All @@ -64,18 +96,26 @@ s3://external/test.csv?access-key=${access-key}&secret-access-key=${secret-acces
- `storage-class`: Specifies the storage class of the uploaded objects (for example, `STANDARD` or `COLDLINE`)
- `predefined-acl`: Specifies the predefined ACL of the uploaded objects (for example, `private` or `project-private`)

<CustomContent platform="tidb">

The following is an example of a GCS URI for TiDB Lightning and BR. In this example, you need to specify a specific file path `testfolder`.

```shell
gcs://external/testfolder?credentials-file=${credentials-file-path}
```

</CustomContent>

<CustomContent platform="tidb-cloud">

The following is an example of a GCS URI for [`IMPORT INTO`](/sql-statements/sql-statement-import-into.md). In this example, you need to specify a specific filename `test.csv`.

```shell
gcs://external/test.csv?credentials-file=${credentials-file-path}
```

</CustomContent>

## Azure Blob Storage URI format

- `scheme`: `azure` or `azblob`
Expand Down
10 changes: 0 additions & 10 deletions sql-statements/sql-statement-backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,18 +112,8 @@ BR supports backing up data to S3 or GCS:
BACKUP DATABASE `test` TO 's3://example-bucket-2020/backup-05/?access-key={YOUR_ACCESS_KEY}&secret-access-key={YOUR_SECRET_KEY}';
```

<CustomContent platform="tidb">

The URL syntax is further explained in [URI Formats of External Storage Services](/external-storage-uri.md).

</CustomContent>

<CustomContent platform="tidb-cloud">

The URL syntax is further explained in [external storage URI](https://docs.pingcap.com/tidb/stable/external-storage-uri).

</CustomContent>

When running on cloud environment where credentials should not be distributed, set the `SEND_CREDENTIALS_TO_TIKV` option to `FALSE`:

{{< copyable "sql" >}}
Expand Down
10 changes: 0 additions & 10 deletions sql-statements/sql-statement-restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,18 +103,8 @@ BR supports restoring data from S3 or GCS:
RESTORE DATABASE * FROM 's3://example-bucket-2020/backup-05/';
```

<CustomContent platform="tidb">

The URL syntax is further explained in [URI Formats of External Storage Services](/external-storage-uri.md).

</CustomContent>

<CustomContent platform="tidb-cloud">

The URL syntax is further explained in [external storage URI](https://docs.pingcap.com/tidb/stable/external-storage-uri).

</CustomContent>

When running on cloud environment where credentials should not be distributed, set the `SEND_CREDENTIALS_TO_TIKV` option to `FALSE`:

{{< copyable "sql" >}}
Expand Down