br: disable restore checksum #20847
br: disable restore checksum #20847
Conversation
Signed-off-by: Wenqi Mou <[email protected]>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hello @Tristan1900, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
Summary of Changes
This pull request updates the documentation for the TiDB Backup & Restore (BR) tool, specifically focusing on clarifying the behavior and configuration of checksums during backup and restore operations. The changes aim to provide users with a better understanding of the performance and data integrity trade-offs associated with enabling or disabling table-level checksums. The documentation is updated to reflect the default behavior of disabling table-level checksums in recent versions to improve performance, while emphasizing the option to enable them for stricter data validation.
Highlights
- Checksum Documentation: A new 'Checksum' section is added to
br/br-snapshot-manual.mdto explain file-level and table-level checksums, their impact on performance, and configuration examples. - Default Checksum Behavior: The documentation is updated to reflect that table-level checksums are disabled by default in recent BR versions (v8.5.0 for backup, v9.5.0 for restore) to improve performance.
- PITR Documentation Update: The
br/br-pitr-manual.mdfile is updated to include a reference to the new Checksum documentation. - Command-Line Tool Documentation Update: The
br/use-br-command-line-tool.mdfile is updated to include the--checksumoption and a reference to the Checksum documentation.
Changelog
- br/br-pitr-manual.md
- Added a reference to the Checksum documentation in
br-snapshot-manual.md.
- Added a reference to the Checksum documentation in
- br/br-snapshot-manual.md
- Added a new 'Checksum' section explaining file-level and table-level checksums.
- Removed the note about disabling table-level checksums by default in v8.5.0 from the general notes section, as this is now covered in the Checksum section.
- Updated the description of the restore process to clarify when table-level checksum verification is performed.
- br/use-br-command-line-tool.md
- Added the
--checksumoption to the list of command-line flags and linked to the Checksum documentation.
- Added the
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
ti-chi-bot
bot
added
missing-translation-status
There was a problem hiding this comment.
Code Review
The pull request introduces changes to the documentation for the BR tool, specifically focusing on disabling checksums by default and providing more information about checksum configuration. The changes seem well-organized and provide useful information to the user. However, there are a few areas where the clarity and consistency can be improved.
Summary of Findings
- Clarity and Consistency: Ensure consistent terminology and clear explanations throughout the documentation, especially when introducing new concepts like checksum levels. Also, ensure that the documentation is easy to understand for TiDB users.[^1]
- Sentence Structure and Flow: Improve sentence structure and logical flow to enhance readability and comprehension. Make sure the documentation is easy to understand for TiDB users.[^1]
- Markdown Style: Ensure adherence to markdown style guidelines, including proper heading hierarchy and spacing. Add a blank line before and after headings and lists.[^2]
Merge Readiness
The pull request is almost ready for merging. Addressing the comments related to clarity, consistency, and markdown style will further improve the quality of the documentation. I am unable to approve the pull request, and users should have others review and approve this code before merging. I recommend addressing the comments before merging.
|
|
||
| ## Checksum | ||
|
|
||
| Checksum is a method used by the BR tool to verify the integrity of backup and restore data. BR supports two levels of checksums: |
There was a problem hiding this comment.
It might be helpful to add a brief introductory sentence before defining checksum to provide context. Make sure the documentation is easy to understand for TiDB users.1
Style Guide References
| Checksum is a method used by the BR tool to verify the integrity of backup and restore data. BR supports two levels of checksums: | |
| ## Checksum | |
| To ensure data integrity, BR uses checksum, a method to verify the integrity of backup and restore data. BR supports two levels of checksums: |
Footnotes
|
@gemini-code-assist[bot]: adding LGTM is restricted to approvers and reviewers in OWNERS files. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
| Checksum is a method used by the BR tool to verify the integrity of backup and restore data. BR supports two levels of checksums: | ||
|
|
||
| 1. **File-level checksum**: Verifies the backup files themselves to ensure integrity during storage and transmission. This checksum is always enabled and cannot be disabled. | ||
| 2. **Table-level checksum**: Verifies the integrity of table data content and confirms the business logic consistency of the data. This checksum is disabled by default but can be enabled through parameters. |
There was a problem hiding this comment.
| 2. **Table-level checksum**: Verifies the integrity of table data content and confirms the business logic consistency of the data. This checksum is disabled by default but can be enabled through parameters. | |
| 2. **Table-level checksum**: Verifies the integrity of table data content and confirms the business logic consistency of the data. You can choose to enable or disable this checksum. |
There was a problem hiding this comment.
hmm, I feel like it will not be clear to customer that we disable it by default and they might think it's still enabled?
|
@BornChanger: adding LGTM is restricted to approvers and reviewers in OWNERS files. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
lilin90
added
the
translation/from-docs-cn
ti-chi-bot
bot
removed
the
missing-translation-status
|
/bot-review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -409,7 +409,8 @@ Expected output: | |||
|
|
|||
| > **Note:** | |||
| > | |||
| > If you specify `--full-backup-storage` as the incremental backup address for `restore point`, for restores of this backup and any previous incremental backups, you need to set the parameter `--allow-pitr-from-incremental` to `true` to make the incremental backups compatible with the subsequent log backups. | |||
| > - If you specify `--full-backup-storage` as the incremental backup address for `restore point`, for restores of this backup and any previous incremental backups, you need to set the parameter `--allow-pitr-from-incremental` to `true` to make the incremental backups compatible with the subsequent log backups. | |||
There was a problem hiding this comment.
The sentence is too long. Please split it into several sentences for better readability.
| > - If you specify `--full-backup-storage` as the incremental backup address for `restore point`, for restores of this backup and any previous incremental backups, you need to set the parameter `--allow-pitr-from-incremental` to `true` to make the incremental backups compatible with the subsequent log backups. | |
| > - If you specify `--full-backup-storage` as the incremental backup address for `restore point`, you need to set the parameter `--allow-pitr-from-incremental` to `true`. This setting is necessary for restores of this backup and any previous incremental backups. It ensures that the incremental backups are compatible with the subsequent log backups. |
| @@ -48,7 +49,6 @@ In the preceding command: | |||
|
|
|||
| > **Note:** | |||
| > | |||
There was a problem hiding this comment.
The note about disabling the table-level checksum calculation during full backups should be retained for clarity and completeness.
| > | |
| > - Starting from v8.5.0, the BR tool disables the table-level checksum calculation during full backups by default (`--checksum=false`) to improve backup performance. The BR tool already supports self-adapting to GC. It automatically registers `backupTS` (the latest PD timestamp by default) to PD's `safePoint` to ensure that TiDB's GC Safe Point does not move forward during the backup, thus avoiding manually setting GC configurations. |
| @@ -178,7 +178,7 @@ | |||
| - `--ratelimit`: The maximum speed **per TiKV** performing restore tasks. The unit is in MiB/s. | |||
| - `--log-file`: The target file where the `br` log is written. | |||
|
|
|||
| During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. Then `br` will verify the restored data to ensure data security. | |||
| During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. After the restoration is complete, if table-level checksum is enabled (see [Checksum](#checksum)), the BR tool performs table data verification to ensure the logical integrity of the data. File-level checksums are always performed to ensure the basic integrity of the restored files. | |||
There was a problem hiding this comment.
The sentence is too long. Please split it into several sentences for better readability.
| During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. After the restoration is complete, if table-level checksum is enabled (see [Checksum](#checksum)), the BR tool performs table data verification to ensure the logical integrity of the data. File-level checksums are always performed to ensure the basic integrity of the restored files. | |
| During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. After the restoration is complete, if table-level checksum is enabled (see [Checksum](#checksum)), the BR tool performs table data verification. This ensures the logical integrity of the data. File-level checksums are always performed to ensure the basic integrity of the restored files. |
|
|
||
| ## Checksum | ||
|
|
||
| Checksum is a method used by the BR tool to verify the integrity of backup and restore data. BR supports two levels of checksums: |
There was a problem hiding this comment.
The sentence is too long. Please split it into several sentences.
| Checksum is a method used by the BR tool to verify the integrity of backup and restore data. BR supports two levels of checksums: | |
| Checksum is a method used by the BR tool to verify the integrity of backup and restore data. BR supports two levels of checksums. These are file-level checksum and table-level checksum. |
| 1. **File-level checksum**: Verifies the backup files themselves to ensure integrity during storage and transmission. This checksum is always enabled and cannot be disabled. | ||
| 2. **Table-level checksum**: Verifies the integrity of table data content and confirms the business logic consistency of the data. This checksum is disabled by default but can be enabled through parameters. | ||
|
|
||
| Balancing performance and security considerations, BR handles table-level checksums as follows: |
There was a problem hiding this comment.
The sentence is too long. Please split it into several sentences.
| Balancing performance and security considerations, BR handles table-level checksums as follows: | |
| Balancing performance and security considerations, BR handles table-level checksums as follows. The following sections describe backup checksum and restore checksum. |
|
|
||
| ### Backup Checksum | ||
|
|
||
| Starting from v8.5.0, when performing full backups, the BR tool does not calculate table-level checksums by default (`--checksum=false`) to improve backup performance. If you need to calculate table-level checksums during backup, you can explicitly specify `--checksum=true`. File-level checksums will always be calculated to ensure the integrity of backup files. |
There was a problem hiding this comment.
The sentence is too long. Please split it into several sentences.
| Starting from v8.5.0, when performing full backups, the BR tool does not calculate table-level checksums by default (`--checksum=false`) to improve backup performance. If you need to calculate table-level checksums during backup, you can explicitly specify `--checksum=true`. File-level checksums will always be calculated to ensure the integrity of backup files. | |
| Starting from v8.5.0, when performing full backups, the BR tool does not calculate table-level checksums by default (`--checksum=false`). This is done to improve backup performance. If you need to calculate table-level checksums during backup, you can explicitly specify `--checksum=true`. File-level checksums will always be calculated to ensure the integrity of backup files. |
|
|
||
| ### Restore Checksum | ||
|
|
||
| Starting from v9.0.0, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default during restore operations to improve restore performance. If you need to perform table-level checksum verification, you can explicitly specify `--checksum=true`. File-level checksum verification is always performed to ensure the basic integrity of restored data. |
There was a problem hiding this comment.
The sentence is too long. Please split it into several sentences.
| Starting from v9.0.0, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default during restore operations to improve restore performance. If you need to perform table-level checksum verification, you can explicitly specify `--checksum=true`. File-level checksum verification is always performed to ensure the basic integrity of restored data. | |
| Starting from v9.0.0, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default during restore operations. This is done to improve restore performance. If you need to perform table-level checksum verification, you can explicitly specify `--checksum=true`. File-level checksum verification is always performed to ensure the basic integrity of restored data. |
|
|
||
| Starting from v9.0.0, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default during restore operations to improve restore performance. If you need to perform table-level checksum verification, you can explicitly specify `--checksum=true`. File-level checksum verification is always performed to ensure the basic integrity of restored data. | ||
|
|
||
| After restoration, data validation is usually performed to ensure data security. When table-level checksums are disabled, the comprehensive validation step for table data is skipped, thereby accelerating the restore process. For scenarios with strict data integrity requirements, you may choose to enable table-level checksums. |
There was a problem hiding this comment.
The sentence is too long. Please split it into several sentences.
| After restoration, data validation is usually performed to ensure data security. When table-level checksums are disabled, the comprehensive validation step for table data is skipped, thereby accelerating the restore process. For scenarios with strict data integrity requirements, you may choose to enable table-level checksums. | |
| After restoration, data validation is usually performed to ensure data security. When table-level checksums are disabled, the comprehensive validation step for table data is skipped. This accelerates the restore process. For scenarios with strict data integrity requirements, you may choose to enable table-level checksums. |
| @@ -62,6 +62,7 @@ A `tiup br` command consists of multiple layers of sub-commands. Currently, br c | |||
| * `--tikv-max-restore-concurrency`: the maximum number of concurrent tasks per TiKV node during snapshot restore. | |||
| * `--compression`: determines the compression algorithm used for generating backup files. It supports `lz4`, `snappy`, and `zstd`, with the default being `zstd` (usually no need to modify). For guidance on choosing different compression algorithms, refer to [this document](https://github.com/EighteenZi/rocksdb_wiki/blob/master/Compression.md). | |||
| * `--compression-level`: sets the compression level corresponding to the chosen compression algorithm for backup. The default compression level for `zstd` is 3. In most cases there is no need to set this option. | |||
| * `--checksum`: controls whether to perform table-level checksum verification during backup and restore. Default is `false`. For more details, refer to [Checksum](/br/br-snapshot-manual.md#checksum). | |||
There was a problem hiding this comment.
The explanation of the default value is missing. It is recommended to clarify the default behavior and its implications.
| * `--checksum`: controls whether to perform table-level checksum verification during backup and restore. Default is `false`. For more details, refer to [Checksum](/br/br-snapshot-manual.md#checksum). | |
| * `--checksum`: controls whether to perform table-level checksum verification during backup and restore. The default is `false`, meaning that checksum verification is not performed unless explicitly enabled. For more details, refer to [Checksum](/br/br-snapshot-manual.md#checksum). |
|
✅ AI review completed, 9 comments generated. |
lilin90
added
the
area/br
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
First-time contributors' checklist
What is changed, added or deleted? (Required)
Chinese version
pingcap/docs-cn#20256
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
Do your changes match any of the following descriptions?