Skip to content

br: pitr filter feature release doc #21109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 93 additions & 3 deletions br/br-pitr-manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -496,6 +496,84 @@ tiup br restore point --pd="${PD_IP}:2379"
--master-key "local:///path/to/master.key"
```

### Restore with filters

Starting from TiDB v9.0.0, you can use filters during PITR to selectively restore specific databases or tables. This allows for more granular control over what data gets restored during point-in-time recovery operations.


Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

There appears to be an extra blank line here (line 503, following line 502 which is also blank). Typically, one blank line is sufficient to separate paragraphs in Markdown. Could you remove one of them for conciseness and standard formatting? 1

Style Guide References

Footnotes

  1. Style guide advises avoiding unnecessary words and repetition, which can be extended to unnecessary spacing for conciseness. (link)

The filter patterns follow the same syntax as [table filters](/table-filter.md) used in other BR operations:

- `'*.*'` - matches all databases and tables
- `'db1.*'` - matches all tables in database `db1`
- `'db1.table1'` - matches specific table `table1` in database `db1`
- `'db*.tbl*'` - matches databases starting with `db` and tables starting with `tbl`
- `'!mysql.*'` - excludes all tables in the `mysql` database

Usage examples:

```shell
# restore specific databases
tiup br restore point --pd="${PD_IP}:2379" \
--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \
--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \
--start-ts "2025-06-02 00:00:00+0800" \
--restored-ts "2025-06-03 18:00:00+0800" \
--filter 'db1.*' --filter 'db2.*'

# restore specific tables
tiup br restore point --pd="${PD_IP}:2379" \
--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \
--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \
--start-ts "2025-06-02 00:00:00+0800" \
--restored-ts "2025-06-03 18:00:00+0800" \
--filter 'db1.users' --filter 'db1.orders'

# restore with pattern matching
tiup br restore point --pd="${PD_IP}:2379" \
--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \
--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \
--start-ts "2025-06-02 00:00:00+0800" \
--restored-ts "2025-06-03 18:00:00+0800" \
--filter 'db*.tbl*'
```

> **Note:**
>
> - When using filters, ensure that the filtered data maintains referential integrity.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

To align with the style guide's preference for addressing users in the second person 1, consider rephrasing this sentence.

Style Guide References

Suggested change
> - When using filters, ensure that the filtered data maintains referential integrity.
When you use filters, ensure that the filtered data maintains referential integrity.

Footnotes

  1. The style guide recommends writing in the second person ('you') when addressing users. (link)

> - Filter options apply to both snapshot and log backup restoration phases.
> - Multiple `--filter` options can be specified to include or exclude different patterns.
> - PITR filtering does not support restoring system tables. System tables are restored according to the full backup data and cannot be selectively filtered during PITR operations. If you need to restore specific system tables, use `br restore full` with filters instead, which will restore only the snapshot backup data (not log backup data).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This bullet point contains a lot of information in a single sentence, which can make it a bit dense. For improved readability 1, would you consider rephrasing it slightly or breaking it into more digestible parts within the bullet point?

Style Guide References

Suggested change
> - PITR filtering does not support restoring system tables. System tables are restored according to the full backup data and cannot be selectively filtered during PITR operations. If you need to restore specific system tables, use `br restore full` with filters instead, which will restore only the snapshot backup data (not log backup data).
PITR filtering does not support restoring system tables. System tables are restored based on the full backup data and cannot be selectively filtered during PITR operations. If you need to restore specific system tables, use `br restore full` with filters. This command restores only the snapshot backup data, not the log backup data.

Footnotes

  1. The style guide emphasizes clarity, simplicity, and readability. (link)

> - The target databases and tables specified in the filter must not exist in the cluster, or the restore will fail with an error.

### Concurrent restore operations

Starting from TiDB v9.0.0, you can run multiple PITR operations concurrently. This feature allows you to restore different datasets simultaneously, improving restore efficiency for large-scale operations.

Usage example for concurrent restores:

```shell
# terminal 1 - restore database db1
tiup br restore point --pd="${PD_IP}:2379" \
--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \
--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \
--start-ts "2025-06-02 00:00:00+0800" \
--restored-ts "2025-06-03 18:00:00+0800" \
--filter 'db1.*'

# terminal 2 - restore database db2 (can run simultaneously)
tiup br restore point --pd="${PD_IP}:2379" \
--storage='s3://backup-101/logbackup?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \
--full-backup-storage='s3://backup-101/snapshot-20250602000000?access-key=${ACCESS-KEY}&secret-access-key=${SECRET-ACCESS-KEY}' \
--start-ts "2025-06-02 00:00:00+0800" \
--restored-ts "2025-06-03 18:00:00+0800" \
--filter 'db2.*'
```

> **Note:**
>
> - Each concurrent restore operation must target different databases or non-overlapping table sets. Attempting to restore overlapping datasets concurrently will result in an error.


Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

There seems to be an extra blank line here before the next section heading. To maintain consistent formatting and conciseness 1, could you please remove it?

Style Guide References

Footnotes

  1. The style guide specifies adding a blank line before and after headings and lists, implying that excessive blank lines should be avoided. (link)

### Compatibility between ongoing log backup and snapshot restore

Starting from v9.0.0, when a log backup task is running, if all of the following conditions are met, you can still perform snapshot restore (`br restore [full|database|table]`) and allow the restored data to be properly recorded by the ongoing log backup (hereinafter referred to as "log backup"):
Expand All @@ -507,13 +585,25 @@ Starting from v9.0.0, when a log backup task is running, if all of the following
- The data to be restored uses the same type of external storage as the target storage for the log backup.
- Neither the data to be restored nor the log backup has enabled local encryption. For details, see [log backup encryption](#encrypt-the-log-backup-data) and [snapshot backup encryption](/br/br-snapshot-manual.md#encrypt-the-backup-data).

If any of the above conditions are not met, or if you need to perform a point-in-time recovery, while a log backup task is running, BR refuses to proceed with the data recovery. In this case, you can complete the recovery by following these steps:

If any of the above conditions are not met, you can complete the recovery by following these steps:
1. [Stop the log backup task](#stop-a-log-backup-task).
2. Perform the data restore.
3. After the restore is complete, perform a new snapshot backup.
4. [Restart the log backup task](#restart-a-log-backup-task).

> **Note:**
>
> When restoring a log backup that contains records of snapshot (full) restore data, you must use BR v9.0.0 or later. Otherwise, restoring the recorded full restore data might fail.
> When restoring a log backup that contains records of snapshot (full) restore data, you must use BR v9.0.0 or later. Otherwise, restoring the recorded full restore data might fail.

### Compatibility between ongoing log backup and PITR operations

Starting from TiDB v9.0.0, you can perform PITR operations while a log backup task is running by default. The system automatically handles compatibility between these operations.

#### Important limitation for PITR with ongoing log backup

When performing PITR operations while log backup is running, the restored data will be recorded in the log backup. However, during the restore time window, the data may not be consistent due to the nature of log restore operations. The system writes metadata to external storage to mark both the time range and data range where consistency cannot be guaranteed.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

To maintain consistency with the style guide's recommendation to use the second person ('you') 1, could the beginning of this sentence be rephrased?

Style Guide References

Suggested change
When performing PITR operations while log backup is running, the restored data will be recorded in the log backup. However, during the restore time window, the data may not be consistent due to the nature of log restore operations. The system writes metadata to external storage to mark both the time range and data range where consistency cannot be guaranteed.
When you perform PITR operations while log backup is running, the restored data will be recorded in the log backup. However, during the restore time window, the data may not be consistent due to the nature of log restore operations. The system writes metadata to external storage to mark both the time range and data range where consistency cannot be guaranteed.

Footnotes

  1. The style guide recommends writing in the second person ('you') when addressing users. (link)


If such inconsistency occurs during the time range `[t1, t2)`, you cannot restore data from that time period. Instead, you must:

- Restore data up to `t1` (before the inconsistent period), or
- Take a new snapshot backup after `t2` and use that for future PITR operations

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

It's a common best practice for Markdown files (and text files in general) to end with a newline character. This can prevent potential issues with some tools or systems that process these files and improves diff readability. Could you please add a newline at the very end of this file?