Rawkv backup and restore (#104)

* Add backup & restore section to API v2

Signed-off-by: haojinming <[email protected]>

Author: haojinming

text/0069-api-v2.md

@@ -236,7 +236,30 @@ More about `GC` of deleted/expired entries:
 
 ### Backup and Restore
 
-*To be supplemented in another PR*
+Due to the significant change on storage format, only if the existing TiKV cluster is empty or storing only `TiDB` data, users can enable or disable API v2 smoothly. In other scenario, we need implement one tool to migrate data, which is called [TiKV-BR].  
+
+[TiKV-BR] forks from [TiDB BR] and need some improvements:
+- Storage data conversion from `API V1`(with or without [TTL]) to `API V2`.
+- Backup supports return `backup-ts` in `API V2` TiKV cluster. [RawKV CDC] can use `backup-ts` as `start-ts` for replication tasks.
+
+#### API Version Conversion
+
+To support API version conversion, a parameter `dst-api-version` is introduced in [TiKV-BR] and pass it to TiKV stores in [`BackupRequest`].
+During the backup process, TiKV stores scan on all raw key-value entries and convert from current `api-version` to `dst-api-version` if they are different, then write the converted data to [SST] files. Restoration does not need the rewriting and conversion, which can speed up the restoration process.
+
+#### Backup Ts
+
+`backup-ts` only takes effect in the `API V2` cluster, which is defined as the timestamp that all data written before which have been backed up.  
+As RawKV uses pre-fetches `TSO` for writing, the latest `TSO` from PD does not satisfy the `backup-ts` requirements obviously.  
+
+The process to get `backup-ts` is as following:
+1. Get current `TSO` in [TiKV-BR] at the beginning of backup.
+2. Flush cached `TSO` in every TiKV store before scanning process during backup, to make sure that all writes afterward will have larger timestamps.
+3. Subtract `safe-interval` from current `TSO`.
+
+The third step is introduced because there would be **inflight** RawKV entries with timestamp before the current `TSO`. The `safe-interval` is a safe enough time duration that during which **inflight** writes should have finished. This is an empirical value and defaults to 1 minute, which is safe enough even in a high-pressure system.
+
+Besides, this process can be optimized after we implement `safe-ts` for RawKV in stale read scenario, which is a timestamp that all RawKV writes before this timestamp can be seen **safely**.
 
 ### Change Data Capture
 
@@ -269,4 +292,9 @@ Upgrade to the latest TiKV Go Client and use `V1` mode.
 [TiDB BR]: https://docs.pingcap.com/tidb/stable/backup-and-restore-overview
 [TiCDC]: https://docs.pingcap.com/tidb/stable/ticdc-overview
 [GC Overview]: https://docs.pingcap.com/tidb/dev/garbage-collection-overview#gc-overview
+[TiKV-BR]: https://github.com/tikv/migration/tree/main/br
+[TTL]: https://docs.pingcap.com/zh/tidb/dev/tikv-configuration-file#enable-ttl
+[`BackupRequest`]: https://github.com/pingcap/kvproto/blob/3debb6820e46da7a8f310e3f081222183cdd8030/proto/brpb.proto#L170
+[creating change feed]: https://tikv.org/docs/dev/concepts/explore-tikv-features/cdc/cdc/#manage-replication-tasks-changefeed
 [RawKV Change Data Capture]: ./0083-rawkv-change-data-capture.md
+[RawKV CDC]: ./0083-rawkv-change-data-capture.md