ticdc v5.2.0

Feature Enhancements

• Add the binary MQ format designed for TiDB. It is more compact than the open protocols based on JSON #1621
• Remove support for file sorter #2114
• Support log rotation configurations #2182

Improvements

• Add the concurrency limit to the incremental scan of kv client #1899
• TiCDC can always pull the old value internally #2271
• TiCDC can fail and exit fast when unrecoverable DML errors occur #1928
• resolve lock cannot be run immediately after a Region is initialized #2235
• Optimize workerpool to reduce the number of goroutines under high concurrency #2201

Bug Fixes

• Fix a bug that TiCDC owner exits abnormally when refreshing the checkpoint #1902
• Fix a bug that changefeed fails immediately after its successful creation #2113
• Fix a bug that changefeed fails due to the invalid format of rules filter #1625
• Fix the potential DDL loss issue when the TiCDC owner panics #1260
• Fix the CLI compatibility issue with 4.0.x clusters on the default sort-engine option #2373
• Fix a bug that changefeed might be reset unexpectedly when TiCDC gets the ErrSchemaStorageTableMiss error #2422
• Fix a bug that changefeed cannot be removed when TiCDC gets the ErrGCTTLExceeded error #2391
• Fix a bug that TiCDC fails to synchronize large tables to cdclog #1259 #2424
• Fix a bug that multiple processors might write data to the same table when TiCDC is rescheduling the table #2230

ticdc v5.1.1

Improvements

• Remove file sorter #2327
• Improve the error message returned when a PD endpoint misses the certificate #1973

Bug fixes

• Fix the support for the new collation feature #2301
• Fix the issue that an unsynchronized access to a shared map at runtime might cause panic #2300
• Fix the potential DDL loss issue that occurs when the owner crashes while executing the DDL statement #2290
• Fix the issue of trying to resolve locks in TiDB prematurely #2188
• Fix a bug that might cause data loss if a TiCDC node is killed immediately after a table migration #2033
• Fix the handling logic of changefeed update on --sort-dir and --start-ts #1921

ticdc v4.0.14

Improvements

• Improve the error message returned when a PD endpoint misses the certificate #2184
• Make the sorter I/O errors more user-friendly #1976
• Add a concurrency limit on the Region incremental scan in the KV client to reduce the pressure of TiKV #1926
• Add metrics for the table memory consumption #1884
• Add capture-session-ttl to the TiCDC server configuration #2169

Bug fixes

• Fix the issue of extra partition dispatching after adding new table partitions #2205
• Fix the panic issue that occurs when TiCDC fails to read /proc/meminfo #2023
• Reduce TiCDC's runtime memory consumption #2011 #1957
• Fix a bug that some MySQL connection might leak after MySQL sink meets the error and pauses #1945
• Fix the issue that TiCDC changefeed cannot be created when start TS is less than current TS minus GC TTL #1839
• Reduce memory malloc in sort heap to avoid too much CPU overhead #1853
• Fix a bug that the replication task might stop when moving a table #1827

ticdc v5.0.3

Feature Enhancements

• Add an HTTP API to get the changefeed information and the health information of the node #1955
• Add the SASL/SCRAM support for the kafka sink #1942
• Make TiCDC support --data-dir at the server level #2070

Improvements

• Refine gRPC's reconnection logic and increase the KV client's throughput #1586 #1501 #1682 #1393 #1847 #1905 #1904
• Make the sorter I/O errors more user-friendly

Bug Fixes

• Fix the issue that TiCDC owner exits when refreshing the checkpoint #1902
• Fix a bug that some MySQL connection might leak after MySQL sink meets the error and pauses #1946
• Fix the panic issue that occurs when TiCDC fails to read /proc/meminfo #2024
• Reduce TiCDC's runtime memory consumption #2012 #1958
• Fix a bug that might cause TiCDC server panic due to the late calculation of resolved ts #1576
• Fix the potential deadlock issue for the processor #2142

ticdc v5.1.0

Improvements

• Improve the descriptions of some log messages to be clearer and more useful for diagnosing problems #1759
• Support the back pressure feature to allow the TiCDC scanning speed to sense the downstream processing capacity #10151
• Reduce memory usage when TiCDC performs the initial scan #10133
• Improve the cache hit rate for the TiCDC Old Value in pessimistic transactions #10089

Bug Fixes

• Fix the concurrency issue in Unified Sorter and filter the unhelpful error messages #1678
• Fix a bug that the creation of redundant directories might interrupt the replication with MinIO #1463
• Set the default value of the explicit_defaults_for_timestamp session variable to ON to make the MySQL 5.7 downstream keep the same behavior with the upstream TiDB #1585
• Fix the issue that the incorrect handling of io.EOF might cause replication interruption #1633
• Correct the TiKV CDC endpoint CPU metric in the TiCDC dashboard #1645
• Increase defaultBufferChanSize to avoid replication blocking in some cases #1259
• Fix the issue that the time zone information is lost in the Avro output #1712
• Support cleaning up stale temporary files in Unified Sorter and forbid sharing the sort-dir directory #1742
• Fix a deadlock bug in the KV client that occurs when many stale Regions exist #1599
• Fix the wrong help information in the --cert-allowed-cn flag #1697
• Revert the update for explicit_defaults_for_timestamp which requires the SUPER privilege when replicating data to MySQL #1750
• Support the sink flow control to reduce the risk of memory overflow #1840
• Fix a bug that the replication task might stop when moving a table #1828
• Fix the issue that the TiKV GC safe point is blocked due to the stagnation of TiCDC changefeed checkpoint #1759

v4.0.13 hotfix

v4.0.13-20210610-ffbdae3

kv/client: add incremental scan region count limit (#1926)

ticdc v5.0.2

Compatibility Changes

• Deprecate --sort-dir in the cdc cli changefeed command. Instead, users can set --sort-dir in the cdc server command. #1795

Improvements

• Add monitoring metrics for the table memory consumption #1885
• Optimize the memory and CPU usages during the sorting stage #1863
• Delete some useless log information that might cause user confusion #1759

Bug Fixes

• Fix the issue that the time zone information is lost in the Avro output #1712
• Support cleaning up stale temporary files in Unified Sorter and forbid sharing the sort-dir directory #1742
• Fix a deadlock bug in the KV client that occurs when many stale Regions exist #1801
• Fix the wrong help information in the --cert-allowed-cn flag #1697
• Revert the update for explicit_defaults_for_timestamp which requires the SUPER privilege when replicating data to MySQL #1750
• Support the sink flow control to reduce the risk of memory overflow #1840
• Fix a bug that the replication task might stop when moving a table #1828
• Fix the issue that the TiKV GC safe point is blocked due to the stagnation of TiCDC changefeed checkpoint #1759

Pre Release v5.2.0-alpha

sorter: provide more information on sorter IO errors (#1969)

Pre Release v5.1.0-alpha

Pre Release v5.1.0-alpha

ticdc v4.0.13

Improvements

• Implement the processor flow control to avoid memory overflow (OOM) #1751
• Support cleaning up stale temporary files in Unified Sorter and prevent multiple cdc server instances from sharing the same sort-dir directory #1741
• Add the HTTP handler for the failpoint #1732

Bug Fixes

• Fix the deadlock issue caused by the flow control when the sorter's input channel has been blocked #1779
• Fix the issue that the TiKV GC safe point is blocked due to the stagnation of TiCDC changefeed checkpoint #1756
• Revert the update in explicit_defaults_for_timestamp which requires the SUPER privilege when replicating data to MySQL #1749