ticdc v4.0.16

Compatibility changes

• Change the default value of Kafka Sink max-message-bytes to 1 MB to prevent TiCDC from sending too large messages to Kafka clusters #2962
• Change the default value of Kafka Sink partition-num to 3 so that TiCDC distributes messages across Kafka partitions more evenly #3337

Improvements

• Add a tick frequency limit to EtcdWorker to prevent frequent etcd writes from affecting PD services #3112
• Optimize rate limiting control on TiKV reloads to reduce gPRC congestion during changefeed initialization #3110

Bug fixes

• Fix the negative value error in the changefeed checkpoint lag #3010
• Fix OOM in container environments #1798
• Fix the TiCDC replication interruption issue when multiple TiKVs crash or during a forced restart #3288
• Fix the memory leak issue after processing DDLs #3174
• Fix the issue that changefeed does not fail fast enough when the ErrGCTTLExceeded error occurs #3111
• Fix the issue that TiCDC replication task might terminate when the upstream TiDB instance unexpectedly exits #3061
• Fix the issue that TiCDC process might panic when TiKV sends duplicate requests to the same Region #2386
• Fix the issue that the volume of Kafka messages generated by TiCDC is not constrained by max-message-size #2962
• Fix the issue that tikv_cdc_min_resolved_ts_no_change_for_1m keeps alerting when there is no changefeed #11017
• Fix the issue that TiCDC sync task might pause when an error occurs during writing a Kafka message #2978
• Fix the issue that some partitioned tables without valid indexes might be ignored when force-replicate is enabled #2834
• Fix the memory leak issue when creating a new changefeed #2389
• Fix the issue that might cause inconsistent data due to Sink components advancing resolved ts early #3503
• Fix the issue that scanning stock data might fail due to TiKV performing GC when scanning stock data takes too long #2470
• Fix the issue that the changefeed update command does not recognize global command line parameters #2803

ticdc v5.3.0

Improvements

• Reduce the default value of the Kafka sink configuration item MaxMessageBytes from 64 MB to 1 MB to fix the issue that large messages are rejected by the Kafka Broker #3104
• Reduce memory usage in the replication pipeline #2553#3037 #2726
• Optimize monitoring items and alert rules to improve observability of synchronous links, memory GC, and stock data scanning processes #2735 #1606 #3000 #2985 #2156
• When the sync task status is normal, no more historical error messages are displayed to avoid misleading users #2242

Bug Fixes

• Fix the issue that TiCDC replication task might terminate when the upstream TiDB instance unexpectedly exits #3061
• Fix the issue that TiCDC process might panic when TiKV sends duplicate requests to the same Region #2386
• Fix unnecessary CPU consumption when verifying downstream TiDB/MySQL availability #3073
• Fix the issue that the volume of Kafka messages generated by TiCDC is not constrained by max-message-size #2962
• Fix the issue that TiCDC sync task might pause when an error occurs during writing a Kafka message #2978
• Fix the issue that some partitioned tables without valid indexes might be ignored when force-replicate is enabled #2834
• Fix the issue that scanning stock data might fail due to TiKV performing GC when scanning stock data takes too long #2470
• Fix a possible panic issue when encoding some types of columns into Open Protocol format #2758
• Fix a possible panic issue when encoding some types of columns into Avro format #2648

ticdc v5.2.3

There is no release note for this version.

ticdc v5.2.2

Compatibility changes

• Set compatible version from 5.1.0-alpha to 5.2.0-alpha #2659
• Prohibit operating TiCDC clusters across major and minor versions #2599
• Fix the CLI compatibility issue with 4.0.x clusters on the default sort-engine option #2373

Improvements

• Optimize memory management when the Unified Sorter is using memory to sort #2553
• Optimize workerpool for fewer goroutines when concurrency is high #2211
• Reduce goroutine usage when a table's region transfer away from a TiKV node #2284

Bug Fixes

• Fix a bug that json encoding may cause panic when processing a string type value that is string or []byte. #2758
• Reduce gRPC window size to avoid OOM #2202
• Fix gRPC keepalive error under high memory pressure. #2202
• Fix a bug that an unsigned tinyint causes TiCDC to panic. #2648
• Fix empty value issue in open protocol. An empty value is no longer output when there is no change in one transaction. #2612
• Fix a bug in DDL handling during manual restarts #2603
• Fix the issue that EtcdWorker's snapshot isolation might be wrongly violated when managing the metadata #2559
• Fix the issue that multiple processors could write in the same table when re-scheduling this table #2230
• Fix the issue of the ErrSchemaStorageTableMiss error and that a changefeed is reset by accident. #2422
• Fix the issue that changefeed cannot be removed when the GcTTL Exceeded Error occors #2391
• Fix the issue that replicating large tables to cdclog fails #1259#2424

ticdc v5.1.3

There is no release note for this version.

ticdc v5.1.2

Compatibility changes

• Set compatible version from 5.1.0-alpha to 5.2.0-alpha #2659
• Prohibit operating TiCDC clusters across major and minor versions #2599
• Fix the CLI compatibility issue with 4.0.x clusters on the default sort-engine option #2373

Improvements

• Optimize memory management when the Unified Sorter is using memory to sort #2553
• Optimize workerpool for fewer goroutines when concurrency is high #2211
• Reduce goroutine usage when a table's region transfer away from a TiKV node #2284

Bug Fixes

• Fix a bug that json encoding may cause panic when processing a string type value that is string or []byte. #2758
• Reduce gRPC window size to avoid OOM #2202
• Fix gRPC keepalive error under high memory pressure. #2202
• Fix a bug that an unsigned tinyint causes TiCDC to panic. #2648
• Fix empty value issue in open protocol. An empty value is no longer output when there is no change in one transaction. #2612
• Fix a bug in DDL handling during manual restarts #2603
• Fix the issue that EtcdWorker's snapshot isolation might be wrongly violated when managing the metadata #2559
• Fix the issue that multiple processors could write in the same table when re-scheduling this table #2230
• Fix the issue of the ErrSchemaStorageTableMiss error and that a changefeed is reset by accident. #2422
• Fix the issue that changefeed cannot be removed when the GcTTL Exceeded Error occors #2391
• Fix the issue that replicating large tables to cdclog fails #1259#2424

ticdc v4.0.15

Improvements

• TiCDC always pulls old values from TiKV internally #2304
• Reduce the goroutine usage when a table's Regions are all transferred away from a TiKV node #2284
• Optimize workerpool for fewer goroutines when concurrency is high #2211
• Execute DDL statements asynchronously to avoid affecting other changefeeds #2471
• Add a global gRPC connection pool and share gRPC connections among KV clients #2531
• Fail fast for unrecoverable DML errors 2315
• Optimize memory management when unified sorter is using memory to sort #2553
• Add Prometheus metrics for DDL executions #2681
• Prohibit operating TiCDC clusters across major or minor versions #2601
• Remove file sorter #2325
• Clean up changefeed metrics when a changefeed is removed, and clean up processor metrics when a processor exits #2313
• Optimize the lock-resolving algorithm after a Region is initialized #2264

Bug fixes

• Fix the ErrSchemaStorageTableMiss error that occurs when the DDL Job duplication is encountered in the integrated test #2422
• Fix the bug that a changefeed cannot be removed if the ErrGCTTLExceeded error occurs #2391
• Fix the issue that outdated capture might appear in the output of the capture list command #2388
• Fix the deadlock issue in the TiCDC processor #2017
• Fix a data inconsistency issue that occurs because multiple processors might write data to the same table when this table is being re-scheduled #2495#2727
• Fix a bug that the EtcdWorker snapshot isolation is violated in metadata management #2557
• Fix the issue that the changefeed cannot be stopped due to the DDL sink error #2556
• Fix the issue of TiCDC Open Protocol: TiCDC outputs an empty value when there is no change in a transaction #2619
• Fix a bug that causes TiCDC to panic on the unsigned TINYINT type #2648
• Decrease the gRPC window size to avoid the OOM that occurs when TiCDC captures too many Regions #2202
• Fix the OOM issue that occurs when TiCDC captures too many Regions #2723
• Fix the issue of process panic that occurs when encoding the data types such as mysql.TypeString, mysql.TypeVarString, mysql.TypeVarchar into JSON #2758
• Fix the a memory leak issue that might occur when creating a new changefeed #2623
• Fix a bug that DDL handling fails when a changefeed starts at the finish TS of a schema change #2603
• Fix the issue of potential DDL loss when the owner crashes when executing DDL statements #1260
• Fix the issue of insecure concurrent access to the map in SinkManager #2298

ticdc v5.0.5

There is no release note for this version.

ticdc v5.0.4

Feature Enhancements

• Add an HTTP API to get the changefeed information and the health information of the node #1955
• Add the SASL/SCRAM support for the kafka sink #1942
• Make TiCDC support --data-dir at the server level #2070

Improvements

• Refine gRPC's reconnection logic and increase the KV client's throughput #1586 #1501 #1682 #1393 #1847 #1905 #1904
• Make the sorter I/O errors more user-friendly

Bug Fixes

• Fix the issue that TiCDC owner exits when refreshing the checkpoint #1902
• Fix a bug that some MySQL connection might leak after MySQL sink meets the error and pauses #1946
• Fix the panic issue that occurs when TiCDC fails to read /proc/meminfo #2024
• Reduce TiCDC's runtime memory consumption #2012 #1958
• Fix a bug that might cause TiCDC server panic due to the late calculation of resolved ts #1576
• Fix the potential deadlock issue for the processor #2142

ticdc v5.2.1

owner: fix gc safepoint larger by one (#2647)