tiflow v5.3.2

Improvements

• Support Syncer using the working directory of the DM-worker rather than /tmp to write internal files, and cleaning the directory after the task is stopped #4107

Bug Fixes

• Fix the issue that DM occupies more disk space after the task automatically resumes #3734 #5344
• Fix an issue that the uppercase table cannot be replicated when case-sensitive: true is not set #5255
• Fix the issue that in some cases manually executing the filtered DDL in the downstream might cause task resumption failure #5272
• Fix the DM worker panic issue that occurs when the primary key is not first in the index returned by the SHOW CREATE TABLE statement #5159
• Fix the issue that CPU usage may increase and a large amount of log is printed when GTID is enabled or when the task is automatically resumed #5063
• Fix the issue that the relay log may be disabled after the DM-master reboots #4803
• Fix incorrect metrics caused by owner changes #4774
• Fix the bug that the redo log manager flushes logs before writing logs #5486
• Fix the bug that the resolved ts moves too fast when some tables are not maintained by the redo writer #5486
• Add the UUID suffix to the redo log file name to fix the issue that file name conflicts may cause data loss #5486
• Fix the bug that MySQL Sink may save a wrong checkpointTs #5107
• Fix the issue that TiCDC clusters may panic after upgrade #5266
• Fix the issue that changefeed gets stuck when tables are repeatedly scheduled in the same node #4464
• Fix the issue that TiCDC fails to start when the first PD set in --pd is not available after TLS is enabled #4777
• Fix a bug that querying status through open API may be blocked when the PD node is abnormal #4778
• Fix a stability problem in workerpool used by Unified Sorter #4447
• Fix a bug that sequence is incorrectly replicated in some cases #4563

tiflow v6.1.0

Bug fixes

• Fix excessive memory usage by optimizing the way DDL schemas are buffered #1386
• Fix OOM caused by large transactions #5280
• Fix data loss that occurs in special incremental scanning scenarios #5468
• Fix the start-time time zone issue and change DM behavior from using the downstream time zone to using the upstream time zone #5271
• Fix the issue that DM occupies more disk space after the task automatically resumes #3734 #5344
• Fix the problem that checkpoint flush may cause the data of failed rows to be skipped #5279
• Fix the issue that in some cases manually executing the filtered DDL in the downstream might cause task resumption failure #5272
• Fix an issue that the uppercase table cannot be replicated when case-sensitive: true is not set #5255
• Fix the DM worker panic issue that occurs when the primary key is not first in the index returned by the SHOW CREATE TABLE statement #5159
• Fix the issue that CPU usage may increase and a large amount of log is printed when GTID is enabled or when the task is automatically resumed #5063
• Fix the offline option and other usage issues in DM WebUI #4993
• Fix the issue that incremental tasks fail to start when GTID is empty in the upstream #3731
• Fix the issue that empty configurations may cause dm-master to panic #3732

tiflow v5.4.1

Improvements

• Support multiple Kubernetes clusters in Grafana dashboards #4665
• Expose configuration parameters of the Kafka producer to make them configurable in TiCDC #4385

Bug Fixes

• Fix incorrect metrics caused by owner changes #4774
• Fix the TiCDC panic issue that might occur because Canal-JSON does not support nil #4736
• Fix a stability problem in workerpool used by Unified Sorter #4447
• Fix a bug that sequence is incorrectly replicated in some cases #4563
• Fix the TiCDC panic issue that might occur when Canal-JSON incorrectly handles string #4635
• Fix a bug that a TiCDC node exits abnormally when a PD leader is killed #4248
• Fix a bug that MySQL sink generates duplicated replace SQL statements when batch-replace-enable is disabled #4501
• Fix the DML construct error caused by the rename tables DDL #5059
• Fix the issue that in rare cases replication can be stuck if the owner is changed and the new scheduler is enabled (disabled by default) #4963
• Fix the issue that the error ErrProcessorDuplicateOperations is reported when the new scheduler is enabled (disabled by default) #4769
• Fix the issue that TiCDC fails to start when the first PD set in --pd is not available after TLS is enabled #4777
• Fix the issue that the checkpoint metrics are missing when tables are being scheduled #4714

tiflow v5.2.4

Improvements

• Add a Lag analyze panel in Grafana #4891
• Support placement rules #4846
• Synchronize HTTP API handling #1710
• Add the exponential backoff mechanism for restarting a changefeed #3329
• Set the default isolation level of MySQL sink to read-committed to reduce deadlocks in MySQL #3589
• Validate changefeed parameters upon creation and refine error messages #1716 #1718 #1719 #4472
• Expose configuration parameters of the Kafka producer to make them configurable in TiCDC #4385
• Support starting a task when upstream table schemas are inconsistent and in optimistic mode #3629 #3708 #3786
• Support creating a task in the stopped state #4484
• Support Syncer using the working directory of the DM-worker rather than /tmp to write internal files, and cleaning the directory after the task is stopped #4107
• Precheck has improved. Some important checks are no longer skipped. #3608

Bug Fixes

• Fix a bug that MySQL sink generates duplicated replace SQL statements whenbatch-replace-enable is disabled #4501
• Fix a bug that a TiCDC node exits abnormally when a PD leader is killed #4248
• Fix the error Unknown system variable 'transaction_isolation' for some MySQL versions #4504
• Fix the TiCDC panic issue that might occur when Canal-JSON incorrectly handles string #4635
• Fix a bug that sequence is incorrectly replicated in some cases #4563
• Fix the TiCDC panic issue that might occur because Canal-JSON does not support nil #4736
• Fix the wrong data mapping for avro codec of type Enum/Set and TinyText/MediumText/Text/LongText #4454
• Fix a bug that Avro converts a NOT NULL column to a nullable field #4818
• Fix an issue that TiCDC cannot exit #4699
• Fix the issue that syncer metrics are updated only when querying the status #4281
• Fix the issue that execution errors of the update statement in safemode may cause the DM-worker panic #4317
• Fix a bug that long varchars report an error Column length too big #4637
• Fix the conflict issue caused by multiple DM-workers writing data from the same upstream #3737
• Fix the issue that hundreds of "checkpoint has no change, skip sync flush checkpoint" print in the log and the replication is very slow #4619
• Fix the DML loss issue when merging shards and replicating incremental data from upstream in the pessimistic mode #5002

tiflow v6.0.0

Improvements

• Add a Lag analyze panel in Grafana #4891
• Support placement rules #4846
• Synchronize HTTP API handling #1710
• Add the exponential backoff mechanism for restarting a changefeed #3329
• Set the default isolation level of MySQL sink to read-committed to reduce deadlocks in MySQL #3589
• Validate changefeed parameters upon creation and refine error messages #1716 #1718 #1719 #4472
• Expose configuration parameters of the Kafka producer to make them configurable in TiCDC #4385
• Support starting a task when upstream table schemas are inconsistent and in optimistic mode #3629 #3708 #3786
• Support creating a task in the stopped state #4484
• Support Syncer using the working directory of the DM-worker rather than /tmp to write internal files, and cleaning the directory after the task is stopped #4107
• Precheck has improved. Some important checks are no longer skipped. #3608

Bug Fixes

• Fix a bug that MySQL sink generates duplicated replace SQL statements whenbatch-replace-enable is disabled #4501
• Fix a bug that a TiCDC node exits abnormally when a PD leader is killed #4248
• Fix the error Unknown system variable 'transaction_isolation' for some MySQL versions #4504
• Fix the TiCDC panic issue that might occur when Canal-JSON incorrectly handles string #4635
• Fix a bug that sequence is incorrectly replicated in some cases #4563
• Fix the TiCDC panic issue that might occur because Canal-JSON does not support nil #4736
• Fix the wrong data mapping for avro codec of type Enum/Set and TinyText/MediumText/Text/LongText #4454
• Fix a bug that Avro converts a NOT NULL column to a nullable field #4818
• Fix an issue that TiCDC cannot exit #4699
• Fix the issue that syncer metrics are updated only when querying the status #4281
• Fix the issue that execution errors of the update statement in safemode may cause the DM-worker panic #4317
• Fix a bug that long varchars report an error Column length too big #4637
• Fix the conflict issue caused by multiple DM-workers writing data from the same upstream #3737
• Fix the issue that hundreds of "checkpoint has no change, skip sync flush checkpoint" print in the log and the replication is very slow #4619
• Fix the DML loss issue when merging shards and replicating incremental data from upstream in the pessimistic mode #5002

v6.0.0-alpha: validator(dm): full mode validator, part 1 (#4539)

ref pingcap/tiflow#4488

ticdc v5.3.1

Compatibility changes

• Set the default value of max-message-bytes to 10M #4041

Feature enhancements

• Add the exponential backoff mechanism for restarting a changefeed. #3329
• Reduce the replication latency when replicating many tables #3900
• Add metrics for observing the remaining time of incremental scan #2985
• Reduce the count of "EventFeed retry rate limited" logs #4006
• Add more Promethous and Grafana monitoring metrics and alerts, including no owner alert, mounter row, table sink total row, and buffer sink total row #4054 #1606
• Optimize rate limiting control on TiKV reloads to reduce gPRC congestion during changefeed initialization #3110
• Reduce the time for the KV client to recover when a TiKV store is down #3191

Bug fixes

• Fix a bug that MySQL sink generates duplicated replace SQL statements if batch-replace-enable is disabled #4501
• Fix the issue that the cached region monitoring metric is negative #4300
• Fix the issue that replication cannot be performed when min.insync.replicas is smaller than replication-factor #3994
• Fix the potential panic issue that occurs when a replication task is removed #3128
• Fix the issue of potential data loss caused by inaccurate checkpoint #3545
• Fix the potential issue that the deadlock causes a replication task to get stuck #4055
• Fix the issue that special comments in DDL statements cause the replication task to stop #3755
• Fix a bug that EtcdWorker might hang the owner and processor #3750
• Fix the issue that stopped changefeeds resume automatically after a cluster upgrade #3473
• Fix the issue that default values cannot be replicated #3793
• Fix data inconsistency caused by TiCDC default value padding exceptions #3918 #3929
• Fix a bug that an owner gets stuck when a PD leader shuts down and transfers to a new node #3615
• Fix the TiCDC panic issue that occurs when manually cleaning the task status in etcd #2980
• Fix the issue that the service cannot be started because of a timezone issue in the RHEL release #3584
• Fix the issue of overly frequent warnings caused by MySQL sink deadlock #2706
• Fix the bug that the enable-old-value configuration item is not automatically set to true on Canal and Maxwell protocols #3676
• Fix the issue that Avro sink does not support parsing JSON type columns #3624
• Fix the negative value error in the changefeed checkpoint lag #3010
• Fix the OOM issue in the container environment #1798
• Fix the TiCDC replication interruption issue when multiple TiKVs crash or during a forced restart #3288
• Fix the memory leak issue after processing DDLs #3174
• Fix the issue that changefeed does not fail fast enough when the ErrGCTTLExceeded error occurs #3111
• Fix the issue that TiCDC replication task might terminate when the upstream TiDB instance unexpectedly exits #3061
• Fix the issue that TiCDC process might panic when TiKV sends duplicate requests to the same Region #2386
• Fix the issue that the volume of Kafka messages generated by TiCDC is not constrained by max-message-size #2962
• Fix the issue that Kafka may send excessively large messages by setting the default value of max-message-bytes to 10M #3081
• Fix the issue that TiCDC sync task might pause when an error occurs during writing a Kafka message #2978

ticdc v5.1.4

Compatibility changes

• Set the default value of max-message-bytes to 10M #4041

Feature enhancements

• Add the exponential backoff mechanism for restarting a changefeed. #3329
• Reduce the replication latency when replicating many tables #3900
• Add metrics for observing the remaining time of incremental scan #2985
• Reduce the count of "EventFeed retry rate limited" logs #4006
• Add more Promethous and Grafana monitoring metrics and alerts, including no owner alert, mounter row, table sink total row, and buffer sink total row #4054 #1606
• Optimize rate limiting control on TiKV reloads to reduce gPRC congestion during changefeed initialization #3110
• Reduce the time for the KV client to recover when a TiKV store is down #3191

Bug fixes

• Fix a bug that MySQL sink generates duplicated replace SQL statements if batch-replace-enable is disabled #4501
• Fix the issue that the cached region monitoring metric is negative #4300
• Fix the issue that replication cannot be performed when min.insync.replicas is smaller than replication-factor #3994
• Fix the potential panic issue that occurs when a replication task is removed #3128
• Fix the issue of potential data loss caused by inaccurate checkpoint #3545
• Fix the potential issue that the deadlock causes a replication task to get stuck #4055
• Fix the issue that special comments in DDL statements cause the replication task to stop #3755
• Fix a bug that EtcdWorker might hang the owner and processor #3750
• Fix the issue that stopped changefeeds resume automatically after a cluster upgrade #3473
• Fix the issue that default values cannot be replicated #3793
• Fix data inconsistency caused by TiCDC default value padding exceptions #3918 #3929
• Fix a bug that an owner gets stuck when a PD leader shuts down and transfers to a new node #3615
• Fix the TiCDC panic issue that occurs when manually cleaning the task status in etcd #2980
• Fix the issue that the service cannot be started because of a timezone issue in the RHEL release #3584
• Fix the issue of overly frequent warnings caused by MySQL sink deadlock #2706
• Fix the bug that the enable-old-value configuration item is not automatically set to true on Canal and Maxwell protocols #3676
• Fix the issue that Avro sink does not support parsing JSON type columns #3624
• Fix the negative value error in the changefeed checkpoint lag #3010
• Fix the OOM issue in the container environment #1798
• Fix the TiCDC replication interruption issue when multiple TiKVs crash or during a forced restart #3288
• Fix the memory leak issue after processing DDLs #3174
• Fix the issue that changefeed does not fail fast enough when the ErrGCTTLExceeded error occurs #3111
• Fix the issue that TiCDC replication task might terminate when the upstream TiDB instance unexpectedly exits #3061
• Fix the issue that TiCDC process might panic when TiKV sends duplicate requests to the same Region #2386
• Fix the issue that the volume of Kafka messages generated by TiCDC is not constrained by max-message-size #2962
• Fix the issue that Kafka may send excessively large messages by setting the default value of max-message-bytes to 10M #3081
• Fix the issue that TiCDC sync task might pause when an error occurs during writing a Kafka message #2978

tiflow v5.4.0

Improvements

• Reduce the count of "EventFeed retry rate limited" logs #4006
• Reduce the replication latency when replicating many tables #3900
• Reduce the time for the KV client to recover when a TiKV store is down #3191

Bug fixes

• Fix the issue that replication cannot be performed when min.insync.replicas is smaller than replication-factor #3994
• Fix the issue that the cached region monitoring metric is negative #4300
• Fix the issue that mq sink write row does not have monitoring data #3431
• Fix the compatibility issue of sql mode #3810
• Fix the potential panic issue that occurs when a replication task is removed #3128
• Fix the issue of panic and data inconsistency that occurs when outputting the default column value #3929
• Fix the issue that default values cannot be replicated #3793
• Fix the potential issue that the deadlock causes a replication task to get stuck #4055
• Fix the issue that no log is output when the disk is fully written #3362
• Fix the issue that special comments in DDL statements cause the replication task to stop #3755
• Fix the issue that the service cannot be started because of a timezone issue in the RHEL release #3584
• Fix the issue of potential data loss caused by inaccurate checkpoint #3545
• Fix the OOM issue in the container environment #1798
• Fix the issue of replication stop caused by the incorrect configuration of config.Metadata.Timeout #3352

ticdc v5.0.6

Compatibility changes

• Change the output of the cdc server command error from stdout to stderr #3133
• Set the default value of Kafka sink max-message-bytes to 10M #3081

Improvements

• Add the default configuration for config.Metadata.Timeout in Kafka sink #3352
• Set the default value of max-message-bytes to 10M, to reduce the probability that Kafka messages cannot be sent #3081
• Add more Promethous and Grafana monitoring metrics and alerts, including no owner alert, mounter row, table sink total row, and buffer sink total row #4054 #1606

Bug fixes

• Fix the issue that cdc cli silently truncates user parameters when receiving unexpected parameters, causing the user input parameters to be lost #2303
• Fix the issue of overly frequent warnings caused by MySQL sink deadlock #2706
• Fix the issue that Avro sink does not support parsing JSON type columns #3624
• Fix the bug that TiCDC reads the incorrect schema snapshot from TiKV when the TiKV owner restarts #2603
• Fix the bug that the enable-old-value configuration item is not automatically set to true on Canal and Maxwell protocols #3676
• Fix the timezone error that occurs when the cdc server command runs on some Red Hat Enterprise Linux releases (such as 6.8 and 6.9) #3584
• Fix the issue of the inaccurate txn_batch_size monitoring metric for Kafka sink #3431
• Fix the TiCDC panic issue that occurs when manually cleaning the task status in etcd #2980