tiflow v7.1.0-rc.0

For new features, improvements, and bug fixes released in 7.1.0-rc.0 for tiflow, see TiDB 7.1.0-rc.0 release notes.

tiflow v6.5.2

For new features, improvements, and bug fixes released in 6.5.2 for tiflow, see TiDB 6.5.2 release notes.

tiflow v6.1.6

For new features, improvements, and bug fixes released in 6.1.6 for tiflow, see TiDB 6.1.6 release notes.

tiflow v7.0.0

For new features, improvements, and bug fixes released in 7.0.0 for tiflow, see TiDB 7.0.0 release notes.

tiflow v6.5.1

Improvements

• Support batch UPDATE statements to improve TiCDC replication performance #8084 @amyangfei
• Enable pull-based sink to optimize system throughput #8232 @hi-rustin
• Support storing redo logs to GCS-compatible or Azure-compatible object storage #7987 @CharlesCheung96
• Implement MQ sink and MySQL sink in the asynchronous mode to improve the sink throughput #5928 @amyangfei @CharlesCheung96

Bug fixes

• Fix the issue that changefeed might get stuck in special scenarios such as when scaling in or scaling out TiKV or TiCDC nodes #8174 @hicqu

• Fix the issue that precheck is not performed on the storage path of redo log #6335 @CharlesCheung96

• Fix the issue of insufficient duration that redo log can tolerate for S3 storage failure #8089 @CharlesCheung96

• Fix the issue that transaction_atomicity and protocol cannot be updated via the configuration file #7935 @CharlesCheung96

• Fix the issue that the checkpoint cannot advance when TiCDC replicates an excessively large number of tables #8004 @overvenus

• Fix the issue that applying redo log might cause OOM when the replication lag is excessively high #8085 @CharlesCheung96

• Fix the issue that the performance degrades when redo log is enabled to write meta #8074 @CharlesCheung96

• Fix a bug that the context deadline is exceeded when TiCDC replicates data without splitting large transactions #7982 @hi-rustin

• Fix the issue that pausing a changefeed when PD is abnormal results in incorrect status #8330 @sdojjy

• Fix the data inconsistency that occurs when replicating data to a TiDB or MySQL sink and when CHARACTER SET is specified on the column that has the non-null unique index without a primary key #8420 @asddongmen

• Fix the panic issue in table scheduling and blackhole sink #8024 #8142 @hicqu

• Fix the issue that the binlog-schema delete command fails to execute #7373 @liumengya94

• Fix the issue that the checkpoint does not advance when the last binlog is a skipped DDL #8175 @D3Hunter

• Fix a bug that when the expression filters of both "update" and "non-update" types are specified in one table, all UPDATE statements are skipped #7831 @lance6716

tiflow v6.1.5

Bug fixes

• Fix the issue that applying redo log might cause OOM when the replication lag is excessively high #8085 @CharlesCheung96
• Fix the issue that the performance degrades when redo log is enabled to write meta #8074 @CharlesCheung96
• Fix the issue that the binlog-schema delete command fails to execute #7373 @liumengya94
• Fix the issue that the checkpoint does not advance when the last binlog is a skipped DDL #8175 @D3Hunter

tiflow v6.6.0

Improvements

• Support batch UPDATE DML statements to improve TiCDC replication performance #8084 @amyangfei

• Implement MQ sink and MySQL sink in the asynchronous mode to improve the sink throughput #5928 @hicqu @hi-rustin

• Optimize DM alert rules and content #7376 @D3Hunter

Previously, alerts similar to "DM_XXX_process_exits_with_error" were raised whenever a related error occured. But some alerts are caused by idle database connections, which can be recovered after reconnecting. To reduce this kind of alerts, DM divides errors into two types: automatically recoverable errors and unrecoverable errors:

    * For an error that is automatically recoverable, DM reports the alert only if the error occurs more than 3 times within 2 minutes.
    * For an error that is not automatically recoverable, DM maintains the original behavior and reports the alert immediately.

• Optimize relay performance by adding the async/batch relay writer #4287 @GMHDBJD

Bug fixes

• Fix the issue that transaction_atomicity and protocol cannot be updated via the configuration file #7935 @CharlesCheung96
• Fix the issue that precheck is not performed on the storage path of redo log #6335 @CharlesCheung96
• Fix the issue of insufficient duration that redo log can tolerate for S3 storage failure #8089 @CharlesCheung96
• 修复 changefeed 在 tikv、CDC 节点扩缩容特殊场景下卡住的问题。 #8197 @hicqu
• Fix the issue of too high traffic among TiKV nodes #14092 @overvenus
• Fix the performance issues of TiCDC in terms of CPU usage, memory control, and throughput when the pull-based sink is enabled #8142 #8157 #8001 #5928 @hicqu @hi-rustin
• Fix the issue that the binlog-schema delete command fails to execute #7373 @liumengya94
• Fix the issue that the checkpoint does not advance when the last binlog is a skipped DDL #8175 @D3Hunter
• Fix a bug that when the expression filters of both "update" and "non-update" types are specified in one table, all UPDATE statements are skipped #7831 @lance6716
• Fix a bug that when only one of update-old-value-expr or update-new-value-expr is set for a table, the filter rule does not take effect or DM panics #7774 @lance6716

tiflow v6.1.4

Improvements

• Add the DML batch operation mode to improve the throughput when SQL statements are generated in batches #7653 @asddongmen
• Support storing redo logs to GCS- or Azure-compatible object storage #7987 @CharlesCheung96

Bug fixes

• Fix the issue that the checkpoint cannot advance when TiCDC replicates an excessively large number of tables #8004 @asddongmen
• Fix the issue that transaction_atomicity and protocol cannot be updated via the configuration file #7935 @CharlesCheung96
• Fix the issue that TiCDC mistakenly reports an error when the version of TiFlash is later than that of TiCDC #7744 @overvenus
• Fix the issue that OOM occurs when TiCDC replicates large transactions #7913 @overvenus
• Fix a bug that the context deadline is exceeded when TiCDC replicates data without splitting large transactions #7982 @hi-rustin
• Fix the issue that sasl-password in the changefeed query result is not masked #7182 @dveeden
• Fix the issue that data is lost when a user quickly deletes a replication task and then creates another one with the same task name #7657 @overvenus
• Fix a bug that DM might raise an error during precheck when the downstream database name in SHOW GRANTS contains a wildcard ("*") #7645 @lance6716
• Fix the issue that DM prints too many logs caused by "COMMIT" in binlog query events #7525 @liumengya94
• Fix the issue that the DM task fails to start when only ssl-ca is configured for SSL #7941 @liumengya94
• Fix a bug that when the expression filters of both "update" and "non-update" types are specified in one table, all UPDATE statements are skipped #7831 @lance6716
• Fix a bug that when only one of update-old-value-expr or update-new-value-expr is set for a table, the filter rule does not take effect or DM panics #7774 @lance6716

tiflow v6.5.0

For new features, improvements, and bug fixes released in 6.5.0 for ticdc and dm, see TiDB 6.5.0 release notes.

tiflow v5.1.5

Bug fixes

• Fix data loss that occurs in special incremental scanning scenarios #5468
• Fix the issue that there are no sorter metrics #5690
• Fix excessive memory usage by optimizing the way DDL schemas are buffered #1386