Skip to content

Fix conversion bug when string data comes from binlog #1571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mattbooks
Copy link

@mattbooks mattbooks commented Jul 14, 2025

Related issue: #1568

Description

This PR fixes a string conversion bug by applying character set conversions when the input is []uint8, in addition to the existing case of when the input is string.

When using gh-ost to migrate a table from latin1 to utf8mb3 character encoding, the initial data copy works correctly, but new data with special characters inserted during the migration via binlog replication fails with "Incorrect string value" errors.

The reason for this is that the data is a binary byte array when converted from the binlog, so the character set conversion is not applied.

I added a new test for latin1-encoded byte array input to this method and confirmed that the reproduction from the linked issue is fixed with this change.

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

Fixes github#1568

When using gh-ost to migrate a table from latin1 to utf8mb3 character encoding, the initial data copy works correctly, but new data with special characters inserted during the migration via binlog replication fails with "Incorrect string value" errors.

The reason for this is that the data is a binary byte array when
converted from the binlog, so the character set conversion is not
applied.

This fix updates the character set conversion logic to apply to both
string and []uint8 types when the column has a character set conversion.

I added a new test for latin1 input to this method and confirmed that
the reproduction from the linked issue is fixed with this change.
@Copilot Copilot AI review requested due to automatic review settings July 14, 2025 22:58
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes a character set conversion bug in gh-ost when handling string data from binlog replication. The issue occurs when migrating tables from latin1 to utf8mb3 encoding, where special characters in binlog data fail with "Incorrect string value" errors because the data comes as a byte array ([]uint8) rather than a string, bypassing the existing character set conversion logic.

  • Extends the convertArg method to handle []uint8 input in addition to existing string input
  • Updates character set conversion to use Bytes() method instead of String() for more consistent handling
  • Adds comprehensive test coverage for latin1-encoded byte array conversion

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
go/sql/types.go Modified convertArg method to handle []uint8 input and use byte-based charset conversion
go/sql/types_test.go Added test case for latin1 byte array character set conversion
Comments suppressed due to low confidence (1)

go/sql/types.go:56

  • [nitpick] The variable name 'arg2Bytes' is unclear and could be confusing. Consider renaming it to 'argBytes' or 'inputBytes' to better reflect its purpose as the byte representation of the input argument.
	var arg2Bytes []byte

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant