Inconsistent error messages reported when executed within and outside the transaction

[sayJason]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

/* init */ DROP TABLE IF EXISTS t1;
/* init */ CREATE TABLE t1 (c1 BIT NOT NULL, c2 MEDIUMTEXT, PRIMARY KEY (c1, c2(4)));
/* init */ CREATE UNIQUE INDEX i1 ON t1(c1);
/* init */ INSERT INTO t1(c1, c2) VALUES (b'0', 'abcdefg'), (b'1', 'hijklmn');

-- not in transaction
/* s */ UPDATE t1 SET c2 = 'a1b2c3d4', c1 = b'1’; -- [1]

-- in transaction
/* t1 */ BEGIN;
/* t1 */ UPDATE t1 SET c2 = 'a1b2c3d4', c1 = b'1'; -- [2]

2. What did you expect to see? (Required)

The error messages reported by the UPDATE statements [1] and [2] are the same.

3. What did you see instead (Required)

The UPDATE statements [1] and [2] report different errors.

[1] UPDATE not in a transaction:

mysql> UPDATE t1 SET c2 = 'a1b2c3d4', c1 = b'1';
ERROR 1062 (23000): Duplicate entry '' for key 't1.i1'

[2] UPDATE in a transaction:

mysql> BEGIN;
Query OK, 0 rows affected (0.00 sec)

mysql> UPDATE t1 SET c2 = 'a1b2c3d4', c1 = b'1';
ERROR 1062 (23000): Duplicate entry '-a1b2' for key 't1.PRIMARY'

4. What is your TiDB version? (Required)

Release Version: v8.5.3
Edition: Community
Git Commit Hash: dc2548aac79a712265e831cff2a3a896bc0a5a38
Git Branch: HEAD
UTC Build Time: 2025-07-31 13:54:43
GoVersion: go1.23.8
Race Enabled: false
Check Table Before Drop: false
Store: tikv