x/text/encoding/traditionalchinese: wrong coding mapping

[beikege]

What version of Go are you using (go version)?

1.9
go get -u golang.org/x/text/

What did you do?

package main

import (
	"fmt"
	"log"
	"unicode/utf8"

	"golang.org/x/text/encoding/traditionalchinese"
)

func main() {
	str := "包"
	b, err := traditionalchinese.Big5.NewEncoder().Bytes([]byte(str))
	if err != nil {
		log.Fatalln(err)
	}
	r, _ := utf8.DecodeRuneInString(str)
	fmt.Printf("unicode:0x%X big5:0x%X\n", r, b) //incorrect
}

What did you expect to see?

unicode:0x5305 big5:0xA55D

What did you see instead?

unicode:0x5305 big5:0xFABD

reference:
http://moztw.org/docs/big5/table/cp950-u2b.txt
http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP950.TXT
https://encoding.spec.whatwg.org/index-big5.txt

@mpvl

[ghost]

Does a grep of the following files clarify anything?

encoding/simplifiedchinese/tables.go

encoding/traditionalchinese/tables.go

[beikege]

https://github.com/golang/text/blob/f3d52e02631e937ee2825b81ff4cd86f0ad40790/encoding/traditionalchinese/tables.go#L4541

5681:  0x00005305 

https://github.com/golang/text/blob/f3d52e02631e937ee2825b81ff4cd86f0ad40790/encoding/traditionalchinese/tables.go#L17905

19088: 0x00005305  //repeat

Sorry, my english is very poor.
The correct value is overwritten by the wrong value.

[ghost]

% grep A55D golang.org/x/text/encoding/traditionalchinese/tables.go
% grep B0A8 golang.org/x/text/encoding/traditionalchinese/tables.go
39340 - 11904: 0xB0A8,
%

Try substituting the character "馬".

[beikege]

package main

import (
	"bytes"
	"fmt"
	"log"
	"unicode/utf8"

	"golang.org/x/text/encoding/traditionalchinese"
)

func main() {

	src := []byte{165, 93} //big5 : 包

	// big5 to utf8
	b1, err := traditionalchinese.Big5.NewDecoder().Bytes(src)
	if err != nil {
		log.Fatalln(err)
	}

	r, _ := utf8.DecodeRune(b1)
	fmt.Printf("包 unicode:0x%X big5:0x%X\n", r, src)

	// utf8 to big5
	b2, err := traditionalchinese.Big5.NewEncoder().Bytes(b1)
	if err != nil {
		log.Fatalln(err)
	}

	// not equal
	fmt.Println(src, b2, bytes.Equal(src, b2))

	fmt.Println("--------------------------")

	src = []byte{176, 168} //big5 : 馬

	// big5 to utf8
	b1, err = traditionalchinese.Big5.NewDecoder().Bytes(src)
	if err != nil {
		log.Fatalln(err)
	}

	r, _ = utf8.DecodeRune(b1)
	fmt.Printf("馬 unicode:0x%X big5:0x%X\n", r, src)

	// utf8 to big5
	b2, err = traditionalchinese.Big5.NewEncoder().Bytes(b1)
	if err != nil {
		log.Fatalln(err)
	}

	// equal
	fmt.Println(src, b2, bytes.Equal(src, b2))

}

包 unicode:0x5305 big5:0xA55D
[165 93] [250 189] false
--------------------------
馬 unicode:0x99AC big5:0xB0A8
[176 168] [176 168] true

[ghost]

Have you read this?

https://golangtc.com/t/541560ab320b527a3b0001d9

[ghost]

There's a "big5" thread here.

https://lists.w3.org/Archives/Public/public-webapps-github/2016Oct/0063.html

[ianlancetaylor]

CC @mpvl

[ghost]

包 bag
馬 horse

[beikege]

Fixes #43581