Skip to content

Chinese character encoding result incorrect #3

Open
@an9xyz

Description

@an9xyz

Example:
你好,我叫凯文。
Expected:
[19526, 254, 25001, 121, 171, 120, 234, 22755, 239, 20998, 104, 49035, 107, 23877, 229, 16764]
Actual:
[8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423, 8423]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions