·
3 commits
to main
since this release
e47fe20
This commit was signed with the committer’s verified signature.
Systemcluster
Chris
This release adds support for conditional normalization to match the recent changes in Hugging Face Tokenizers.
Web version of the tokenizer is available on kitoken.dev.