Implementing the BPE Tokenizer from Scratch

[rasbt]

Yesterday, a reader recently shared an insightful discussion (#485 ) about training an LLM tokenizer for Nepali. This reminded me that I had originally implemented a Byte Pair Encoding (BPE) Tokenizer for my LLMs from Scratch book but never uploaded it to the bonus materials.

While my book explains BPE, I opted to use the highly-performant tiktoken library (which is used for GPT-4 and now also used for Llama 3 as well) for practical purposes: the book focuses on LLMs rather than tokenizer development. However, one limitation of tiktoken (please correct me if I'm wrong) is that it doesn’t support training a tokenizer on custom datasets, so if you want to support new languages or other special structures, you are kind of out of luck.

Several readers asked me about this over the last few months, so to provide a hands-on resource for tinkerers, I've uploaded my standalone notebook showcasing my BPE implementation with training support. It's for educational purposes and hopefully makes for a fun weekend project if you're interested in experimenting with tokenization.

Surprisingly, my tokenizer implementation is only 2x slower than tiktoken (Rust implementation) and 5x faster than the Hugging Face tokenizer... what an unexpected surprise!

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[Sunilyadav03]

Thanks a lot sir, Because you opened new window for students those are just jumped in the research field of LLMs.

[d-kleine]

@rasbt You can also add .decode_single_token_bytes() to display the byte representation of each token (useful for tokens that cannot be printed):
https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken#4-turn-tokens-into-text-with-encodingdecode

for i in range(300):
    decoded = gpt2_tokenizer.decode_single_token_bytes(i)
    print(f"{i}: {decoded}")

[rasbt]

Thanks for the suggestion, but I am not sure this has that much practical use case and might clutter the code even more 😅

[d-kleine]

I think it would be helpful for understanding as the BPE tokenizer stores tokens represented as bytes for compression (that's why the first 256 elements are blocked as the represent all possible byte values). I think also the trained vocab.json should be converted to to bytes, or at least adding an information to the text.

[rasbt]

Yes, I should iterate on that more ... but yeah this itself could be a whole book haha