FAQ (please read before opening a new issue) #98
Description
Usage help
Check out this awesome tokeniser app https://tiktokenizer.vercel.app/ built by Diagram!
Check out the OpenAI cookbook!
In particular, the following are great examples of using tiktoken:
- How to count tokens with tiktoken
- How to format inputs to ChatGPT models
- Embedding long inputs
- Entity extraction for long documents
Other useful resources:
Support for other languages
- See Unofficial bindings / ports in other languages #97 for suggestions
Publishing an official NPM package
I have no current plans to do this, but it is likely that in the future OpenAI will provide an official package. Discuss at #22
Publishing an official Rust crate
In the future when I have time, I will publish the core BPE logic as its own crate. Discuss at #24
I'm short on time currently, but in the longer term I want to explore ways to make maintaining downstream bindings easier. Apologies to the people who've reached out about this — y'all are great, sorry for being a bottleneck.
Packaging
Issues with pyinstaller
I've never used pyinstaller, but people report --hidden-import=tiktoken_ext.openai_public --hidden-import=tiktoken_ext helps make things work. See #43 for more details.
Wheels for other platforms
I'm reluctant to publish wheels for additional platforms, particularly if I can't easily build or test code for them. I'm unlikely to do this unless there's significant interest. Please add reactions to the following issues if you'd like to see those wheels. No promises, but I'll reopen any issue that gets >10 reactions.