Search results

Support for newer GPT models

Status: Open.

#105 In zurawiki/tiktoken-rs;

· Mohana-Manikandan opened on May 5, 2025

Gemini Tokenizer Support

Status: Open.

#101 In zurawiki/tiktoken-rs;

· notpeter opened on Feb 9, 2025

Getting decoder error on specific token id when using GPT2 BPE tokenizer

Status: Open.

#97 In zurawiki/tiktoken-rs;

· nerdai opened on Nov 30, 2024

Dependency Dashboard

Status: Open.

#88 In zurawiki/tiktoken-rs;

· renovate opened on Oct 12, 2024

<code>num_tokens_from_messages()</code> needs updating

Status: Open.

#86 In zurawiki/tiktoken-rs;

· OTheDev opened on Oct 12, 2024

Is there a way to cache the bpe when using <code>num_tokens_from_messages</code>

Status: Open.

#81 In zurawiki/tiktoken-rs;

· xd009642 opened on Aug 13, 2024

Support for other LLMs?

Status: Open.

#62 In zurawiki/tiktoken-rs;

· carlca opened on Feb 24, 2024

async-openai 0.18 support

Status: Open.

#56 In zurawiki/tiktoken-rs;

· ChristopherJMiller opened on Jan 5, 2024

Update to async-openai 0.16.2

Status: Open.

#50 In zurawiki/tiktoken-rs;

· Dreaming-Codes opened on Nov 16, 2023

Make encode token generic, accepting usize and u32

Status: Open.

#45 In zurawiki/tiktoken-rs;

· mokeyish opened on Nov 9, 2023

Token counts are incorrect when request includes function messages

Status: Open.

#40 In zurawiki/tiktoken-rs;

· kylewlacy opened on Oct 5, 2023

BPE memory leak

help wantedExtra attention is neededExtra attention is needed

Status: Open.

#39 In zurawiki/tiktoken-rs;

· venual opened on Sep 29, 2023