Skip to content

Commit 2c9ccad

Browse files
authored
Merge pull request #40 from github/look-patch-1
Add missing word
2 parents 451b1bf + 5db2618 commit 2c9ccad

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

crates/bpe/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -243,7 +243,7 @@ This type of algorithm is interesting for use cases where a certain token budget
243243
This benchmark shows the runtime for the appending encoder when a text is encoded byte-by-byte.
244244
For comparison we show the runtime of the backtracking encoder when it encodes the whole text at once.
245245

246-
The benchmark measured the runtime of encoding of slices of lengths 10, 100, 1000, and 10000 from a random 20000 token original using the o200k token set.
246+
The benchmark measured the runtime of encoding of slices of lengths 10, 100, 1000, and 10000 from a random 20000 token original text using the o200k token set.
247247

248248
The graph below shows encoding runtime vs slice length.
249249
The overall runtime of byte-by-byte incremental encoder for encoding the full text is comparable to the runtime of the backtracking encoder, with only a constant factor overhead.

0 commit comments

Comments
 (0)