Skip to content

Commit 252d5a7

Browse files
committed
Make clear that correct_cursor not part of API yet
1 parent e4feeae commit 252d5a7

File tree

1 file changed

+7
-6
lines changed

1 file changed

+7
-6
lines changed

src/lib.rs

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -83,12 +83,13 @@ enum State {
8383
/// stream: Python file-like object / stream to read JSON from. Can be
8484
/// either in text mode or in binary mode (so long as the bytes are valid
8585
/// UTF-8).
86-
/// correct_cursor: Whether it is required that the cursor is left in the
87-
/// correct position (behind the last processed character) after
88-
/// park_cursor() has been called. If set to False, performance for
89-
/// unseekable streams is drastically improved at the cost of the cursor
90-
/// ending up in places unrelated to the actual tokenization progress. For
91-
/// seekable streams, the improvement shouldn't be noticable.
86+
/// correct_cursor: *(not part of API yet, may be removed at any point)*
87+
/// Whether it is required that the cursor is left in the correct position
88+
/// (behind the last processed character) after park_cursor() has been
89+
/// called. If set to False, performance for unseekable streams is
90+
/// drastically improved at the cost of the cursor ending up in places
91+
/// unrelated to the actual tokenization progress. For seekable streams, the
92+
/// improvement shouldn't be noticable.
9293
#[pyclass]
9394
#[pyo3(text_signature = "(stream, *, correct_cursor=True)")]
9495
struct RustTokenizer {

0 commit comments

Comments
 (0)