File tree Expand file tree Collapse file tree 1 file changed +7
-6
lines changed Expand file tree Collapse file tree 1 file changed +7
-6
lines changed Original file line number Diff line number Diff line change @@ -83,12 +83,13 @@ enum State {
8383/// stream: Python file-like object / stream to read JSON from. Can be
8484/// either in text mode or in binary mode (so long as the bytes are valid
8585/// UTF-8).
86- /// correct_cursor: Whether it is required that the cursor is left in the
87- /// correct position (behind the last processed character) after
88- /// park_cursor() has been called. If set to False, performance for
89- /// unseekable streams is drastically improved at the cost of the cursor
90- /// ending up in places unrelated to the actual tokenization progress. For
91- /// seekable streams, the improvement shouldn't be noticable.
86+ /// correct_cursor: *(not part of API yet, may be removed at any point)*
87+ /// Whether it is required that the cursor is left in the correct position
88+ /// (behind the last processed character) after park_cursor() has been
89+ /// called. If set to False, performance for unseekable streams is
90+ /// drastically improved at the cost of the cursor ending up in places
91+ /// unrelated to the actual tokenization progress. For seekable streams, the
92+ /// improvement shouldn't be noticable.
9293#[ pyclass]
9394#[ pyo3( text_signature = "(stream, *, correct_cursor=True)" ) ]
9495struct RustTokenizer {
You can’t perform that action at this time.
0 commit comments