Add option to receive strings as file-like streams

[RamiAwar]

I noticed this in the future work section: supporting long string values as streams themselves. Is this already implemented by any chance?

If not, do you have a plan on how you're going to implement it? Maybe you can point me in the right direction and I can give it a try.

My use-case is streaming chunks of a json string and reading the key values in a "persistent" way as described in your docs, but also partial values. For example if I were to receive {"name": "John Doe"} then I first receive {"name": "J then ohn Doe"},
and I want to be able to iterate over it in Python like:

for x in response["name"]:
    print(x)

# J
# ohn Doe

Or some similar API that achieves the same thing.

[daggaz]

Hey, thanks for the issue.

If you're looking for something that's going to stream chunks as they are received exactly as it was chunked in network packets, then this isn't going to be the library for you.

However, assuming what you really want is for partially transmitted string data (that is embedded in json) to be made available as soon as possible, then something might be able to be done, as per my "future work" idea.

I think the best API would be if a file-like object was returned in place of the string. You can then read this like any file, and it will EOF when the particular json string is completed.

The API would be something like:

data = json_stream.load(f, strings_as_files=True)
long_string = data['name']
while s := long_string.read(1024):
    print(s)

I will think about this...

[daggaz]

@smheidrich I feel this this might end up in significant changes to the python tokenizer, which implies work for you, so I'll only proceed if you think you can support this in the rust tokenizer too.

[smheidrich]

@daggaz Sure, that looks like it should be possible 👍

[daggaz]

In pursuance of this, I have just created two PRs.

The first, #46, changes the tokenizer to read the stream into a buffer instead of reading a character at a time

The second builds on this buffered to allow strings to be return as file-like objects, basically implementing the API above.

@smheidrich there are some pretty big changes here so we can only proceed if you agree

[smheidrich]

@daggaz I'll have a look as soon as possible and try to write an analogous MR, but I have to do some other stuff first (namely fix the build and align the Unicode surrogate pair code with #42), so it will take a while.

[RamiAwar]

Thanks for the quick response! A file-like will get the job done for me. Much cleaner than the quick implementation I went for.

Would it be possible to also wrap this file-like obj with a generator? Then it's even easier to consume. Do you see any downsides with that vs the file-like read?

[daggaz]

@smheidrich it occurs to me that this option should only be available for transient streams (or...we make them seekable in that case?) and that this has implications for #30

[daggaz]

@RamiAwar line-based file-like objects are already iterable, but JsonStreamReader is missing the required readline() implementation. I will update to add this. The API would then be:

data = json_stream.load(f, strings_as_files=True)
long_string = data['name']
for line in long_string:
    print(line)

If your data is not line based, obviously this doesn't work, in which case I think having access to the full file API is best.

[RamiAwar]

@RamiAwar line-based file-like objects are already iterable, but JsonStreamReader is missing the required readline() implementation. I will update to add this. The API would then be:

data = json_stream.load(f, strings_as_files=True)
long_string = data['name']
for line in long_string:
    print(line)

If your data is not line based, obviously this doesn't work, in which case I think having access to the full file API is best.

I like that yeah! Thanks!

[daggaz]

I've added the readline() support (and hence the iterator support).

[daggaz]

@smheidrich the testswill not pass on this branch until the rust tokenizer is implemented

[daggaz]

@smheidrich I'm considering the .completed part of the API optional. If you don't want to implement it, I can strip it out of the tests