Binary array encoding (i.e. json+binref) not supported by Python SDK

[jpbrodrick89]

Summary

Currently, the from_image and from_tesseract_api Python SDK allows output_format="json+binref" but this is not supported by _decode_array (after #422 is merged this option will be explicitly removed for clarity). The current implementation of _decode_array only offers support for json+base64 encoding explicitly and otherwise defaults to raw list handling:

def _decode_array(encoded_arr: dict) -> np.ndarray:
    if "data" in encoded_arr:
        if encoded_arr["data"]["encoding"] == "base64":
            data = base64.b64decode(encoded_arr["data"]["buffer"])
            arr = np.frombuffer(data, dtype=encoded_arr["dtype"])
        else:
            arr = np.array(encoded_arr["data"]["buffer"], dtype=encoded_arr["dtype"])

To support json+binref, we need the following:

1. Add failing tests
2. Allow output_path to be provided as argument to from_url (or to query this with a request)
3. Ensure output_path is provided when using json+binref encoding (or queried with a request if absent)
4. Add fsspec to SDK dependencies to allow for reading of cloud file storage (for interim local solution can just use Python's built-in open)
5. After safely reading file convert bytes to numpy array in _decode_array using np.frombuffer
6. Double-check this works properly with from_url (not sure if binaries get downloaded with request otherwise we might need to request them directly).

Why is this needed?

The performance improvement of json+binref is unlikely to be material until output sizes exceed at least 100MB-5GB (depending on network speed). In these cases the

1. 25% lower memory footprint of pure binaries over base64,
2. more manageable json file sizes,
3. minimal encode/decode time (the least material advantage as base64 encoding/decoding is already highly optimized)

could add up to a material improvement in serialization plus transfer time.

Happy to address when we determine there is sufficient advantage for this or to perform minimal tests on base64 vs binref encoding/decoding time.

Usage example

from tesseract_core import Tesseract

with Tesseract.from_image(..., output_format="json+binref"):
    ....

[dionhaefner]

To me, json+binref is only valuable if you're planning on writing results to disk anyway. If you're intending to keep things in memory (as we currently do when returning data from the Python API), going via disk should be a net negative, no?

[jpbrodrick89]

This is a really astute observation! I hadn't appreciated that even over a remote network a RAM->RAM communication has material advantages over one including disk writes (the incorrect picture in my head was that the json was written to disk).

Therefore, as long as our base64 encodings fit in memory (both in container and locally) and our Tesseract is not logging array outputs we should be able to entirely avoid disk writes using json+base64.

If arrays exceed available RAM, we'd hit swap regardless, at which point both approaches suffer. However, json+binref would at least give us control over which data hits disk (structured bin files) versus unpredictable swap behavior.

As such, the main driver for json+binref (when using the Python API) would likely come from speed/memory overhead of json parsing. Adopting a faster JSON parser such as orjson should buy us more time (at least up to 2GB json files) especially with shallow nesting.

If power-usage with multi-GB outputs becomes very common we could consider supporting pure binary output files (whether raw, MessagePack, CBOR or Protocol Buffers) but this would be a huge change in our underlying libraries/tooling and lose the feature of even being slightly human readable.

Whatever we decide to surface as a sensible option to the user when using from_image or from_tesseract_api, the core of this issue remains since served Tesseract may use json+binref for other reasons (e.g. persistence, CLI interaction).

As this is all hypothetical currently, I see no need for immediate action other than clearly documenting the status quo, having fallback strategies in mind and potentially adopting orjson now as it seems a quick and easy win.

[dionhaefner]

since served Tesseract may use json+binref for other reasons (e.g. persistence, CLI interaction)

You're conflating input and output data formats. Any Tesseract can read any input format. The toggle in this issue is only controlling outputs.

[dionhaefner]

Oh I see, we don't support decoding it in the Python API. Well that should be easy enough to add :)