Allow Sinter to read CSV fields larger than 128 KiB

[mg551-duke]

csv.field_size_limit() is not raised in sinter/_data/_existing_data.py, so reading a CSV with a large custom_counts field triggers

_csv.Error: field larger than field limit (131072)
Adding one line at import time would fix it:

import sys, csv
csv.field_size_limit(sys.maxsize)

(With a small try/except OverflowError loop for 32-bit builds.)
Let me know and I can open a PR with this two-line patch.

Credit: https://github.com/Zihan-Chen-PhMA/Cultiv_T_RP2/blob/main/installation_guide.md

[Strilanc]

That fix looks like it would globally affect the setting? It doesn't seem appropriate to change a global field like that on import. At the very least it should only be temporarily changed during the read.

[mg551-duke]

How about this:

...
    import csv
    ################## RP2 Paper ##################
    # Temporarily raise the max CSV-field limit.
    import sys
    old_limit = csv.field_size_limit()          # save current value

    maxInt = sys.maxsize
    while True:  # shrink until the platform accepts the value
        try:
            csv.field_size_limit(maxInt)
            break
        except OverflowError:
            maxInt //= 10
    ################## RP2 Paper ##################

    if isinstance(path_or_file, (str, pathlib.Path)):
...

        json_metadata=json.loads(row['json_metadata']),
    ))

csv.field_size_limit(old_limit)             # restore original
return result

[Strilanc]

That seems really janky to me. I'll think about it. It definitely at least needs to restore the value in a finally block. But it seems really unpythonic for this to come up at all.

[mg551-duke]

Happy to adjust—if you’d prefer a different best-practice fix instead of modifying from_file directly, just let me know and I’ll follow that approach. Opportunity to learn on my end