Skip to content

Commit 94be4d3

Browse files
gittingraphael0202
authored andcommitted
fix: increase csv field_size_limit to accommodate large fields
1 parent 6faadc1 commit 94be4d3

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

openfoodfacts/dataset.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@
1414

1515
logger = get_logger(__name__)
1616

17+
# Increase field_size to accommodate large fields.
18+
# sys.maxsize will overflow on windows so using max 32-bit integer instead.
19+
csv.field_size_limit(pow(2, 31) - 1)
20+
1721

1822
DEFAULT_CACHE_DIR = Path("~/.cache/openfoodfacts/datasets").expanduser()
1923
DATASET_FILE_NAMES = {

0 commit comments

Comments
 (0)