Support for chunksize

[mohr023]

Considering some of SIA files like PA##.dbc, loading them into a dataframe directly basically burns 20gb RAM.

Should we add support for pandas Dataframe's chunksize parameter, to handle this correctly? If so, can you identify some sort of caveat about this approach @fccoelho ?

[mohr023]

One alternative I see for implementing this is by returning a generator of DataFrames, such as this:

(pd.DataFrame([next(iter(testdbf.records)) for j in range(i,i+chunksize)]) for i in range(0,len(testdbf), chunksize))

For this test, I'm using DBF object from dbfread.

[fccoelho]

This is a real problem @mohr023. If we iterate over the DBF record as we read, we would also need to iterate over them when saving the cachefile, and cannot return the full dataframe after downloading.

If you have a good Idea for solving this, feel free to submit a Pull-request.