fix(breadbox): workaround for memory problem reading parquet files. #122
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pgm
changed the title
snwessel
reviewed
Nov 5, 2024
| # to > 30GB and would take down breadbox. Reading it in as a table, and then | ||
| # processing it column by column seems to avoid this problem. Not 100% sure that this | ||
| # results in identical behavior, but it seems valid for numerical matrices | ||
| table = pq.read_table(filename) |
There was a problem hiding this comment.
If this fixes the issue, I think it looks good and I feel good about you merging it (especially since it's needed for the release).
But I am curious... I found this github issue, suggesting the problem is the pyarrow engine that pandas uses. I'd be curious if you could use a simpler solution of just doing:
pandas.read_parquet(engine='fastparquet')
There was a problem hiding this comment.
The memory leak described there looks fairly different then how I'd characterize what I've observed. However, that's a good point about maybe the issue stems from pyarrow and not pandas.
I tried
pandas.read_parquet(engine='fastparquet')
and yes, that works fine. We'll have to add fastparquet as a dependency, but that sounds fine.
I'll switch the code to use that instead.
snwessel
approved these changes
Nov 5, 2024
pgm
added a commit
that referenced
this pull request
Nov 7, 2024
…122) * workaround for memory problem reading parquet files. * Switched to using fastparquet * fixed type check error
pgm
added a commit
that referenced
this pull request
Nov 7, 2024
…122) * workaround for memory problem reading parquet files. * Switched to using fastparquet * fixed type check error
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Very small change to work around issue with pd.read_parquet(). (memory utilization exploding when trying to read a large-ish parquet file)