Reduce memory usage in delta format code paths #723
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PatrickJin-db
force-pushed
the
PatrickJin-db/kernel-explicit-iterator-to_pandas-args
branch
from
0a141b3 to
ade4ee3
Compare
PatrickJin-db
changed the title
PatrickJin-db
force-pushed
the
PatrickJin-db/kernel-explicit-iterator-to_pandas-args
branch
2 times, most recently
from
d2748d0 to
d99d480
Compare
PatrickJin-db
changed the title
PatrickJin-db
force-pushed
the
PatrickJin-db/kernel-explicit-iterator-to_pandas-args
branch
from
d99d480 to
09e1a38
Compare
PatrickJin-db
force-pushed
the
PatrickJin-db/kernel-explicit-iterator-to_pandas-args
branch
2 times, most recently
from
72f7bfc to
bc0cc4b
Compare
linzhou-db
reviewed
May 29, 2025
| @@ -130,8 +129,10 @@ def __to_pandas_kernel(self): | |||
| schema = scan.execute(interface).schema | |||
| return pd.DataFrame(columns=schema.names) | |||
|
|
|||
| table = pa.Table.from_batches(scan.execute(interface)) | |||
| result = table.to_pandas() | |||
| batches = scan.execute(interface) | |||
There was a problem hiding this comment.
isn't there a parameter controlling the batch behavior?
delta-kernel-rs uses a batch size of 1024 rows and the parquet format code path uses a batch size of 65536 by default. Should this be added to the docstring? |
linzhou-db
approved these changes
May 29, 2025
| result = pd.DataFrame(columns=schema.names) | ||
| elif self._convert_in_batches: |
There was a problem hiding this comment.
do you think it's worth adding some logging on the batch behavior for debugging/verification purpose?
There was a problem hiding this comment.
added a print statement for number of batches
PatrickJin-db
force-pushed
the
PatrickJin-db/kernel-explicit-iterator-to_pandas-args
branch
2 times, most recently
from
1e40d45 to
492b0b9
Compare
PatrickJin-db
force-pushed
the
PatrickJin-db/kernel-explicit-iterator-to_pandas-args
branch
from
492b0b9 to
6679f11
Compare
PatrickJin-db
added a commit
to PatrickJin-db/delta-sharing
that referenced
this pull request
May 29, 2025
PatrickJin-db
added a commit
to PatrickJin-db/delta-sharing
that referenced
this pull request
May 29, 2025
PatrickJin-db
added a commit
that referenced
this pull request
May 29, 2025
* fix python lint and reformat scripts (#668) * Reduce memory usage in parquet format code path (#737) * refactor: Resolve lint errors in python release script. (#660) * Reduce memory usage in delta format code paths (#723) * Update Python connector version to 1.3.3 --------- Co-authored-by: Kyle Chui <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reduces memory consumption when using delta-kernel-rs by converting each batch into a pandas dataframe and concatenating the results together, rather than creating a pyarrow table from all the batches and converting that table with all the results. The pyarrow table usually remains in memory during the conversion to pandas, so it is best to convert smaller batches to pandas rather than the entire table.
This change is gated behind the
convert_in_batchesparam.Tested with integration test.
Also added some unit tests for #737 involving parquet format batch convert + limit.