Add initial Arrow PyCapsule support #517
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jonmmease
commented
Oct 12, 2024
| for i in range(20): | ||
| # Break cache by removing one row each iteration | ||
| movies_inner = movies.iloc[i:] | ||
| movies["Title"] = movies["Title"].astype(str) |
There was a problem hiding this comment.
wish polars/pyarrow could handle loading mixed str/num columns from pandas
jonmmease
commented
Oct 12, 2024
| func: Arc::new((*STR_TO_UTC_TIMESTAMP_UDF).clone()), | ||
| args: vec![field_col, lit(default_input_tz)], | ||
| }) | ||
| } |
There was a problem hiding this comment.
I'm sure there are a lot more of these that will need to be updated, but just did this one to get tests passing
jonmmease
added a commit
that referenced
this pull request
Nov 16, 2024
* Add initial Arrow PyCapsule support * Bump python in actions to 3.11 * fmt * update hang test to use polars with pycapsule path * Add system python of 3.11 * Add more efficient hashing and rechunk for DataFusion * fmt * use narwhals to remove unused columns. * toward removing arrow/pyarrow flag * Remove arrow-rs pyarrow flag, use pyo3-arrow * Rename pyarrow feature flag to `py` * toward using narwhals to process transformed data * Remove python datasource * Handle dict and non-narwhals pycapsule types * update default extraction to arro3 * fix type checking * build py before type checking * skip empty fields in window transform * Try normalize category order * lower min Python back to 3.9 * clear wheel build dir first * try rename artifacts * cache?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Supersedes #501 now that everything needed has been updated.
Adopts arro3 / pyo3-arrow / narwhals for managing DataFrames in Python and the passing to Rust in arrow format with zero copy.
Uses
get_column_usageto determine which columns are needed for each input inline dataset, and then uses narwhals to downproject prior to wrapping in anarro3table to pass to rust. This approach removes the need for the Python data source framework, so I removed this all together.Performance of this workflow is improved from prior handling of pandas/polars, and is faster than the duckdb code path in my limited testing. We'll want to do more comprehensive benchmarks before release.