Skip to content

Conversation

@rjzamora
Copy link
Member

@rjzamora rjzamora commented Feb 28, 2025

Description

I propose that we add a simple management utility for GPUEngine configuration options in cudf-polars.

In order to support a variety of configurable groupby/shuffle/sort/join options for multi-gpu execution, we will want to make it easy to pass through these options to IR nodes at translation time. We will also want it to be "easy" set/get specific options at traversal/evaluation time.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

rjzamora and others added 30 commits December 3, 2024 16:29
@vyasr
Copy link
Contributor

vyasr commented Mar 5, 2025

We may want to introduce a dedicated ExecutorOptions class to manage these kinds of configuration options, but I don't personally have the bandwidth to iterate on a solid API. Therefore, I'm proposing that we keep it as a simple dictionary for now, and use a convenience API to manage that dictionary until we have a better design in mind.

OK that's fine with me then.

@rjzamora
Copy link
Member Author

@wence- I'd like to get this in after #17503, unless you think 17503 needs more work (in which case, I'd like to merge this so I can clean up config handling across several multi-gpu PRs)

Copy link
Contributor

@wence- wence- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Rick

@rjzamora rjzamora added 5 - Ready to Merge Testing and reviews complete, ready to merge 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress 5 - Ready to Merge Testing and reviews complete, ready to merge labels Mar 13, 2025
@rjzamora
Copy link
Member Author

/merge

@rjzamora rjzamora added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 3 - Ready for Review Ready for review by team labels Mar 13, 2025
@rapids-bot rapids-bot bot merged commit 63459ca into rapidsai:branch-25.04 Mar 13, 2025
127 checks passed
@rjzamora rjzamora deleted the cudf-polars-config-options branch March 13, 2025 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

5 - Ready to Merge Testing and reviews complete, ready to merge cudf-polars Issues specific to cudf-polars improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Python Affects Python cuDF API.

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

5 participants