Skip to content

refactor(arrow,datafusion): Reuse PartitionValueCalculator in RecordBatchPartitionSplitter #1781

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Oct 28, 2025
Merged

refactor(arrow,datafusion): Reuse PartitionValueCalculator in RecordBatchPartitionSplitter #1781

merged 6 commits into from
Oct 28, 2025
+580 −248

Conversation

@CTTY
Copy link
Contributor

@CTTY CTTY commented Oct 23, 2025 •
edited
Loading

Which issue does this PR close?

What changes are included in this PR?

  • Move PartitionValueCalculator to core/arrow so it can be reused by RecordBatchPartitionSplitter
  • Allow skipping partition value calculation in partition splitter for projected batches
  • Return <PartitionKey, RecordBatch> rather than <Struct, RecordBatch> pairs in RecordBatchPartitionSplitter::split

Are these changes tested?

Added uts

@CTTY CTTY marked this pull request as ready for review October 26, 2025 01:53
liurenjie1024
liurenjie1024 reviewed Oct 27, 2025
View reviewed changes
Copy link
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @CTTY for this pr, LGTM! Just one minor comment

liurenjie1024
liurenjie1024 approved these changes Oct 28, 2025
View reviewed changes
Copy link
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @CTTY for this fix!

@liurenjie1024 liurenjie1024 merged commit 0a06c3e into apache:main Oct 28, 2025
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liurenjie1024 liurenjie1024 liurenjie1024 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Refactor PartitionValueCalculator and RecordBatchPartitionSplitter

2 participants

@CTTY @liurenjie1024