Skip to content

Optimize dedup to avoid oom #568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Optimize dedup to avoid oom #568

wants to merge 1 commit into from
+4 −4

Conversation

@coolderli
Copy link

@coolderli coolderli commented Feb 7, 2025

  • distinct() will storage the data, and the downstream will read from the shuffle. We do not need the cache any more.
  • Use the count() to instead the collect() to avoid the drive OOM.

@HYLcool HYLcool added enhancement New feature or request good first issue Good for newcomers dj:dist issues/PRs about distributed data processing dj:efficiency regarding to efficiency issues and enhancements dj:tools issues/PRs about specific tools labels Feb 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chenyushuo chenyushuo Awaiting requested review from chenyushuo

@pan-x-c pan-x-c Awaiting requested review from pan-x-c

@garyzhang99 garyzhang99 Awaiting requested review from garyzhang99

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

dj:dist issues/PRs about distributed data processing dj:efficiency regarding to efficiency issues and enhancements dj:tools issues/PRs about specific tools enhancement New feature or request good first issue Good for newcomers

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@coolderli @HYLcool