Skip to content

feat: Allow cancelling of grouping operations which are CPU bound #16196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 77 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
b18aeaa
feat: support inability to yeild cpu for loop when it's not using Tok…
zhuqi-lucas May 27, 2025
2a965aa
Fix fuzz test
zhuqi-lucas May 28, 2025
67ca44b
polish code
zhuqi-lucas May 28, 2025
82a179d
add comments
zhuqi-lucas May 28, 2025
da3c2d5
fix corner case when huge data
zhuqi-lucas May 28, 2025
6cf3bf0
Also add grouping case
zhuqi-lucas May 28, 2025
3251990
Address comments
zhuqi-lucas May 30, 2025
a8da370
Merge remote-tracking branch 'upstream/main' into issue_16193
zhuqi-lucas May 30, 2025
311849d
fmt
zhuqi-lucas May 30, 2025
2a2ead9
Move YieldStream into physical-plan crate
alamb May 30, 2025
3b69287
Merge remote-tracking branch 'apache/main' into issue_16193
alamb May 30, 2025
3ff9252
Use existing RecordBatchStreamAdapter
alamb May 30, 2025
5547c3c
Add timeout testing for cancellation
zhuqi-lucas May 31, 2025
aeac0ef
fmt
zhuqi-lucas May 31, 2025
6d56b78
add license
zhuqi-lucas May 31, 2025
4ddd1e5
Support sort exec for cancellation
zhuqi-lucas May 31, 2025
b2ffec7
poc: unified yield exec for leaf node
zhuqi-lucas Jun 2, 2025
4587c3b
polish code phase 1
zhuqi-lucas Jun 2, 2025
098b1ec
Add license
zhuqi-lucas Jun 2, 2025
bc65c1a
Fix testing
zhuqi-lucas Jun 2, 2025
da58d0b
Support final path
zhuqi-lucas Jun 2, 2025
021fb92
fix test
zhuqi-lucas Jun 2, 2025
5027087
polish code
zhuqi-lucas Jun 2, 2025
8509d0a
fix testing and address suggestions
zhuqi-lucas Jun 2, 2025
97c1bb7
fix
zhuqi-lucas Jun 2, 2025
5c3a14c
remove buffer
zhuqi-lucas Jun 3, 2025
97923b8
address comments
zhuqi-lucas Jun 3, 2025
118f801
fmt
zhuqi-lucas Jun 3, 2025
e7a678a
Fix test
zhuqi-lucas Jun 3, 2025
54260aa
fix
zhuqi-lucas Jun 3, 2025
cb344af
fix
zhuqi-lucas Jun 3, 2025
89a4e93
fix slt
zhuqi-lucas Jun 3, 2025
896fd59
fix tpch sql
zhuqi-lucas Jun 3, 2025
2887f87
Add flag for yield insert and disable default
zhuqi-lucas Jun 4, 2025
2de4afb
recover testing
zhuqi-lucas Jun 4, 2025
485d55a
fix
zhuqi-lucas Jun 4, 2025
e31a997
Update doc
zhuqi-lucas Jun 4, 2025
b65ab60
Address comments
zhuqi-lucas Jun 4, 2025
1bf2ad3
fix fmt
zhuqi-lucas Jun 4, 2025
8ab64b3
Support config for yield frequency
zhuqi-lucas Jun 4, 2025
c7774a7
add built-in yield support
zhuqi-lucas Jun 5, 2025
7f64f5c
Merge remote-tracking branch 'upstream/main' into issue_16193
zhuqi-lucas Jun 5, 2025
3eb785c
Add LazyMemoryExec built-in Yield
zhuqi-lucas Jun 5, 2025
5737309
Update datafusion/datasource/src/source.rs
zhuqi-lucas Jun 5, 2025
74eb5c2
Update datafusion/core/tests/physical_optimizer/enforce_distribution.rs
zhuqi-lucas Jun 5, 2025
23d7a4b
Update datafusion/physical-optimizer/src/optimizer.rs
zhuqi-lucas Jun 5, 2025
e7c7eea
Update datafusion/physical-plan/src/memory.rs
zhuqi-lucas Jun 5, 2025
7df28e5
Update datafusion/physical-plan/src/memory.rs
zhuqi-lucas Jun 5, 2025
d16e6e5
Update datafusion/proto/src/physical_plan/mod.rs
zhuqi-lucas Jun 5, 2025
f6ccecb
Address comments
zhuqi-lucas Jun 5, 2025
8c74620
Add interleave reproducer
zhuqi-lucas Jun 5, 2025
5e2dde8
remove unused config
zhuqi-lucas Jun 5, 2025
aa73a2d
Add join with aggr case
zhuqi-lucas Jun 6, 2025
8d9f091
Merge remote-tracking branch 'upstream/main' into issue_16193
zhuqi-lucas Jun 6, 2025
b825ea6
fix clippy
zhuqi-lucas Jun 6, 2025
be88233
add reproducer for filter and add workaround in rule
zhuqi-lucas Jun 6, 2025
4018f8f
Add reproducer and comment it
zhuqi-lucas Jun 6, 2025
3b70846
adjust test
zhuqi-lucas Jun 6, 2025
11035e5
Harden tests, add failing tests for TDD, minor code refactors
ozankabak Jun 6, 2025
b05cb85
Add remaining test
zhuqi-lucas Jun 6, 2025
e136a03
Add sort merge case
zhuqi-lucas Jun 6, 2025
29bc2c7
change rule solution
zhuqi-lucas Jun 6, 2025
187a945
fix test
zhuqi-lucas Jun 6, 2025
3cc4095
add more built-in case
zhuqi-lucas Jun 6, 2025
4614048
fix
zhuqi-lucas Jun 6, 2025
0b8427c
Merge remote-tracking branch 'upstream/main' into issue_16193
zhuqi-lucas Jun 6, 2025
8047534
fix user defined exec
zhuqi-lucas Jun 6, 2025
7f4c93e
Reduce test diff
ozankabak Jun 6, 2025
ca7f061
Format imports
ozankabak Jun 6, 2025
5806bdb
Update documentation
ozankabak Jun 6, 2025
b45d3dc
Update information schema per docs
ozankabak Jun 6, 2025
a21c6ce
Only retain the with_cooperative_yields API for now
ozankabak Jun 6, 2025
c820e36
Format imports
ozankabak Jun 6, 2025
697bfae
Remove unnecessary clones
ozankabak Jun 6, 2025
884386e
Merge branch 'apache_main' into issue_16193
ozankabak Jun 7, 2025
d3952d9
Fix logical conflict
ozankabak Jun 7, 2025
56361a4
Exercise DRY, refactor common code to a helper function. Use period i…
ozankabak Jun 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions datafusion/common/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -722,6 +722,15 @@ config_namespace! {
/// then the output will be coerced to a non-view.
/// Coerces `Utf8View` to `LargeUtf8`, and `BinaryView` to `LargeBinary`.
pub expand_views_at_output: bool, default = false

/// When DataFusion detects that a plan might not be promply cancellable
/// due to the presence of tight-looping operators, it will attempt to
/// mitigate this by inserting explicit yielding (in as few places as
/// possible to avoid performance degradation). This value represents the
/// yielding period (in batches) at such explicit yielding points. The
/// default value is 64. If set to 0, no DataFusion will not perform
/// any explicit yielding.
pub yield_period: usize, default = 64
}
}

Expand Down
Loading