[WIP] Use "selectivity-hints" for query planning

[rjzamora]

Description

• Changes GroupByandUnique` behavior to preserve partition count by default
  • Hints or statistics are needed to reduce the partition count
• Adds a new "selectivity-hints" configuration option
  • Users can specify a mapping between specific operations (by _ir_repr label) and their "selectivity".
  • Used to reduce the output partition count for labeled GoupBy and Union
  • Used to introduce a Repartition after other labeled nodes

Notes:

• New pdsh hints were informed by "profiling" the row count processed by each IR node in an sf300 run

  • "Profiling" feature proposed in [WIP] Use "selectivity-hints" for query planning #21054

  Performance:
  There are probably still opportunites for improvement, but this branch is giving good 8xH100 sf3000 numbers for TPC-H (especially q9!):

  Iteration Summary
=======================================
...
=======================================
query: 9
path: /raid/rapidsmpf/data/tpch/scale-3000
scale_factor: 3000
executor: streaming
stream_policy: None
runtime: rapidsmpf
cluster: distributed
blocksize: 2000000000
shuffle_method: None
broadcast_join_limit: 3
stats_planning: False
native_parquet: False
n_workers: 8
threads: 1
rmm_async: True
rapidsmpf_oom_protection: False
spill_device: 0.5
rapidsmpf_spill: False
iterations: 2
---------------------------------------
min time : 6.4565
max time : 6.5054
mean time: 6.4809
=======================================
...
=======================================
Total mean time across all queries: 189.6582 seconds

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[rjzamora]

/ok to test

[rjzamora]

/ok to test

[rjzamora]

/ok to test

[rjzamora]

/ok to test