[BugFix] fix concat expr with multiple args nullsFraction #52683

stephen-shelby · 2024-11-06T14:37:57Z

Why I'm doing:

The current nullsFraction method for concat calculation is incorrect and may be negative. This will cause the cardinality of the join node to be 1.

  8:HASH JOIN
  |  join op: LEFT OUTER JOIN (BROADCAST)
  |  equal join conjunct: [44: xxx, LARGEINT, true] = [127: ccc, LARGEINT, true]
  |  other predicates: concat[(cast([9: CLAIM_STATUS_CD, BIGINT, true] as VARCHAR), ' - ', [126: LOOKUP_DETAIL, VARCHAR, true]); args: VARCHAR; result: VARCHAR; args nullable: true; result nullable: true] = '3 - CLEARED'
  |  output columns: 3, 8, 9, 43, 54, 126
  |  can local shuffle: false
  |  cardinality: 1
  |  column statistics: 
  |  * aaa-->[4.446678308E9, 1.8487815906E10, 0.0, 8.0, 1.0] ESTIMATE
  |  * vvv-->[-Infinity, Infinity, 0.0, 1.0000000994337923, 1.0] ESTIMATE
  |  * ccc-->[1.0, 42.0, 0.0, 8.0, 1.0] ESTIMATE
  |  * www-->[-1.3145022026298428E38, 1.2734555494779917E38, 0.0, 16.0, 1.0] ESTIMATE
  |  * qqqq-->[-1.5620653915393402E38, 1.2734555494779917E38, 0.0, 16.0, 1.0] ESTIMATE
  |  * rrr-->[-1.7007341574472433E38, 1.7003487503261273E38, 0.0, 16.0, 1.0] ESTIMATE
  |  * ccaa-->[-Infinity, Infinity, 0.0, 7.076923076923077, 1.0] ESTIMATE
  |  * aafff-->[-1.5620653915393402E38, 1.2734555494779917E38, 0.0, 16.0, 1.0] ESTIMATE
  |  
  |----7:EXCHANGE
  |       distribution type: BROADCAST
  |       cardinality: 13
  |    
  5:OlapScanNode
     table: table, rollup: table
     preAggregation: on
     partitionsRatio=1/1, tabletsRatio=814/814
     tabletList=30250908,30250912,30250916,30250920,30250924,30250928,30250932,30250936,30250940,30250944 ...
     actualRows=643644364, avgRowSize=65.0
     cardinality: 643644364
     column statistics: 
     * afffa-->[4.446678308E9, 1.8487815906E10, 0.0, 8.0, 6.4845024E8] ESTIMATE
     * dsdf-->[-Infinity, Infinity, 0.0, 1.0000000994337923, 7.0] ESTIMATE
     * ccc-->[1.0, 42.0, 0.0, 8.0, 10.0] ESTIMATE
     * fff-->[-1.3145022026298428E38, 1.2734555494779917E38, 0.0, 16.0, 7.0] ESTIMATE
     * aaaa-->[-1.5620653915393402E38, 1.2734555494779917E38, 0.0, 16.0, 10.0] ESTIMATE
     * wwww-->[-1.7007341574472433E38, 1.7003487503261273E38, 0.0, 16.0, 9105.0] ESTIMATE

What I'm doing:

Reference binaryExpressionCalculate nullsFraction algorithm is 1 - ((1 - left.getNullsFraction ()) * ( 1 - right.getNullsFraction ())). Fix multiaryExpressionCalculate

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

murphyatwork · 2024-11-07T02:34:49Z

...core/src/main/java/com/starrocks/sql/optimizer/statistics/ExpressionStatisticCalculator.java

-                    nullsFraction = 1 - childColumnStatisticList.stream().mapToDouble(ColumnStatistic::getAverageRowSize)
-                            .reduce(1, (a, b) -> (1 - a) * (1 - b));
+                    nullsFraction = 1 - childColumnStatisticList.stream().mapToDouble(ColumnStatistic::getNullsFraction)
+                            .reduce(1.0, (a, b) -> a * (1 - b));


why no (1-a) * (1-b) ?

if nullsFraction are [a, b, c], isn't our expectation 1 - ((1 - a) * (1 - b) * (1- c)) ?

let's rename the parameters ? not a/b

Signed-off-by: stephen <[email protected]>

sonarcloud · 2024-11-07T06:24:25Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-11-07T07:53:25Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-11-07T07:53:58Z

[FE Incremental Coverage Report]

✅ pass : 2 / 2 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/sql/optimizer/statistics/ExpressionStatisticCalculator.java	2	2	100.00%	[]

github-actions · 2024-11-07T07:54:12Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-11-07T08:13:36Z

@Mergifyio backport branch-3.3

github-actions · 2024-11-07T08:13:38Z

@Mergifyio backport branch-3.2

github-actions · 2024-11-07T08:13:39Z

@Mergifyio backport branch-3.1

mergify · 2024-11-07T08:14:01Z

backport branch-3.3

✅ Backports have been created

#52695 [BugFix] fix concat expr with multiple args nullsFraction (backport #52683) has been created for branch branch-3.3

mergify · 2024-11-07T08:14:02Z

backport branch-3.2

✅ Backports have been created

#52696 [BugFix] fix concat expr with multiple args nullsFraction (backport #52683) has been created for branch branch-3.2

mergify · 2024-11-07T08:14:04Z

backport branch-3.1

✅ Backports have been created

#52697 [BugFix] fix concat expr with multiple args nullsFraction (backport #52683) has been created for branch branch-3.1 but encountered conflicts

Signed-off-by: stephen <[email protected]> (cherry picked from commit 3b8e683)

Signed-off-by: stephen <[email protected]> (cherry picked from commit 3b8e683) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/ExpressionStatisticCalculator.java

Signed-off-by: stephen <[email protected]> (cherry picked from commit 3b8e683)

…52683) (#52696) Co-authored-by: stephen <[email protected]>

…52683) (#52695) Co-authored-by: stephen <[email protected]>

…52683) Signed-off-by: stephen <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>

stephen-shelby requested a review from a team as a code owner November 6, 2024 14:37

mergify bot assigned stephen-shelby Nov 6, 2024

stephen-shelby force-pushed the fix_concat_nullsFraction branch from a37a84c to b74ec4d Compare November 6, 2024 15:20

stephen-shelby changed the title ~~[BugFix] fix concat nullsFractionc~~ [BugFix] fix concat expr with multiple args nullsFraction Nov 7, 2024

github-actions bot added 3.3 3.2 3.1 labels Nov 7, 2024

murphyatwork reviewed Nov 7, 2024

View reviewed changes

[BugFix] fix concat nullsFractionc

1ee869e

Signed-off-by: stephen <[email protected]>

stephen-shelby force-pushed the fix_concat_nullsFraction branch from b74ec4d to 1ee869e Compare November 7, 2024 06:16

Seaven approved these changes Nov 7, 2024

View reviewed changes

Seaven enabled auto-merge (squash) November 7, 2024 07:56

murphyatwork approved these changes Nov 7, 2024

View reviewed changes

Seaven merged commit 3b8e683 into StarRocks:main Nov 7, 2024
50 checks passed

github-actions bot removed the 3.3 label Nov 7, 2024

github-actions bot removed the 3.2 label Nov 7, 2024

github-actions bot removed the 3.1 label Nov 7, 2024

mergify bot pushed a commit that referenced this pull request Nov 7, 2024

[BugFix] fix concat expr with multiple args nullsFraction (#52683)

75ff1e2

Signed-off-by: stephen <[email protected]> (cherry picked from commit 3b8e683)

mergify bot pushed a commit that referenced this pull request Nov 7, 2024

[BugFix] fix concat expr with multiple args nullsFraction (#52683)

f70965b

Signed-off-by: stephen <[email protected]> (cherry picked from commit 3b8e683)

wanpengfei-git pushed a commit that referenced this pull request Nov 7, 2024

[BugFix] fix concat expr with multiple args nullsFraction (backport #…

9b52771

…52683) (#52696) Co-authored-by: stephen <[email protected]>

github-actions bot added 3.2-merged 3.3 3.2 labels Nov 7, 2024

wanpengfei-git pushed a commit that referenced this pull request Nov 7, 2024

[BugFix] fix concat expr with multiple args nullsFraction (backport #…

76af279

…52683) (#52695) Co-authored-by: stephen <[email protected]>

github-actions bot added the 3.3-merged label Nov 7, 2024

renzhimin7 pushed a commit to renzhimin7/starrocks that referenced this pull request Nov 7, 2024

[BugFix] fix concat expr with multiple args nullsFraction (StarRocks#…

5bde6f1

…52683) Signed-off-by: stephen <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] fix concat expr with multiple args nullsFraction #52683

[BugFix] fix concat expr with multiple args nullsFraction #52683

stephen-shelby commented Nov 6, 2024 •

edited

Loading

murphyatwork Nov 7, 2024

stephen-shelby Nov 7, 2024

Seaven Nov 7, 2024 •

edited

Loading

stephen-shelby Nov 7, 2024

sonarcloud bot commented Nov 7, 2024

github-actions bot commented Nov 7, 2024

github-actions bot commented Nov 7, 2024

github-actions bot commented Nov 7, 2024

github-actions bot commented Nov 7, 2024

github-actions bot commented Nov 7, 2024

github-actions bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024 •

edited

Loading

mergify bot commented Nov 7, 2024 •

edited

Loading

mergify bot commented Nov 7, 2024 •

edited

Loading

[BugFix] fix concat expr with multiple args nullsFraction #52683

[BugFix] fix concat expr with multiple args nullsFraction #52683

Conversation

stephen-shelby commented Nov 6, 2024 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

murphyatwork Nov 7, 2024

Choose a reason for hiding this comment

stephen-shelby Nov 7, 2024

Choose a reason for hiding this comment

Seaven Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

stephen-shelby Nov 7, 2024

Choose a reason for hiding this comment

sonarcloud bot commented Nov 7, 2024

Quality Gate passed

github-actions bot commented Nov 7, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Nov 7, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Nov 7, 2024

[BE Incremental Coverage Report]

github-actions bot commented Nov 7, 2024

github-actions bot commented Nov 7, 2024

github-actions bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Nov 7, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Nov 7, 2024 • edited Loading

✅ Backports have been created

stephen-shelby commented Nov 6, 2024 •

edited

Loading

Seaven Nov 7, 2024 •

edited

Loading

mergify bot commented Nov 7, 2024 •

edited

Loading

mergify bot commented Nov 7, 2024 •

edited

Loading

mergify bot commented Nov 7, 2024 •

edited

Loading