[SPARK-51046][SQL][TEST] Reduce numCols in withFilter() to prevent SubExprEliminationBenchmark from failing due to a Codegen error

[wayneguow]

What changes were proposed in this pull request?

This PR aims to reduce numCols in withFilter() to prevent SubExprEliminationBenchmark from failing due to a Codegen error.

I did some debug investigation and found that in the current master branch, the codegen code has a huge processNext method, but in branch-3.5, it is split into many small methods. Because the logic of codegen is different, the previous benchmark cannot run normally.

master branch:

protected void processNext() throws java.io.IOException {

	while ( inputadapter_input_0.hasNext()) {
                 ...
        }
}

3.5 branch:

        public boolean eval(InternalRow i) {
		boolean value_2997 = Or_498(i);
		return !globalIsNull_498 && value_2997;
	}
        private boolean Or_498(InternalRow i) {
                ...
        }
        ...
        private boolean Or_xxxx(InternalRow i) {
               ...
        }    

Why are the changes needed?

If we run SubExprEliminationBenchmark:

build/sbt "sql/Test/runMain org.apache.spark.sql.execution.SubExprEliminationBenchmark"

It fails at CodeGenerator, details:

[info] Running benchmark: from_json as subExpr in Filter
[info] Running case: subExprElimination false, codegen: true
[info] 00:08:31.509 ERROR org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: Failed to compile the generated Java code.
[info] org.codehaus.commons.compiler.InternalCompilerException: Compiling "GeneratedClass" in File 'generated.java', Line 1, Column 1: File 'generated.java', Line 24, Column 16: Compiling "processNext()"

The root cause is:

[error] Caused by: org.codehaus.commons.compiler.InternalCompilerException: Code grows beyond 64 KB
[error] at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:699)
[error] at org.codehaus.janino.CodeContext.write(CodeContext.java:558)
[error] at org.codehaus.janino.UnitCompiler.write(UnitCompiler.java:13079)
[error] at org.codehaus.janino.UnitCompiler.store(UnitCompiler.java:12752)
[error] at org.codehaus.janino.UnitCompiler.store(UnitCompiler.java:12730)
[error] at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2742)
[error] ... 164 more

Does this PR introduce any user-facing change?

No, just fix a test benchmark failure.

How was this patch tested?

Run benchmark tests manually.

Was this patch authored or co-authored using generative AI tooling?

No.

[wayneguow]

Benchmark jdk17: https://github.com/wayneguow/spark/actions/runs/13310975547
Benchmark jdk21: https://github.com/wayneguow/spark/actions/runs/13310979208

[wayneguow]

What confuses me is that when subExprElimination set to true, codegen set to true, there is an obvious regression in performance compared to before?

[dongjoon-hyun]

Yes, it's true. The performance regression was the actual root cause why we didn't make a decision. Here, FYI.

• [SPARK-50767][SQL] Remove codegen of from_json #49411

[wayneguow]

cc @dongjoon-hyun , Could you take a look when you have some time? maybe we can have some discussion.

[dongjoon-hyun]

cc @panbingkun , @cloud-fan , @LuciferYang

It seems that we need to make a decision. Are we good with this codeine perf regression of from_json?

[dongjoon-hyun]

BTW, Thank you for your active contributions, @wayneguow !

[wayneguow]

BTW, Thank you for your active contributions, @wayneguow !

Happy to contribute！😀 Keep going！

[LuciferYang]

cc @panbingkun , @cloud-fan , @LuciferYang

It seems that we need to make a decision. Are we good with this codeine perf regression of from_json?

@panbingkun Can #49573 be completed before the 4.0 release? If so, we can wait until the optimization is finished before refreshing this result. Additionally, if #49573 is completed, will we still need to change numCols from 500 to 330?