Skip to content

[BUG] Get ANSI mode from expression, not config #13144

@revans2

Description

@revans2

Describe the bug
Currently we have a lot of code like

private val ansiEnabled = SQLConf.get.ansiEnabled
...
GpuAdd(lhs, rhs, failOnError = ansiEnabled)

But this is not correct. In 3.2.1, there is a failOnError boolean, that could be set to true or false outside of actual ANSI mode.

https://github.com/apache/spark/blob/v3.2.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala

In 3.4.0 this was updated to have an EvalMode, which again could be set ignoring the configuration value.

https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala#L419-L423

If we are going to do this correctly we need to check for ANSI mode looking at individual expressions and not the config that is set. I know this is much more difficult to do because the code will not be generic, but the code is technically wrong if we don't do this.

Metadata

Metadata

Assignees

Labels

Spark 4.0+Spark 4.0+ issuesbugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions