Skip to content

feat: Support ANSI mode for round #466

@andygrove

Description

@andygrove

What is the problem the feature request solves?

Comet does not support ANSI mode for round.

Create test data

val df = Seq(Int.MaxValue, Int.MinValue).toDF("a")
df.write.parquet("/tmp/int.parquet")
spark.read.parquet("/tmp/int.parquet").createTempView("t")

Test with ANSI enabled

Comet does not throw an exception but Spark does.

scala> spark.conf.set("spark.comet.ansi.enabled", true)

scala> spark.conf.set("spark.sql.ansi.enabled", true)

scala> spark.conf.set("spark.comet.enabled", true)

scala> spark.sql("select a, round(a,-1) from t").show
24/05/23 14:10:30 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 14:10:30 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because CollectLimit is not supported
+-----------+------------+
|          a|round(a, -1)|
+-----------+------------+
| 2147483647| -2147483646|
|-2147483648|  2147483646|
+-----------+------------+


scala> spark.conf.set("spark.comet.enabled", false)

scala> spark.sql("select a, round(a,-1) from t").show
24/05/23 14:10:37 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 14:10:38 ERROR Executor: Exception in task 0.0 in stage 15.0 (TID 15)
org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] Overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 11) ==
select a, round(a,-1) from t
          ^^^^^^^^^^^

Describe the potential solution

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions