-
Notifications
You must be signed in to change notification settings - Fork 246
Open
Labels
enhancementNew feature or requestNew feature or requestgood first issueGood for newcomersGood for newcomershelp wantedExtra attention is neededExtra attention is needed
Description
What is the problem the feature request solves?
Comet does not support ANSI mode for round
.
Create test data
val df = Seq(Int.MaxValue, Int.MinValue).toDF("a")
df.write.parquet("/tmp/int.parquet")
spark.read.parquet("/tmp/int.parquet").createTempView("t")
Test with ANSI enabled
Comet does not throw an exception but Spark does.
scala> spark.conf.set("spark.comet.ansi.enabled", true)
scala> spark.conf.set("spark.sql.ansi.enabled", true)
scala> spark.conf.set("spark.comet.enabled", true)
scala> spark.sql("select a, round(a,-1) from t").show
24/05/23 14:10:30 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 14:10:30 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because CollectLimit is not supported
+-----------+------------+
| a|round(a, -1)|
+-----------+------------+
| 2147483647| -2147483646|
|-2147483648| 2147483646|
+-----------+------------+
scala> spark.conf.set("spark.comet.enabled", false)
scala> spark.sql("select a, round(a,-1) from t").show
24/05/23 14:10:37 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 14:10:38 ERROR Executor: Exception in task 0.0 in stage 15.0 (TID 15)
org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] Overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 11) ==
select a, round(a,-1) from t
^^^^^^^^^^^
Describe the potential solution
No response
Additional context
No response
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestgood first issueGood for newcomersGood for newcomershelp wantedExtra attention is neededExtra attention is needed