[SPARK-55975][SQL][TESTS] NaN comparison can cause false UT failures due to different NaNs

### What changes were proposed in this pull request?

In the UTs, currently NaN comparisons can cause UT failures on the CI (which is also hardware dependent) as on some hardware NaN can be represented in different byte formats (see IEEE-754 that allows multiple bit patterns for NaN). This has caused spurious UT failures e.g. in #54676.

### Why are the changes needed?

Fix NaN comparison so that we do not run into hardware-dependent UT failures and correctly passes when there are two NaNs.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Added UT.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #54772 from mgaido91/SPARK-55975.

Authored-by: Marco Gaido <mgaido@fbk.eu>
Signed-off-by: Peter Toth <peter.toth@gmail.com>
(cherry picked from commit e7bbd32)
Signed-off-by: Peter Toth <peter.toth@gmail.com>

Author: mgaido91

sql/core/src/test/scala/org/apache/spark/sql/MathFunctionsSuite.scala

@@ -117,6 +117,15 @@ class MathFunctionsSuite extends QueryTest with SharedSparkSession {
     )
   }
 
+  test("different NaN comparison") {
+    assert(QueryTest.compare(
+      java.lang.Double.longBitsToDouble(0x7ff8000000000000L),
+      java.lang.Double.longBitsToDouble(0xfff8000000000000L)))
+    assert(QueryTest.compare(
+      java.lang.Float.intBitsToFloat(0x7fc00000),
+      java.lang.Float.intBitsToFloat(0xffc00000)))
+  }
+
   test("sin") {
     testOneToOneMathFunction(sin, math.sin)
   }

sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala

@@ -405,9 +405,12 @@ object QueryTest extends Assertions {
     case (a: Row, b: Row) =>
       compare(a.toSeq, b.toSeq)
     // 0.0 == -0.0, turn float/double to bits before comparison, to distinguish 0.0 and -0.0.
+    // in some hardware NaN can be represented with different bits, so first check for it
     case (a: Double, b: Double) =>
+      a.isNaN && b.isNaN ||
       java.lang.Double.doubleToRawLongBits(a) == java.lang.Double.doubleToRawLongBits(b)
     case (a: Float, b: Float) =>
+      a.isNaN && b.isNaN ||
       java.lang.Float.floatToRawIntBits(a) == java.lang.Float.floatToRawIntBits(b)
     case (a, b) => a == b
   }