Skip to content

Commit f0235d7

Browse files
authored
Account for LinkedIn Spark versioning (#551)
### What changes are proposed in this pull request, and why are they necessary? In `TransportUDFTransformer` we have a function which decides the Scala version to use for a given Spark version. In LinkedIn we have changed the versioning scheme we use internally for Spark 3.5+, where the major version of the internal Spark distribution denotes the full OSS Spark version. E.g. For Spark 3.5.2, this would be 352.x.y (where x and y are internal versions). This PR accounts for this new versioning format. ### How was this patch tested? Existing unit tests. Tested the regex separately via regex101 <img width="231" alt="image" src="https://github.com/user-attachments/assets/779eb267-b839-4329-8925-a86900b7467f" />
1 parent f6ce2be commit f0235d7

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

coral-spark/src/main/java/com/linkedin/coral/spark/transformers/TransportUDFTransformer.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/**
2-
* Copyright 2018-2024 LinkedIn Corporation. All rights reserved.
2+
* Copyright 2018-2025 LinkedIn Corporation. All rights reserved.
33
* Licensed under the BSD-2 Clause license.
44
* See LICENSE in the project root for license information.
55
*/
@@ -88,7 +88,7 @@ public ScalaVersion getScalaVersionOfSpark() {
8888
String sparkVersion = SparkSession.active().version();
8989
if (sparkVersion.matches("2\\.[\\d\\.]*")) {
9090
return ScalaVersion.SCALA_2_11;
91-
} else if (sparkVersion.matches("3\\.[\\d\\.]*")) {
91+
} else if (sparkVersion.matches("3\\d*\\.[\\d\\.]*")) {
9292
return ScalaVersion.SCALA_2_12;
9393
} else {
9494
throw new IllegalStateException(String.format("Unsupported Spark Version %s", sparkVersion));

0 commit comments

Comments
 (0)