Presto on Spark3.4.1

[wangkewen]

Description

This initial PR prepares the release of Presto on Spark 3.4.1 by incorporating updates from this pull request by @shrinidhijoshi.

Prior to the stable release of Presto on Spark 3 (e.g., Spark 3.4.1), code compatibility is maintained for both Spark 2.4 (the current version) and Spark 3.4. To support this, a Maven profile named sapphire has been created to facilitate building Presto on Spark 3.

Motivation and Context

It is the 1st step towards upgrading pos to support Spark3 and beyond.

Impact

Test Plan

spark 2.4, jdk 1.8

$ mvn clean install -U -DskipTests -Dmaven.javadoc.skip=true -T1C -pl '!presto-docs'

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for presto-root 0.294-SNAPSHOT:
[INFO] 
[INFO] presto-root ........................................ SUCCESS [  3.513 s]
[INFO] presto-testng-services ............................. SUCCESS [  4.811 s]
[INFO] presto-common ...................................... SUCCESS [  9.124 s]
[INFO] presto-spi ......................................... SUCCESS [  4.337 s]
[INFO] presto-plugin-toolkit .............................. SUCCESS [  2.989 s]
[INFO] presto-client ...................................... SUCCESS [  3.919 s]
[INFO] presto-parser ...................................... SUCCESS [  8.101 s]
[INFO] presto-geospatial-toolkit .......................... SUCCESS [  5.883 s]
[INFO] presto-analyzer .................................... SUCCESS [  1.676 s]
[INFO] presto-bytecode .................................... SUCCESS [  3.319 s]
[INFO] presto-matching .................................... SUCCESS [  1.736 s]
[INFO] presto-memory-context .............................. SUCCESS [  1.554 s]
[INFO] presto-expressions ................................. SUCCESS [  2.902 s]
[INFO] presto-function-namespace-managers-common .......... SUCCESS [  3.061 s]
[INFO] presto-tpch ........................................ SUCCESS [  3.836 s]
[INFO] presto-thrift-api .................................. SUCCESS [  3.538 s]
[INFO] presto-function-namespace-managers ................. SUCCESS [  4.559 s]
[INFO] presto-ui .......................................... SUCCESS [ 27.520 s]
[INFO] presto-main-base ................................... SUCCESS [ 54.397 s]
[INFO] presto-main ........................................ SUCCESS [  8.002 s]
[INFO] presto-blackhole ................................... SUCCESS [  3.249 s]
[INFO] presto-resource-group-managers ..................... SUCCESS [  4.818 s]
[INFO] presto-thrift-testing-udf-server ................... SUCCESS [  3.601 s]
[INFO] presto-tests ....................................... SUCCESS [  8.752 s]
[INFO] presto-atop ........................................ SUCCESS [  7.490 s]
[INFO] presto-hdfs-core ................................... SUCCESS [  4.698 s]
[INFO] presto-cache ....................................... SUCCESS [  4.787 s]
[INFO] presto-jmx ......................................... SUCCESS [  7.878 s]
[INFO] presto-record-decoder .............................. SUCCESS [  3.462 s]
[INFO] presto-kafka ....................................... SUCCESS [ 11.605 s]
[INFO] presto-redis ....................................... SUCCESS [  8.202 s]
[INFO] presto-accumulo .................................... SUCCESS [ 12.901 s]
[INFO] presto-cassandra ................................... SUCCESS [  9.211 s]
[INFO] presto-bigquery .................................... SUCCESS [ 11.019 s]
[INFO] presto-memory ...................................... SUCCESS [  8.979 s]
[INFO] presto-orc ......................................... SUCCESS [ 11.634 s]
[INFO] presto-benchmark ................................... SUCCESS [  3.707 s]
[INFO] presto-parquet ..................................... SUCCESS [  5.982 s]
[INFO] presto-rcfile ...................................... SUCCESS [  5.049 s]
[INFO] presto-hive-common ................................. SUCCESS [  2.392 s]
[INFO] presto-hive-metastore .............................. SUCCESS [  9.912 s]
[INFO] presto-tpcds ....................................... SUCCESS [  7.124 s]
[INFO] presto-testing-docker .............................. SUCCESS [  6.647 s]
[INFO] presto-hive ........................................ SUCCESS [ 18.249 s]
[INFO] presto-hive-hadoop2 ................................ SUCCESS [ 18.345 s]
[INFO] presto-iceberg ..................................... SUCCESS [ 24.956 s]
[INFO] presto-i18n-functions .............................. SUCCESS [  6.289 s]
[INFO] presto-teradata-functions .......................... SUCCESS [  6.698 s]
[INFO] presto-example-http ................................ SUCCESS [  3.752 s]
[INFO] presto-local-file .................................. SUCCESS [  3.751 s]
[INFO] presto-base-jdbc ................................... SUCCESS [  7.426 s]
[INFO] presto-mysql ....................................... SUCCESS [  5.473 s]
[INFO] presto-oracle ...................................... SUCCESS [  4.362 s]
[INFO] presto-postgresql .................................. SUCCESS [  4.274 s]
[INFO] presto-prometheus .................................. SUCCESS [  9.287 s]
[INFO] presto-redshift .................................... SUCCESS [  3.399 s]
[INFO] presto-sqlserver ................................... SUCCESS [  3.358 s]
[INFO] presto-mongodb ..................................... SUCCESS [  7.851 s]
[INFO] presto-ml .......................................... SUCCESS [  9.011 s]
[INFO] presto-jdbc ........................................ SUCCESS [ 39.751 s]
[INFO] presto-cli ......................................... SUCCESS [  4.374 s]
[INFO] presto-product-tests ............................... SUCCESS [ 24.970 s]
[INFO] presto-pinot-toolkit ............................... SUCCESS [  7.481 s]
[INFO] presto-pinot ....................................... SUCCESS [  7.859 s]
[INFO] presto-benchmark-driver ............................ SUCCESS [  5.519 s]
[INFO] presto-thrift-testing-server ....................... SUCCESS [ 15.215 s]
[INFO] presto-thrift-connector ............................ SUCCESS [  2.707 s]
[INFO] presto-password-authenticators ..................... SUCCESS [  3.892 s]
[INFO] presto-hudi ........................................ SUCCESS [ 18.835 s]
[INFO] presto-hive-function-namespace ..................... SUCCESS [  7.075 s]
[INFO] presto-hana ........................................ SUCCESS [  4.072 s]
[INFO] presto-druid ....................................... SUCCESS [ 15.709 s]
[INFO] presto-cluster-ttl-providers ....................... SUCCESS [  3.321 s]
[INFO] presto-singlestore ................................. SUCCESS [  4.100 s]
[INFO] presto-node-ttl-fetchers ........................... SUCCESS [  3.070 s]
[INFO] presto-session-property-managers ................... SUCCESS [  3.517 s]
[INFO] presto-kudu ........................................ SUCCESS [ 11.637 s]
[INFO] presto-elasticsearch ............................... SUCCESS [ 12.080 s]
[INFO] presto-spark-classloader-interface ................. SUCCESS [  6.354 s]
[INFO] presto-spark-classloader-spark2 .................... SUCCESS [  1.775 s]
[INFO] presto-spark-common ................................ SUCCESS [  1.845 s]
[INFO] presto-spark-base .................................. SUCCESS [  8.708 s]
[INFO] presto-native-execution ............................ SUCCESS [ 11.886 s]
[INFO] presto-native-sidecar-plugin ....................... SUCCESS [  4.714 s]
[INFO] presto-delta ....................................... SUCCESS [ 17.574 s]
[INFO] presto-clickhouse .................................. SUCCESS [  9.301 s]
[INFO] presto-server ...................................... SUCCESS [ 38.718 s]
[INFO] presto-verifier .................................... SUCCESS [ 43.536 s]
[INFO] presto-testing-server-launcher ..................... SUCCESS [ 14.839 s]
[INFO] presto-benchto-benchmarks .......................... SUCCESS [  6.126 s]
[INFO] presto-proxy ....................................... SUCCESS [  7.140 s]
[INFO] presto-benchmark-runner ............................ SUCCESS [  6.578 s]
[INFO] presto-spark ....................................... SUCCESS [  2.700 s]
[INFO] presto-spark-package ............................... SUCCESS [ 41.257 s]
[INFO] presto-spark-launcher .............................. SUCCESS [ 14.328 s]
[INFO] presto-spark-testing ............................... SUCCESS [  4.735 s]
[INFO] presto-thrift-spec ................................. SUCCESS [  4.377 s]
[INFO] presto-google-sheets ............................... SUCCESS [  8.258 s]
[INFO] presto-grpc-api .................................... SUCCESS [  7.372 s]
[INFO] presto-grpc-testing-udf-server ..................... SUCCESS [  1.628 s]
[INFO] presto-lark-sheets ................................. SUCCESS [  8.144 s]
[INFO] presto-test-coverage ............................... SUCCESS [  9.733 s]
[INFO] presto-plan-checker-router-plugin .................. SUCCESS [  3.523 s]
[INFO] presto-router ...................................... SUCCESS [  9.929 s]
[INFO] presto-native-tests ................................ SUCCESS [  2.579 s]
[INFO] presto-open-telemetry .............................. SUCCESS [  2.968 s]
[INFO] redis-hbo-provider ................................. SUCCESS [  8.650 s]
[INFO] presto-openapi ..................................... SUCCESS [  7.202 s]
[INFO] presto-base-arrow-flight ........................... SUCCESS [ 11.390 s]
[INFO] presto-function-namespace-managers ................. SUCCESS [ 16.910 s]
[INFO] presto-router-example-plugin-scheduler ............. SUCCESS [  2.889 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:24 min (Wall Clock)
[INFO] Finished at: 2025-07-25T12:00:36-07:00
[INFO] ------------------------------------------------------------------------

spark 3.4, jdk 11

$ mvn clean install -U -DskipTests -Dmaven.javadoc.skip=true -Dupgrade=true -T1C -pl '!presto-docs'

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for presto-root 0.294-SNAPSHOT:
[INFO] 
[INFO] presto-root ........................................ SUCCESS [  3.473 s]
[INFO] presto-testng-services ............................. SUCCESS [  4.206 s]
[INFO] presto-common ...................................... SUCCESS [  7.541 s]
[INFO] presto-spi ......................................... SUCCESS [  3.720 s]
[INFO] presto-plugin-toolkit .............................. SUCCESS [  2.675 s]
[INFO] presto-client ...................................... SUCCESS [  3.248 s]
[INFO] presto-parser ...................................... SUCCESS [  8.386 s]
[INFO] presto-geospatial-toolkit .......................... SUCCESS [  5.323 s]
[INFO] presto-analyzer .................................... SUCCESS [  1.309 s]
[INFO] presto-bytecode .................................... SUCCESS [  3.303 s]
[INFO] presto-matching .................................... SUCCESS [  1.559 s]
[INFO] presto-memory-context .............................. SUCCESS [  1.349 s]
[INFO] presto-expressions ................................. SUCCESS [  2.251 s]
[INFO] presto-function-namespace-managers-common .......... SUCCESS [  2.309 s]
[INFO] presto-tpch ........................................ SUCCESS [  3.806 s]
[INFO] presto-thrift-api .................................. SUCCESS [  3.471 s]
[INFO] presto-function-namespace-managers ................. SUCCESS [  3.450 s]
[INFO] presto-ui .......................................... SUCCESS [ 26.648 s]
[INFO] presto-main-base ................................... SUCCESS [01:03 min]
[INFO] presto-main ........................................ SUCCESS [  4.302 s]
[INFO] presto-blackhole ................................... SUCCESS [  1.859 s]
[INFO] presto-resource-group-managers ..................... SUCCESS [  2.777 s]
[INFO] presto-thrift-testing-udf-server ................... SUCCESS [  3.305 s]
[INFO] presto-tests ....................................... SUCCESS [  6.834 s]
[INFO] presto-atop ........................................ SUCCESS [  4.329 s]
[INFO] presto-hdfs-core ................................... SUCCESS [  2.351 s]
[INFO] presto-cache ....................................... SUCCESS [  2.782 s]
[INFO] presto-jmx ......................................... SUCCESS [  3.984 s]
[INFO] presto-record-decoder .............................. SUCCESS [  1.878 s]
[INFO] presto-kafka ....................................... SUCCESS [  5.372 s]
[INFO] presto-redis ....................................... SUCCESS [  4.851 s]
[INFO] presto-accumulo .................................... SUCCESS [  7.180 s]
[INFO] presto-cassandra ................................... SUCCESS [  5.372 s]
[INFO] presto-bigquery .................................... SUCCESS [  5.021 s]
[INFO] presto-memory ...................................... SUCCESS [  4.893 s]
[INFO] presto-orc ......................................... SUCCESS [  9.719 s]
[INFO] presto-benchmark ................................... SUCCESS [  2.166 s]
[INFO] presto-parquet ..................................... SUCCESS [  4.340 s]
[INFO] presto-rcfile ...................................... SUCCESS [  2.440 s]
[INFO] presto-hive-common ................................. SUCCESS [  1.277 s]
[INFO] presto-hive-metastore .............................. SUCCESS [  6.267 s]
[INFO] presto-tpcds ....................................... SUCCESS [  3.018 s]
[INFO] presto-testing-docker .............................. SUCCESS [  4.195 s]
[INFO] presto-hive ........................................ SUCCESS [ 15.415 s]
[INFO] presto-hive-hadoop2 ................................ SUCCESS [ 11.237 s]
[INFO] presto-iceberg ..................................... SUCCESS [ 17.194 s]
[INFO] presto-i18n-functions .............................. SUCCESS [  3.775 s]
[INFO] presto-teradata-functions .......................... SUCCESS [  3.888 s]
[INFO] presto-example-http ................................ SUCCESS [  2.201 s]
[INFO] presto-local-file .................................. SUCCESS [  2.136 s]
[INFO] presto-base-jdbc ................................... SUCCESS [  4.435 s]
[INFO] presto-mysql ....................................... SUCCESS [  3.361 s]
[INFO] presto-oracle ...................................... SUCCESS [  2.486 s]
[INFO] presto-postgresql .................................. SUCCESS [  2.331 s]
[INFO] presto-prometheus .................................. SUCCESS [  5.081 s]
[INFO] presto-redshift .................................... SUCCESS [  2.020 s]
[INFO] presto-sqlserver ................................... SUCCESS [  1.871 s]
[INFO] presto-mongodb ..................................... SUCCESS [  3.883 s]
[INFO] presto-ml .......................................... SUCCESS [  4.420 s]
[INFO] presto-jdbc ........................................ SUCCESS [ 37.518 s]
[INFO] presto-cli ......................................... SUCCESS [  3.225 s]
[INFO] presto-product-tests ............................... SUCCESS [ 19.314 s]
[INFO] presto-pinot-toolkit ............................... SUCCESS [  3.567 s]
[INFO] presto-pinot ....................................... SUCCESS [  5.391 s]
[INFO] presto-benchmark-driver ............................ SUCCESS [  5.055 s]
[INFO] presto-thrift-testing-server ....................... SUCCESS [  9.857 s]
[INFO] presto-thrift-connector ............................ SUCCESS [  1.893 s]
[INFO] presto-password-authenticators ..................... SUCCESS [  3.242 s]
[INFO] presto-hudi ........................................ SUCCESS [ 11.740 s]
[INFO] presto-hive-function-namespace ..................... SUCCESS [  5.061 s]
[INFO] presto-hana ........................................ SUCCESS [  2.226 s]
[INFO] presto-druid ....................................... SUCCESS [  9.089 s]
[INFO] presto-cluster-ttl-providers ....................... SUCCESS [  2.785 s]
[INFO] presto-singlestore ................................. SUCCESS [  2.238 s]
[INFO] presto-node-ttl-fetchers ........................... SUCCESS [  2.325 s]
[INFO] presto-session-property-managers ................... SUCCESS [  2.933 s]
[INFO] presto-kudu ........................................ SUCCESS [  5.626 s]
[INFO] presto-elasticsearch ............................... SUCCESS [  6.085 s]
[INFO] presto-spark-classloader-interface ................. SUCCESS [ 21.275 s]
[INFO] presto-spark-classloader-spark3 .................... SUCCESS [ 14.919 s]
[INFO] presto-spark-common ................................ SUCCESS [  1.419 s]
[INFO] presto-spark-base .................................. SUCCESS [ 32.085 s]
[INFO] presto-native-execution ............................ SUCCESS [ 18.196 s]
[INFO] presto-native-sidecar-plugin ....................... SUCCESS [  2.580 s]
[INFO] presto-delta ....................................... SUCCESS [ 11.236 s]
[INFO] presto-clickhouse .................................. SUCCESS [  5.044 s]
[INFO] presto-server ...................................... SUCCESS [ 37.512 s]
[INFO] presto-verifier .................................... SUCCESS [ 34.189 s]
[INFO] presto-testing-server-launcher ..................... SUCCESS [  9.405 s]
[INFO] presto-benchto-benchmarks .......................... SUCCESS [  2.304 s]
[INFO] presto-proxy ....................................... SUCCESS [  3.779 s]
[INFO] presto-benchmark-runner ............................ SUCCESS [  2.135 s]
[INFO] presto-spark ....................................... SUCCESS [  0.793 s]
[INFO] presto-spark-package ............................... SUCCESS [ 41.439 s]
[INFO] presto-spark-launcher .............................. SUCCESS [ 23.426 s]
[INFO] presto-spark-testing ............................... SUCCESS [  1.335 s]
[INFO] presto-thrift-spec ................................. SUCCESS [  1.798 s]
[INFO] presto-google-sheets ............................... SUCCESS [  4.425 s]
[INFO] presto-grpc-api .................................... SUCCESS [  6.040 s]
[INFO] presto-grpc-testing-udf-server ..................... SUCCESS [  0.953 s]
[INFO] presto-lark-sheets ................................. SUCCESS [  3.669 s]
[INFO] presto-test-coverage ............................... SUCCESS [ 11.013 s]
[INFO] presto-plan-checker-router-plugin .................. SUCCESS [  2.159 s]
[INFO] presto-router ...................................... SUCCESS [  6.000 s]
[INFO] presto-native-tests ................................ SUCCESS [  1.256 s]
[INFO] presto-open-telemetry .............................. SUCCESS [  2.269 s]
[INFO] redis-hbo-provider ................................. SUCCESS [  3.877 s]
[INFO] presto-openapi ..................................... SUCCESS [  6.379 s]
[INFO] presto-base-arrow-flight ........................... SUCCESS [  4.262 s]
[INFO] presto-function-namespace-managers ................. SUCCESS [ 10.939 s]
[INFO] presto-router-example-plugin-scheduler ............. SUCCESS [  1.971 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:11 min (Wall Clock)
[INFO] Finished at: 2025-07-25T12:07:59-07:00
[INFO] ------------------------------------------------------------------------

Contributor checklist

• Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== NO RELEASE NOTE ==

[linux-foundation-easycla]

• ❌ The email address for the commit (d295f87, a81866b, 7a73f47, b15eb76, aa1370d) is not linked to the GitHub account, preventing the EasyCLA check. Consult this Help Article and GitHub Help to resolve. (To view the commit's email address, add .patch at the end of this PR page's URL.) For further assistance with EasyCLA, please submit a support request ticket.

[unidevel]

@wangkewen Thansk for your contribution, can you sign the EasyCLA and resolve the conflicts first? Also since this is a new feature, it is better to provide release notes and documentation about setup and configurations.