Velox, GCS and cURL | CURL error [77]=Problem with the SSL CA cert (path? access rights?) #9946

A-Mongy · 2025-06-11T20:25:07Z

A-Mongy
Jun 11, 2025

I am trying to test Gluten with Velox backend with spark 3.4.1

I am using the jar from https://downloads.apache.org/incubator/gluten/1.3.0-incubating/ for spark34

GCS configs are correctly set

spark.hadoop.fs.gs.auth.type                         SERVICE_ACCOUNT_JSON_KEYFILE
spark.hadoop.fs.gs.auth.service.account.json.keyfile // path to the json file with the credentials.

curl seems to be working right, I tried to update the ca certificates also, so I am not sure what is the issue here.

I get the following error in the executor logs

25/06/11 19:54:40 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 84.5 KiB, free 1458.3 MiB)
25/06/11 19:54:40 INFO TorrentBroadcast: Reading broadcast variable 0 took 102 ms
25/06/11 19:54:40 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 232.1 KiB, free 1458.1 MiB)
E20250611 19:54:41.385499    92 Exceptions.h:66] Line: /root/src/weiting/gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp:62, Function:checkGcsStatus, Expression:  Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)', Source: RUNTIME, ErrorCode: INVALID_STATE
25/06/11 19:54:41 ERROR TaskResources: Task 0 failed by error: 
org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)'
Retriable: False
Context: Split [Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245] Task Gluten_Stage_0_TID_0_VTID_0
Additional Context: Operator: TableScan[0] 0
Function: checkGcsStatus
File: /root/src/weiting/gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp
Line: 62
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox12_GLOBAL__N_114checkGcsStatusEN6google5cloud5v2_226StatusERKSt17basic_string_viewIcSt11char_traitsIcEERKSsSD_
# 4  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile13preadInternalEmmPc
# 5  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile6preadvEmRKSt6vectorIN5folly5RangeIPcEESaIS7_EE
# 6  _ZN8facebook5velox4dwio6common19ReadFileInputStream4readERKSt6vectorIN5folly5RangeIPcEESaIS8_EEmNS2_10MetricsLog11MetricsTypeE
# 7  _ZN8facebook5velox4dwio6common19DirectCoalescedLoad8loadDataEb
# 8  _ZN8facebook5velox5cache13CoalescedLoad12loadOrFutureEPN5folly10SemiFutureIbEEb
# 9  _ZN8facebook5velox4dwio6common17DirectInputStream12loadPositionEv
# 10 _ZN8facebook5velox4dwio6common17DirectInputStream4NextEPPKvPi
# 11 _ZN8facebook5velox7parquet10ReaderBase16loadFileMetaDataEv
# 12 _ZN8facebook5velox7parquet10ReaderBaseC1ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 13 _ZN8facebook5velox7parquet13ParquetReaderC2ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 14 _ZN8facebook5velox7parquet20ParquetReaderFactory12createReaderESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 15 _ZN8facebook5velox9connector4hive11SplitReader12createReaderEv
# 16 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsE
# 17 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 18 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 19 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv
# 20 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 21 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 22 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 23 _ZN6gluten24WholeStageResultIterator4nextEv
# 24 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 25 0x00007e6f146b9930

        at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:41)
        at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
        at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
        at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
        at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
        at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
        at scala.collection.Iterator.isEmpty(Iterator.scala:387)
        at scala.collection.Iterator.isEmpty$(Iterator.scala:387)
        at org.apache.spark.InterruptibleIterator.isEmpty(InterruptibleIterator.scala:28)
        at org.apache.gluten.execution.VeloxColumnarToRowExec$.toRowIterator(VeloxColumnarToRowExec.scala:122)
        at org.apache.gluten.execution.VeloxColumnarToRowExec.$anonfun$doExecuteInternal$1(VeloxColumnarToRowExec.scala:78)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:853)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:853)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
        at org.apache.spark.scheduler.Task.run(Task.scala:139)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError

A-Mongy · 2025-06-16T15:35:13Z

A-Mongy
Jun 16, 2025
Author

Any help is appreciated here
@zhouyuan, @holdenk, @raboof, @EpsilonPrime

0 replies

zhouyuan · 2025-06-17T09:19:30Z

zhouyuan
Jun 17, 2025
Collaborator

@A-Mongy AFAIK the gcs connector should be working fine. The issue you posted seems related with ca-certs on runtime. I find some docs here (https://bobcares.com/blog/curl-error-77-problem-with-the-ssl-ca-cert/). Since you have already updated ca-certs, would you please do a quick test by using the nightly package?
https://nightlies.apache.org/gluten/

thanks, -yuan

1 reply

A-Mongy Jun 18, 2025
Author

Thanks @zhouyuan for getting back to me on this!
I have tried to change the permission for the etc/ssl/ folder

total 24
drwxr-xr-x 1 root root  4096 Apr 15 04:41 certs
-rwxr-xr-x 1 root root 12332 Oct 27  2024 openssl.cnf
drwxr-xr-x 1 root root  4096 Oct 27  2024 private

So shouldn't be a permission issue.
I have also tried the nightly jars, and I still see the same problem
Here is an updated log of the error, as I turned on debug mode:

20250618 00:30:14.379505    91 HiveConnectorUtil.cpp:286] No need to check type lowercase modeVARCHAR
I20250618 00:30:14.379514    91 HiveConnectorUtil.cpp:286] No need to check type lowercase modeVARCHAR
I20250618 00:30:14.379588    91 HiveDataSource.cpp:306] Adding split Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245
I20250618 00:30:14.384384    91 FileHandle.cpp:65] Generating file handle for: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet uuid: 1
I20250618 00:30:14.384630    96 EventBase.h:883] expCoeff_ -5e-07 folly::EventBase::SmoothLoopTime::SmoothLoopTime(std::chrono::microseconds)
I20250618 00:30:14.384725    96 EventBase.cpp:543] EventBase(): Starting loop.
I20250618 00:30:14.384757    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551576 next 18446744073709551577
I20250618 00:30:14.384773    96 EventBase.cpp:1058] latest 18446744073709551576 next 18446744073709551577
I20250618 00:30:14.384784    96 EventBase.cpp:801] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) startWork_ 29679038962635
I20250618 00:30:14.384800    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551577 next 18446744073709551577
I20250618 00:30:14.384812    96 EventBase.cpp:1058] latest 18446744073709551577 next 18446744073709551577
I20250618 00:30:14.384876    96 EventBase.cpp:1058] latest 18446744073709551577 next 18446744073709551577
I20250618 00:30:14.384821    96 EventBase.cpp:655] EventBase 0x7ee85c00a760 did not timeout  loop time guess: 76 idle time: 40 busy time: 36 avgLoopTime: 0.001368 maxLatencyLoopTime: 0.001368 maxLatency_: 0us notificationQueueSize: 1 nothingHandledYet(): 0
I20250618 00:30:14.384897    96 EventBase.cpp:686] EventBase 0x7ee85c00a760 loop time: 0
I20250618 00:30:14.384910    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551577 next 18446744073709551578
I20250618 00:30:14.384919    96 EventBase.cpp:1058] latest 18446744073709551577 next 18446744073709551578
I20250618 00:30:14.384930    96 EventBase.cpp:801] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) startWork_ 29679039108575
I20250618 00:30:14.384951    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551578 next 18446744073709551578
I20250618 00:30:14.384963    96 EventBase.cpp:1058] latest 18446744073709551578 next 18446744073709551578
I20250618 00:30:14.384982    96 EventBase.cpp:1058] latest 18446744073709551578 next 18446744073709551578
I20250618 00:30:14.384972    96 EventBase.cpp:655] EventBase 0x7ee85c00a760 did not timeout  loop time guess: 151 idle time: 109 busy time: 42 avgLoopTime: 0.008853 maxLatencyLoopTime: 0.008853 maxLatency_: 0us notificationQueueSize: 0 nothingHandledYet(): 0
I20250618 00:30:14.384994    96 EventBase.cpp:686] EventBase 0x7ee85c00a760 loop time: 0
E20250618 00:30:14.421082    91 Exceptions.h:66] Line: /work/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp:62, Function:checkGcsStatus, Expression:  Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)', Source: RUNTIME, ErrorCode: INVALID_STATE
I20250618 00:30:14.421433    91 Task.cpp:2402] Terminating task Gluten_Stage_0_TID_0_VTID_0 with state Failed after running for 43ms
I20250618 00:30:14.421552    91 Task.cpp:1343] All drivers (1) finished for task Gluten_Stage_0_TID_0_VTID_0 after running for 43ms
25/06/18 00:30:14 ERROR TaskResources: Task 0 failed by error: 
org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)'
Retriable: False
Context: Split [Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245] Task Gluten_Stage_0_TID_0_VTID_0
Additional Context: Operator: TableScan[0] 0
Function: checkGcsStatus
File: /work/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp
Line: 62
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox12_GLOBAL__N_114checkGcsStatusEN6google5cloud5v2_226StatusERKSt17basic_string_viewIcSt11char_traitsIcEERKSsSD_
# 4  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile13preadInternalEmmPc
# 5  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile6preadvEmRKSt6vectorIN5folly5RangeIPcEESaIS7_EEPNS0_11filesystems4File7IoStatsE
# 6  _ZN8facebook5velox4dwio6common19ReadFileInputStream4readERKSt6vectorIN5folly5RangeIPcEESaIS8_EEmNS2_10MetricsLog11MetricsTypeE
# 7  _ZN8facebook5velox4dwio6common19DirectCoalescedLoad8loadDataEb
# 8  _ZN8facebook5velox5cache13CoalescedLoad12loadOrFutureEPN5folly10SemiFutureIbEEb
# 9  _ZN8facebook5velox4dwio6common17DirectInputStream12loadPositionEv
# 10 _ZN8facebook5velox4dwio6common17DirectInputStream4NextEPPKvPi
# 11 _ZN8facebook5velox7parquet10ReaderBase16loadFileMetaDataEv
# 12 _ZN8facebook5velox7parquet10ReaderBaseC1ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 13 _ZN8facebook5velox7parquet13ParquetReaderC2ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 14 _ZN8facebook5velox7parquet20ParquetReaderFactory12createReaderESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 15 _ZN8facebook5velox9connector4hive11SplitReader12createReaderEv
# 16 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsE
# 17 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 18 _ZN8facebook5velox4exec9TableScan8getSplitEv
# 19 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 20 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv
# 21 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 22 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEERPNS1_8OperatorERNS1_14BlockingReasonE
# 23 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 24 _ZN6gluten24WholeStageResultIterator4nextEv
# 25 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 26 0x00007ee91c6b9930

You can notice the two lines where the gcs bucket is also mentioned are:

I20250618 00:30:14.379588    91 HiveDataSource.cpp:306] Adding split Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245
I20250618 00:30:14.384384    91 FileHandle.cpp:65] Generating file handle for: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet uuid: 1

So I would say gluten is able to access the gcs bucket? what do you think?

PHILO-HE · 2025-06-18T06:08:34Z

PHILO-HE
Jun 18, 2025
Collaborator

You can notice the two lines where the gcs bucket is also mentioned are:

I20250618 00:30:14.379588    91 HiveDataSource.cpp:306] Adding split Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245
I20250618 00:30:14.384384    91 FileHandle.cpp:65] Generating file handle for: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet uuid: 1

So I would say gluten is able to access the gcs bucket? what do you think?

@A-Mongy, I just reviewed the relevant code. This log cannot tell the bucket is accessible. It is logged before the reader is initialized.

4 replies

A-Mongy Jun 18, 2025
Author

You can notice the two lines where the gcs bucket is also mentioned are:
I20250618 00:30:14.379588    91 HiveDataSource.cpp:306] Adding split Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245
I20250618 00:30:14.384384    91 FileHandle.cpp:65] Generating file handle for: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet uuid: 1
So I would say gluten is able to access the gcs bucket? what do you think?

@A-Mongy, I just reviewed the relevant code. This log cannot tell the bucket is accessible. It is logged before the reader is initialized.

https://github.com/facebookincubator/velox/blob/main/velox/connectors/hive/FileHandle.cpp#L61-L65
In line 61, this looks like it's accessing the gcs bucket, no?
The logging happens in line 65 so this didn't raise an error.

    fileHandle->file = filesystems::getFileSystem(filename, properties_)
                           ->openFileForRead(filename, options);

PHILO-HE Jun 19, 2025
Collaborator

@A-Mongy, I see. Then, it seems Velox can access the object metadata if no error is reported when calling this code.

https://github.com/facebookincubator/velox/blob/b4ee6722a5f446b92888c7a47e8bccb0118688ff/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp#L865

https://github.com/facebookincubator/velox/blob/b4ee6722a5f446b92888c7a47e8bccb0118688ff/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp#L108

A-Mongy Jun 19, 2025
Author

@PHILO-HE, I am confused, is the S3 reference has anything to do with GCS buckets?

Also, on the executor pod, this is the out ofcurl -v https://storage.googleapis.com, and I can see TLSv1.3 (IN), TLS handshake, CERT verify (15): and connection is successful

*   Trying 173.194.64.207:443...
* Connected to storage.googleapis.com (173.194.64.207) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=storage.googleapis.com
*  start date: Jun  2 08:39:34 2025 GMT
*  expire date: Aug 25 08:39:33 2025 GMT
*  issuer: C=US; O=Google Trust Services; CN=WE2
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: storage.googleapis.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5762cb18e780)
> GET / HTTP/2
> Host: storage.googleapis.com
> user-agent: curl/7.88.1
> accept: */*
> 
< HTTP/2 400 
< content-type: application/xml; charset=UTF-8
< x-guploader-uploadid: ABgVH8-mL3I67FxOb9izaMRMV4DNstOyH3lpX9pzsAYWM0fhHTdIFdE0iGAmiCaM4q6ow-0
< content-length: 181
< date: Thu, 19 Jun 2025 10:12:56 GMT
< expires: Thu, 19 Jun 2025 10:12:56 GMT
< cache-control: private, max-age=0
< server: UploadServer
< 
* Connection #0 to host storage.googleapis.com left intact

PHILO-HE Jun 22, 2025
Collaborator

@A-Mongy, it's strange. Looks certificate is already available at executor side.

A-Mongy · 2025-06-22T12:42:48Z

A-Mongy
Jun 22, 2025
Author

@zhztheplayer any hints on this?

0 replies

A-Mongy · 2025-06-24T12:55:42Z

A-Mongy
Jun 24, 2025
Author

I have to add to this, when triggering a spark submit to only read data from GCS, reading a CSV or parquet file. There is no issues and it reads and prints the data from the file with no issue. So definitely not a GCS access issue and not a curl issue.

However, if I try to write data I can see the issue.
Also, with a recent jar I can see the issue is now hidden by the following error instead

Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: Empty Optional cannot be unwrapped
Retriable: False
Function: operator()
File: /work/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
Line: 571
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv.cold
# 4  _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 5  _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEERPNS1_8OperatorERNS1_14BlockingReasonE
# 6  _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 7  _ZN6gluten24WholeStageResultIterator4nextEv
# 8  Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 9  0x00007879946b9930

        at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
        at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
        at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
        ... 28 more
25/06/23 15:04:58 INFO CoarseGrainedExecutorBackend: Got assigned task 1
25/06/23 15:04:58 INFO Executor: Running task 0.1 in stage 0.0 (TID 1)

1 reply

zhouyuan Jun 24, 2025
Collaborator

It looks like something is wrong in the GCS code path as Gluten dont have tests to guard this yet.
Cc: @surnaik to check his comments

Velox, GCS and cURL | CURL error [77]=Problem with the SSL CA cert (path? access rights?) #9946

Uh oh!

A-Mongy Jun 11, 2025

Replies: 5 comments · 6 replies

Uh oh!

A-Mongy Jun 16, 2025 Author

Uh oh!

zhouyuan Jun 17, 2025 Collaborator

Uh oh!

Uh oh!

A-Mongy Jun 18, 2025 Author

Uh oh!

PHILO-HE Jun 18, 2025 Collaborator

Uh oh!

A-Mongy Jun 18, 2025 Author

Uh oh!

PHILO-HE Jun 19, 2025 Collaborator

Uh oh!

A-Mongy Jun 19, 2025 Author

Uh oh!

PHILO-HE Jun 22, 2025 Collaborator

Uh oh!

A-Mongy Jun 22, 2025 Author

Uh oh!

A-Mongy Jun 24, 2025 Author

Uh oh!

zhouyuan Jun 24, 2025 Collaborator

A-Mongy
Jun 11, 2025

Replies: 5 comments 6 replies

A-Mongy
Jun 16, 2025
Author

zhouyuan
Jun 17, 2025
Collaborator

A-Mongy Jun 18, 2025
Author

PHILO-HE
Jun 18, 2025
Collaborator

A-Mongy Jun 18, 2025
Author

PHILO-HE Jun 19, 2025
Collaborator

A-Mongy Jun 19, 2025
Author

PHILO-HE Jun 22, 2025
Collaborator

A-Mongy
Jun 22, 2025
Author

A-Mongy
Jun 24, 2025
Author

zhouyuan Jun 24, 2025
Collaborator