Exception com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype in Cosmos Client

[KoushikSahu]

Query/Question
We are seeing the following exception while trying to use CosmosDB client in our Synapse Spark 3.4 pools. Same code was working fine in Spark 3.3 pools. How do we fix this issue?

2025-02-07 10:40:45,332 INFO RxDocumentClientImpl [Driver]: Initializing DocumentClient [1] with serviceEndpoint [https://marsextractionprodnam.documents.azure.com:443/], connectionPolicy [ConnectionPolicy{httpNetworkRequestTimeout=PT1M, tcpNetworkRequestTimeout=PT5S, connectionMode=GATEWAY, maxConnectionPoolSize=1000, idleHttpConnectionTimeout=PT1M, idleTcpConnectionTimeout=PT0S, userAgentSuffix='', throttlingRetryOptions=RetryOptions{maxRetryAttemptsOnThrottledRequests=9, maxRetryWaitTime=PT30S}, endpointDiscoveryEnabled=true, preferredRegions=null, multipleWriteRegionsEnabled=true, proxyType=null, inetSocketProxyAddress=null, readRequestsFallbackEnabled=true, connectTimeout=PT5S, idleTcpEndpointTimeout=PT1H, maxConnectionsPerEndpoint=130, maxRequestsPerConnection=30, tcpConnectionEndpointRediscoveryEnabled=true, ioThreadPriority=5, ioThreadCountPerCoreFactor=2, tcpHealthCheckTimeoutDetectionEnabled=true, minConnectionPoolSizePerEndpoint=1, openConnectionsConcurrency=1, aggressiveWarmupConcurrency=16}], consistencyLevel [Session]
2025-02-07 10:40:45,481 INFO RxDocumentClientImpl [cosmos-parallel-1]: Getting database account endpoint from https://marsextractionprodnam.documents.azure.com:443/
2025-02-07 10:40:45,776 ERROR ClientCertificateCredential [cosmos-parallel-1]: Azure Identity => ERROR in getToken() call for scopes [https://marsextractionprodnam.documents.azure.com/.default]: com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype
2025-02-07 10:40:45,776 ERROR SimpleTokenCache [cosmos-parallel-1]: {"az.sdk.message":"Failed to acquire a new access token"}
2025-02-07 10:40:45,778 ERROR GlobalEndpointManager [cosmos-parallel-1]: Fail to reach global gateway [https://marsextractionprodnam.documents.azure.com:443/], [com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype]
2025-02-07 10:40:45,778 ERROR GlobalEndpointManager [cosmos-parallel-1]: startRefreshLocationTimerAsync() - Unable to refresh database account from any location. Exception: java.util.ServiceConfigurationError: com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype
java.util.ServiceConfigurationError: com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype
  at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:589)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1237)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
  at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
  at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
  at com.azure.core.implementation.util.Providers.<init>(Providers.java:61)
  at com.azure.core.implementation.http.HttpClientProviders.<clinit>(HttpClientProviders.java:26)
  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:61)
  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:51)
  at com.azure.identity.implementation.IdentityClientBase.getPipeline(IdentityClientBase.java:893)
  at com.azure.identity.implementation.IdentityClientBase.initializeHttpPipelineAdapter(IdentityClientBase.java:874)
  at com.azure.identity.implementation.IdentityClientBase.getConfidentialClient(IdentityClientBase.java:269)
  at com.azure.identity.implementation.IdentityClient.lambda$getConfidentialClientApplication$4(IdentityClient.java:144)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:45)
  at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:143)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
  at reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:102)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
  at reactor.core.publisher.FluxFromMonoOperator.subscribe(FluxFromMonoOperator.java:83)
  at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
  at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
  at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
  at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
  at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
  at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:829)
2025-02-07 10:40:45,791 ERROR RxDocumentClientImpl [Driver]: Client initialization failed. Check if the endpoint is reachable and if your auth token is valid. More info: https://aka.ms/cosmosdb-tsg-service-unavailable-java. More details: com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype
java.util.ServiceConfigurationError: com.azure.core.http.HttpClientProvider: com.microsoft.azure.synapse.tokenlibrary.azurecore.ApacheHttpClientProvider not a subtype
  at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:589)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1237)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
  at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
  at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
  at com.azure.core.implementation.util.Providers.<init>(Providers.java:61)
  at com.azure.core.implementation.http.HttpClientProviders.<clinit>(HttpClientProviders.java:26)
  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:61)
  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:51)
  at com.azure.identity.implementation.IdentityClientBase.getPipeline(IdentityClientBase.java:893)
  at com.azure.identity.implementation.IdentityClientBase.initializeHttpPipelineAdapter(IdentityClientBase.java:874)
  at com.azure.identity.implementation.IdentityClientBase.getConfidentialClient(IdentityClientBase.java:269)
  at com.azure.identity.implementation.IdentityClient.lambda$getConfidentialClientApplication$4(IdentityClient.java:144)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:45)
  at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:143)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
  at reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:102)
  at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
  at reactor.core.publisher.FluxFromMonoOperator.subscribe(FluxFromMonoOperator.java:83)
  at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
  at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
  at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
  at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
  at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
  at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
  at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:829)
2025-02-07 10:40:45,792 ERROR RxDocumentClientImpl [Driver]: unexpected failure in initializing client.

Code snippet we are using is,

this.cosmosClient = new CosmosClientBuilder()
                    .endpoint(endpoint)
                    .credential(new ClientCertificateCredentialBuilder()
                            .tenantId(tenantId)
                            .clientId(clientId)
                            .pfxCertificate(HelperUtils.getCertificate(config))
                            .sendCertificateChain(true)
                            .build())
                    .consistencyLevel(ConsistencyLevel.SESSION)
                    .contentResponseOnWriteEnabled(true)
                    .gatewayMode()
                    .buildClient();

Why is this not a Bug or a feature Request?
From the error message it seems that we are using some wrong version of packages. Doesn't seem like some inherent bug in the sdk itself.

Setup (please complete the following information if applicable):

• OS: NA. Code is running on Azure Synapse Spark 3.4 pools.
• IDE: NA
• Library/Libraries:
  • com.azure.azure-cosmos: 4.61.1
  • com.azure.azure-identity: 1.13.0
  • com.azure.core: 1.49.1
  • com.microsoft.azure.synapse.synapseutils_2.12: 1.4 (We have checked and found that ApacheHttpClientProvider is not coming from this package)

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Query Added
• Setup information Added

[joshfree]

Hi @KoushikSahu just a quick note that types from the older Java 7+ com.microsoft.azure.* libraries from 2018 and earlier can't be consumed by the newer Java 8+ com.azure.* (known as Track2 or the Unified Azure SDK) from 2019 and newer. More details are here: https://learn.microsoft.com/en-us/shows/azure-friday/an-introduction-to-the-unified-azure-sdk; and you can search for com.azure-based Synapse packages here: https://azure.github.io/azure-sdk/releases/latest/index.html.

[github-actions]

Hi @KoushikSahu. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[KoushikSahu]

@joshfree Couple of questions,

1. Do we need to update com.microsoft.azure.synapse.synapseutils as well?
2. We checked and saw that ApacheHttpClientProvider is not coming from the com.microsoft.azure.* package. We are guessing that it is getting introduced by the Synapse runtime and getting used by reflection. So the use of com.microsoft.azure.* cannot be the reason for this exception right?