Polaris Server is unable to use AWS region from Catalog parameters

[dimas-b]

Describe the bug

Start Polaris 1.0.0 RC3 (no AWS_REGION env. var).

Try to create an AWS catalog:

 ./polaris --client-id *** --client-secret *** catalogs create polaris \
 --storage-type S3 \
 --default-base-location 's3://***/pol' \
 --role-arn arn:aws:iam::123456789012:role/test \
 --external-id pol12345 \
 --region us-west-2

Note: the create catalog command specifies a region.

Observe this error in Polaris Server log:

2025-06-27 19:09:50,685 ERROR [org.apa.pol.ser.exc.IcebergExceptionMapper] [,POLARIS] [,,,] (executor-thread-1) Unhandled exception returning INTERNAL_SERVER_ERROR: software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@46f128aa: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@2d79c0bf: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@2e7c2760: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@56798ca0: Unable to contact EC2 metadata service.]
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
	at software.amazon.awssdk.regions.providers.AwsRegionProviderChain.getRegion(AwsRegionProviderChain.java:70)
	at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.resolveRegion(AwsDefaultClientBuilder.java:374)
	at software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:604)
	at software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:593)
	at software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:400)
	at software.amazon.awssdk.utils.AttributeMap$Builder.internalGet(AttributeMap.java:389)
	at software.amazon.awssdk.utils.AttributeMap$Builder.access$1300(AttributeMap.java:201)
	at software.amazon.awssdk.utils.AttributeMap$Builder$1.get(AttributeMap.java:403)
	at software.amazon.awssdk.services.sts.DefaultStsBaseClientBuilder.lambda$finalizeServiceConfiguration$2(DefaultStsBaseClientBuilder.java:109)
	at software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:604)
	at software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:593)
	at software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:400)
	at software.amazon.awssdk.utils.AttributeMap$Builder.internalComputeIfAbsent(AttributeMap.java:331)
	at software.amazon.awssdk.utils.AttributeMap$Builder.putLazyIfAbsent(AttributeMap.java:275)
	at software.amazon.awssdk.core.client.config.SdkClientConfiguration$Builder.lazyOptionIfAbsent(SdkClientConfiguration.java:182)
	at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.finalizeAwsConfiguration(AwsDefaultClientBuilder.java:190)
	at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.finalizeChildConfiguration(AwsDefaultClientBuilder.java:172)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:202)
	at software.amazon.awssdk.services.sts.DefaultStsClientBuilder.buildClient(DefaultStsClientBuilder.java:36)
	at software.amazon.awssdk.services.sts.DefaultStsClientBuilder.buildClient(DefaultStsClientBuilder.java:25)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:169)
	at org.apache.polaris.service.storage.StorageConfiguration.lambda$stsClientSupplier$0(StorageConfiguration.java:76)
	at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:198)
	at org.apache.polaris.service.storage.PolarisStorageIntegrationProviderImpl.getStorageIntegrationForConfig(PolarisStorageIntegrationProviderImpl.java:83)
	at org.apache.polaris.service.storage.PolarisStorageIntegrationProviderImpl_ClientProxy.getStorageIntegrationForConfig(Unknown Source)
	at org.apache.polaris.persistence.relational.jdbc.JdbcBasePersistenceImpl.createStorageIntegration(JdbcBasePersistenceImpl.java:1026)
	at org.apache.polaris.core.persistence.AtomicOperationMetaStoreManager.createCatalog(AtomicOperationMetaStoreManager.java:443)
	at org.apache.polaris.service.admin.PolarisAdminService.createCatalog(PolarisAdminService.java:758)
	at org.apache.polaris.service.admin.PolarisServiceImpl.createCatalog(PolarisServiceImpl.java:149)
	at org.apache.polaris.service.admin.PolarisServiceImpl_ClientProxy.createCatalog(Unknown Source)
	at org.apache.polaris.service.admin.api.PolarisCatalogsApi.createCatalog(PolarisCatalogsApi.java:132)
	at org.apache.polaris.service.admin.api.PolarisCatalogsApi_Subclass.createCatalog$$superforward(Unknown Source)
	at org.apache.polaris.service.admin.api.PolarisCatalogsApi_Subclass$$function$$2.apply(Unknown Source)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.quarkus.hibernate.validator.runtime.interceptor.AbstractMethodValidationInterceptor.validateMethodInvocation(AbstractMethodValidationInterceptor.java:71)
	at io.quarkus.hibernate.validator.runtime.jaxrs.ResteasyReactiveEndPointValidationInterceptor.validateMethodInvocation(ResteasyReactiveEndPointValidationInterceptor.java:21)
	at io.quarkus.hibernate.validator.runtime.jaxrs.ResteasyReactiveEndPointValidationInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$8(FaultToleranceInterceptor.java:364)
	at io.smallrye.faulttolerance.core.Future.from(Future.java:85)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$9(FaultToleranceInterceptor.java:364)
	at io.smallrye.faulttolerance.core.FaultToleranceContext.call(FaultToleranceContext.java:20)
	at io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29)
	at io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:98)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:367)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:205)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
	at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.quarkus.security.runtime.interceptor.SecurityHandler.handle(SecurityHandler.java:27)
	at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor.intercept(RolesAllowedInterceptor.java:29)
	at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
	at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor.intercept(StandardSecurityCheckInterceptor.java:44)
	at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor_RolesAllowedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at org.apache.polaris.service.admin.api.PolarisCatalogsApi_Subclass.createCatalog(Unknown Source)
	at org.apache.polaris.service.admin.api.PolarisCatalogsApi$quarkusrestinvoker$createCatalog_4b9db8c445cafcfcba9631e5d7aa99b1fe89856b.invoke(Unknown Source)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:638)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

To Reproduce

No response

Actual Behavior

No response

Expected Behavior

No response

Additional context

No response

System information

No response

[dimas-b]

I believe this should be fixed by #1913.

[flyrain]

It's more like a feature request instead of a bug.

[adnanhemani]

I agree with @flyrain on this one - I'm not sure that the catalog's region is supposed to be used out of the box. The region that the error message is complaining about is the one to create (and eventually connect to) STS. What the catalog properties typically define is regarding the data storage (S3). And the code for creating STS clients today doesn't seem to have any context written in that it can/should borrow certain properties from the catalog's definition: https://github.com/apache/polaris/blob/main/service/common/src/main/java/org/apache/polaris/service/storage/StorageConfiguration.java#L71-L79. In fact, in my read of things, I'm not sure that it even supporting any catalog properties the way it is written at the moment.

There may be nuances that may occur if we start mixing STS properties into the catalog entity itself; we should be very explicit about those to be careful imo (something like calling a property "sts_xyz" if that property is specifically for STS rather than S3).