Fail to create buckets in Minio Object Storage due to missing region header

[levindecaro]

problem

Attempting to create buckets on Minio Cluster via ACS resulted in an error due to an invalid authorization header.

2025-06-23 19:19:05,701 DEBUG [o.a.c.s.o.BucketApiServiceImpl] (API-Job-Executor-86:[ctx-5f0c251b, job-137139, ctx-e514f1d3]) (logid:93e5ca37) Failed to create bucket with name: test-1 com.cloud.utils.exception.CloudRuntimeException: The authorization header is malformed; the region is wrong; expecting 'ap-east-1'.
        at org.apache.cloudstack.storage.datastore.driver.MinIOObjectStoreDriverImpl.createBucket(MinIOObjectStoreDriverImpl.java:125)
        at org.apache.cloudstack.storage.object.store.ObjectStoreImpl.createBucket(ObjectStoreImpl.java:115)
        at org.apache.cloudstack.storage.object.BucketApiServiceImpl.createBucket(BucketApiServiceImpl.java:147)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy442.createBucket(Unknown Source)
        at org.apache.cloudstack.api.command.user.bucket.CreateBucketCmd.execute(CreateBucketCmd.java:188)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: error occurred
ErrorResponse(code = AuthorizationHeaderMalformed, message = The authorization header is malformed; the region is wrong; expecting 'ap-east-1'., bucketName = test-1, objectName = null, resource = /test-1, requestId = 184BA7F365D8E2FD, hostId = ece579700584d4239837542582e082c7b7b451a09695091575f9aa679ff22c96)
request={method=PUT, url=https://apac-1a-s3.ap-east-1.example.com/test-1, headers=Host: apac-1a-s3.ap-east-1.example.com
Accept-Encoding: identity
User-Agent: MinIO (Linux; amd64) minio-java/dev
Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-date: 20250623T111905Z
Authorization: ██
}
response={code=400, headers=Accept-Ranges: bytes
Content-Length: 409
Content-Type: application/xml
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: ap-east-1
X-Amz-Id-2: ece579700584d4239837542582e082c7b7b451a09695091575f9aa679ff22c96
X-Amz-Request-Id: 184BA7F365D8E2FD
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 1605
X-Ratelimit-Remaining: 1605
X-Xss-Protection: 1; mode=block
Date: Mon, 23 Jun 2025 11:19:03 GMT
}

        at io.minio.S3Base$1.onResponse(S3Base.java:690)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
        ... 3 more

2025-06-23 19:19:05,703 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-86:[ctx-5f0c251b, job-137139, ctx-e514f1d3]) (logid:93e5ca37) Skipping delivering event Event {"description":"{\"details\":\"Bucket Id: f88ce50f-9afa-4188-aeba-d521f1195d58\",\"event\":\"BUCKET.CREATE\",\"status\":\"Completed\"}","eventId":null,"eventType":"BUCKET.CREATE","eventUuid":null,"resourceType":"Bucket","resourceUUID":null} to any webhook as account ID is missing
2025-06-23 19:19:05,703 WARN  [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-86:[ctx-5f0c251b, job-137139, ctx-e514f1d3]) (logid:93e5ca37) Failed to publish event [category: ActionEvent, type: BUCKET.CREATE] on bus webhookEventBus
2025-06-23 19:19:05,709 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-86:[ctx-5f0c251b, job-137139]) (logid:93e5ca37) Complete async job-137139, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create bucket with name: test-1. The authorization header is malformed; the region is wrong; expecting 'ap-east-1'."}
2025-06-23 19:19:05,710 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-86:[ctx-5f0c251b, job-137139]) (logid:93e5ca37) Publish async job-137139 complete on message bus
2025-06-23 19:19:05,710 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-86:[ctx-5f0c251b, job-137139]) (logid:93e5ca37) Wake up jobs related to job-137139
2025-06-23 19:19:05,710 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-86:[ctx-5f0c251b, job-137139]) (logid:93e5ca37) Update db status for job-137139
2025-06-23 19:19:05,711 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-86:[ctx-5f0c251b, job-137139]) (logid:93e5ca37) Wake up jobs joined with job-137139 and disjoin all subjobs created from job- 137139
2025-06-23 19:19:05,715 DEBUG [c.c.a.ApiServer] (API-Job-Executor-86:[ctx-5f0c251b, job-137139]) (logid:93e5ca37) Retrieved cmdEventType from job info: BUCKET.CREATE
2025-06-23 19:19:05,716 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-86:[ctx-5f0c251b, job-137139]) (logid:93e5ca37) Done executing org.apache.cloudstack.api.command.user.bucket.CreateBucketCmd for job-137139
2025-06-23 19:19:05,717 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-86:[ctx-5f0c251b, job-137139]) (logid:93e5ca37) Remove job-137139 from job monitoring

versions

ACS: 4.20.1
OS: OL9 x86_64
hypervisors: KVM
storage: NFS
network: vxlan

The steps to reproduce the bug

1. Configure Object Storage Provider
2. Create buckets

What to do about it?

No response

[abh1sar]

@levindecaro The MinIO Client used by cloudstack uses the default region us-east-1.
Can you rename the region in your Minio cluster to 'us-east-1' and try?

[abh1sar]

I guess it should work with the region ap-east-1 as well if the object store url is something like s3.ap-east-1.amazonaws.com so that minio client can infer the region from the url. Otherwise it will try to create the bucket in us-east-1.

[levindecaro]

@abh1sar is it must start from s3 subdomain name? mine is apac-1a-s3.ap-east-1.example.com already

[abh1sar]

yes, it needs to start with s3 and it should end with amazonaws or similar.

[levindecaro]

@abh1sar I still got same failed result after adjusting fqdn, any suggestion?

2025-06-26 23:27:53,882 DEBUG [o.a.c.s.o.BucketApiServiceImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906, ctx-b9b709d3]) (logid:30bdb6a7) Failed to create bucket with name: test-1 com.cloud.utils.exception.CloudRuntimeException: The authorization header is malformed; the region is wrong; expecting 'ap-east-1'.
        at org.apache.cloudstack.storage.datastore.driver.MinIOObjectStoreDriverImpl.createBucket(MinIOObjectStoreDriverImpl.java:125)
        at org.apache.cloudstack.storage.object.store.ObjectStoreImpl.createBucket(ObjectStoreImpl.java:115)
        at org.apache.cloudstack.storage.object.BucketApiServiceImpl.createBucket(BucketApiServiceImpl.java:147)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy442.createBucket(Unknown Source)
        at org.apache.cloudstack.api.command.user.bucket.CreateBucketCmd.execute(CreateBucketCmd.java:188)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: error occurred
ErrorResponse(code = AuthorizationHeaderMalformed, message = The authorization header is malformed; the region is wrong; expecting 'ap-east-1'., bucketName = test-1, objectName = null, resource = /test-1, requestId = 184CA145288407B8, hostId = 4fba84ceeb976175df772208f0c33f2701fa9e10940910b2ed75f362a95f469c)
request={method=PUT, url=https://s3.ap-east-1.example.com/test-1, headers=Host: s3.ap-east-1.example.com
Accept-Encoding: identity
User-Agent: MinIO (Linux; amd64) minio-java/dev
Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-date: 20250626T152753Z
Authorization: ██
}
response={code=400, headers=Accept-Ranges: bytes
Content-Length: 409
Content-Type: application/xml
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: ap-east-1
X-Amz-Id-2: 4fba84ceeb976175df772208f0c33f2701fa9e10940910b2ed75f362a95f469c
X-Amz-Request-Id: 184CA145288407B8
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 1577
X-Ratelimit-Remaining: 1577
X-Xss-Protection: 1; mode=block
Date: Thu, 26 Jun 2025 15:27:53 GMT
}

        at io.minio.S3Base$1.onResponse(S3Base.java:690)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
        ... 3 more

2025-06-26 23:27:53,887 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906, ctx-b9b709d3]) (logid:30bdb6a7) Skipping delivering event Event {"description":"{\"details\":\"Bucket Id: 6181c9dd-15fc-40ac-aaa7-437074c3fb9c\",\"event\":\"BUCKET.CREATE\",\"status\":\"Completed\"}","eventId":null,"eventType":"BUCKET.CREATE","eventUuid":null,"resourceType":"Bucket","resourceUUID":null} to any webhook as account ID is missing
2025-06-26 23:27:53,887 WARN  [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906, ctx-b9b709d3]) (logid:30bdb6a7) Failed to publish event [category: ActionEvent, type: BUCKET.CREATE] on bus webhookEventBus
2025-06-26 23:27:53,897 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Complete async job-137906, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create bucket with name: test-1. The authorization header is malformed; the region is wrong; expecting 'ap-east-1'."}
2025-06-26 23:27:53,897 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Publish async job-137906 complete on message bus
2025-06-26 23:27:53,898 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Wake up jobs related to job-137906
2025-06-26 23:27:53,898 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Update db status for job-137906
2025-06-26 23:27:53,898 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Wake up jobs joined with job-137906 and disjoin all subjobs created from job- 137906
2025-06-26 23:27:53,903 DEBUG [c.c.a.ApiServer] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Retrieved cmdEventType from job info: BUCKET.CREATE
2025-06-26 23:27:53,904 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Done executing org.apache.cloudstack.api.command.user.bucket.CreateBucketCmd for job-137906
2025-06-26 23:27:53,905 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Remove job-137906 from job monitoring

[abh1sar]

@levindecaro which FQDN did you use?

[levindecaro]

@levindecaro which FQDN did you use?

s3.ap-east-1.[my company domain].com

[abh1sar]

I looked at Minio sdk code.

        private boolean isAwsEndpoint(String endpoint) {
            return (endpoint.startsWith("s3.") || this.isAwsFipsEndpoint(endpoint) || this.isAwsAccelerateEndpoint(endpoint)) && (endpoint.endsWith(".amazonaws.com") || endpoint.endsWith(".amazonaws.com.cn"));
        }
        private void setBaseUrl(HttpUrl url) {
            String host = url.host();
            this.isAwsHost = this.isAwsEndpoint(host);
            this.isAwsChinaHost = false;
            if (this.isAwsHost) {
                this.isAwsChinaHost = host.endsWith(".cn");
                url = url.newBuilder().host(this.isAwsChinaHost ? "amazonaws.com.cn" : "amazonaws.com").build();
                this.isFipsHost = this.isAwsFipsEndpoint(host);
                this.isAccelerateHost = this.isAwsAccelerateEndpoint(host);
                this.isDualStackHost = this.isAwsDualStackEndpoint(host);
                this.regionInUrl = this.extractRegion(host);
                this.useVirtualStyle = true;
            } else {
                this.useVirtualStyle = host.endsWith("aliyuncs.com");
            }

            this.baseUrl = url;
        }

it will refer the region from url only if isAwsEndpoint is true. There are some other logic in how they build the url which again takes amazonaws.com

It might be difficult, but you can try if any strategy works by looking at the code.
Otherwise using the region us-east-1 is a workaround.

I don't think there is anything we can do from CLoudStack's side, as the logic is built into the Minio sdk.