Bucket creation on minio compatible S3 fails

[Cenness]

What happened:
Storage format fails with

Storage minio://https://ENDPOINT/juicefs-dev-kngf/juicefs-dev-kngf/juicefs-dev-vol/ is not configured correctly:
 Failed to create bucket minio://https://ENDPOINT/juicefs-dev-kngf/juicefs-dev-kngf/juicefs-dev-vol/:
 operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: tx000004b3e48dc1f4af75e-006880d5b9-11d3b9ea0-default, 
  HostID: 11d3b9ea0-default-default, 
   api error InvalidLocationConstraint: The specified location-constraint is not valid, 
previous error: operation error S3: PutObject, https response error StatusCode: 404, RequestID: tx00000073a090fc969d60a-006880d5b8-11d566e2a-default, HostID: 11d566e2a-default-default,
 api error NoSuchBucket: UnknownError

What you expected to happen:
Bucket created, pvc mounted, files accessible

How to reproduce it (as minimally and precisely as possible):
Install JuiceFS CSI
Create secret, storageclass and pvc
Run

juicefs format \
  --storage=minio \
  --bucket=https://ENDPOINT/juicefs-dev-kngf \
  --access-key=${accesskey} \
  --secret-key=${secretkey} \
  ${metaurl} juicefs-dev-vol --trash-days=0

Anything else we need to know?
If I create bucket manually, format passes, but files can't be read:

2025/07/23 12:49:27.167746 juicefs[7] <INFO>: Meta address: redis://valk.cicd-tools.svc.cluster.local:6379/1 [NewClient@interface.go:578]
2025/07/23 12:49:27.181938 juicefs[7] <WARNING>: AOF is not enabled, you may lose data if Redis is not shutdown properly. [checkRedisInfo@info.go:84]
2025/07/23 12:49:27.182369 juicefs[7] <INFO>: Ping redis latency: 353.962µs [checkServerConfig@redis.go:3692]
2025/07/23 12:49:27.183523 juicefs[7] <INFO>: Data use minio://https://ENDPOINT/juicefs-dev-kngf/juicefs-dev-kngf/juicefs-dev-vol/ [format@format.go:527]
2025/07/23 12:49:27.567391 juicefs[7] <INFO>: Volume is formatted as {
  "Name": "juicefs-dev-vol",
  "UUID": "fdd1417f-67e8-4ecb-971a-0441d10909d3",
  "Storage": "minio",
  "Bucket": "https://ENDPOINT/juicefs-dev-kngf",
  "AccessKey": "removed",
  "SecretKey": "removed",
  "BlockSize": 4096,
  "Compression": "none",
  "EncryptAlgo": "aes256gcm-rsa",
  "KeyEncrypted": true,
  "TrashDays": 0,
  "MetaVersion": 1,
  "MinClientVersion": "1.1.0-A",
  "DirStats": true,
  "EnableACL": false
} [format@format.go:564]
2025/07/23 12:49:33.177482 juicefs[1] <INFO>: Meta address: redis://valk.cicd-tools.svc.cluster.local:6379/1 [NewClient@interface.go:578]
2025/07/23 12:49:33.304280 juicefs[1] <WARNING>: AOF is not enabled, you may lose data if Redis is not shutdown properly. [checkRedisInfo@info.go:84]
2025/07/23 12:49:33.304663 juicefs[1] <INFO>: Ping redis latency: 284.382µs [checkServerConfig@redis.go:3692]
2025/07/23 12:49:33.306941 juicefs[1] <INFO>: Data use minio://https://ENDPOINT/juicefs-dev-kngf/juicefs-dev-kngf/juicefs-dev-vol/ [mount@mount.go:588]
2025/07/23 12:49:38.772984 juicefs[32] <INFO>: Meta address: redis://valk.cicd-tools.svc.cluster.local:6379/1 [NewClient@interface.go:578]
2025/07/23 12:49:38.878923 juicefs[32] <WARNING>: AOF is not enabled, you may lose data if Redis is not shutdown properly. [checkRedisInfo@info.go:84]
2025/07/23 12:49:38.879346 juicefs[32] <INFO>: Ping redis latency: 342.628µs [checkServerConfig@redis.go:3692]
2025/07/23 12:49:38.881579 juicefs[32] <INFO>: Data use minio://https://ENDPOINT/juicefs-dev-kngf/juicefs-dev-kngf/juicefs-dev-vol/ [mount@mount.go:588]
2025/07/23 12:49:38.881615 juicefs[32] <INFO>: JuiceFS version 1.3.0+2025-07-03.30190ca1 [mount@mount.go:631]
2025/07/23 12:49:38.882951 juicefs[32] <INFO>: Disk cache (/var/jfsCache-emptyDir/fdd1417f-67e8-4ecb-971a-0441d10909d3/): used ratio - [space 0%, inode 0%] [newCacheStore@disk_cache.go:147]
2025/07/23 12:49:38.882990 juicefs[32] <INFO>: Adjusted cache capacity based on freeratio: from 1073741824 to 531502208 bytes [setlimitByFreeRatio@disk_cache.go:171]
2025/07/23 12:49:38.883002 juicefs[32] <INFO>: Adjusted max items based on freeratio: from 0 to 4066764 items [setlimitByFreeRatio@disk_cache.go:176]
2025/07/23 12:49:38.885048 juicefs[32] <INFO>: Create session 7 OK with version: 1.3.0+2025-07-03.30190ca1 [NewSession@base.go:535]
2025/07/23 12:49:38.887053 juicefs[32] <INFO>: Prometheus metrics listening on [::]:9567 [exposeMetrics@mount.go:134]
2025/07/23 12:49:38.887592 juicefs[32] <INFO>: Mounting volume juicefs-dev-vol at /jfs/pvc-36fad39e-ed62-40c4-a67c-bab88b1f6784-mthdpd ... [mountMain@mount_unix.go:1027]
2025/07/23 12:49:39.373131 juicefs[1] <INFO>: �[92mOK�[0m, juicefs-dev-vol is ready at /jfs/pvc-36fad39e-ed62-40c4-a67c-bab88b1f6784-mthdpd [checkMountpoint@mount_unix.go:235]
2025-07-23 12:49:43.569920 I | Unimplemented opcode POLL
2025/07/23 12:49:43.570602 juicefs[1] <INFO>: watching /jfs/pvc-36fad39e-ed62-40c4-a67c-bab88b1f6784-mthdpd, pid 32 [watchdog@mount_unix.go:162]
2025/07/23 12:50:21.038568 juicefs[32] <WARNING>: fail to read sliceId 28 (off:0, size:29, clen: 29, inode: 3): get chunks/0/0/28_0_29: operation error S3: GetObject, https response error StatusCode: 404, RequestID: tx0000026065c413f0c9816-006880da8d-11dbe37d7-default, HostID: 11dbe37d7-default-default, NoSuchKey:  [readSlice@reader.go:826]
2025/07/23 12:50:21.188577 juicefs[32] <WARNING>: fail to read sliceId 13 (off:0, size:319, clen: 319, inode: 3): get chunks/0/0/13_0_319: operation error S3: GetObject, https response error StatusCode: 404, RequestID: tx000001c48911bcf3ba4c1-006880da8d-11dbec199-default, HostID: 11dbec199-default-default, NoSuchKey:  [readSlice@reader.go:826]
2025/07/23 12:50:21.188823 juicefs[32] <WARNING>: compact 3 0 with 228 slices: get chunks/0/0/13_0_319: operation error S3: GetObject, https response error StatusCode: 404, RequestID: tx000001c48911bcf3ba4c1-006880da8d-11dbec199-default, HostID: 11dbec199-default-default, NoSuchKey:  [compactChunk@base.go:2314]

But juicefs appear to write something to bucket:

# rclone lsl target:juicefs-dev-kngf
       29 2025-07-23 12:49:40.597000000 juicefs-dev-vol/chunks/0/24/24577_0_29
       29 2025-07-23 12:49:45.675000000 juicefs-dev-vol/chunks/0/24/24579_0_29
       29 2025-07-23 12:49:50.724000000 juicefs-dev-vol/chunks/0/24/24580_0_29
       29 2025-07-23 12:49:55.783000000 juicefs-dev-vol/chunks/0/24/24581_0_29
       29 2025-07-23 12:50:00.827000000 juicefs-dev-vol/chunks/0/24/24582_0_29
       29 2025-07-23 12:50:05.871000000 juicefs-dev-vol/chunks/0/24/24583_0_29
       29 2025-07-23 12:50:10.928000000 juicefs-dev-vol/chunks/0/24/24584_0_29
       29 2025-07-23 12:50:15.964000000 juicefs-dev-vol/chunks/0/24/24585_0_29
       29 2025-07-23 12:50:35.497000000 juicefs-dev-vol/chunks/0/24/24597_0_29
       29 2025-07-23 12:50:40.562000000 juicefs-dev-vol/chunks/0/24/24598_0_29
       29 2025-07-23 12:50:45.598000000 juicefs-dev-vol/chunks/0/24/24599_0_29
       29 2025-07-23 12:50:50.915000000 juicefs-dev-vol/chunks/0/24/24600_0_29
       29 2025-07-23 12:50:55.963000000 juicefs-dev-vol/chunks/0/24/24601_0_29
       29 2025-07-23 12:51:01.015000000 juicefs-dev-vol/chunks/0/24/24602_0_29
       29 2025-07-23 12:51:06.059000000 juicefs-dev-vol/chunks/0/24/24603_0_29
       29 2025-07-23 12:51:11.101000000 juicefs-dev-vol/chunks/0/24/24604_0_29
       29 2025-07-23 12:51:16.166000000 juicefs-dev-vol/chunks/0/24/24605_0_29
       29 2025-07-23 12:51:21.200000000 juicefs-dev-vol/chunks/0/24/24606_0_29
       29 2025-07-23 12:51:26.250000000 juicefs-dev-vol/chunks/0/24/24607_0_29
       29 2025-07-23 12:51:31.281000000 juicefs-dev-vol/chunks/0/24/24608_0_29
       29 2025-07-23 12:51:36.339000000 juicefs-dev-vol/chunks/0/24/24609_0_29
       29 2025-07-23 12:51:41.382000000 juicefs-dev-vol/chunks/0/24/24610_0_29
       29 2025-07-23 12:51:46.410000000 juicefs-dev-vol/chunks/0/24/24611_0_29
       29 2025-07-23 12:51:51.445000000 juicefs-dev-vol/chunks/0/24/24612_0_29
       29 2025-07-23 12:51:56.497000000 juicefs-dev-vol/chunks/0/24/24613_0_29
       29 2025-07-23 12:52:01.550000000 juicefs-dev-vol/chunks/0/24/24614_0_29
       29 2025-07-23 12:52:06.585000000 juicefs-dev-vol/chunks/0/24/24615_0_29
       29 2025-07-23 12:52:11.621000000 juicefs-dev-vol/chunks/0/24/24616_0_29
       29 2025-07-23 12:52:16.664000000 juicefs-dev-vol/chunks/0/24/24617_0_29
       29 2025-07-23 12:52:21.692000000 juicefs-dev-vol/chunks/0/24/24618_0_29
       29 2025-07-23 12:52:26.737000000 juicefs-dev-vol/chunks/0/24/24619_0_29
       29 2025-07-23 12:52:31.760000000 juicefs-dev-vol/chunks/0/24/24620_0_29
       29 2025-07-23 12:52:36.805000000 juicefs-dev-vol/chunks/0/24/24621_0_29
       29 2025-07-23 12:52:41.881000000 juicefs-dev-vol/chunks/0/24/24622_0_29
       29 2025-07-23 12:52:46.908000000 juicefs-dev-vol/chunks/0/24/24623_0_29
       29 2025-07-23 12:52:51.945000000 juicefs-dev-vol/chunks/0/24/24624_0_29
       29 2025-07-23 12:52:56.992000000 juicefs-dev-vol/chunks/0/24/24625_0_29
       29 2025-07-23 12:53:02.175000000 juicefs-dev-vol/chunks/0/24/24626_0_29
       29 2025-07-23 12:53:07.201000000 juicefs-dev-vol/chunks/0/24/24627_0_29
       29 2025-07-23 12:53:12.248000000 juicefs-dev-vol/chunks/0/24/24628_0_29
       29 2025-07-23 12:53:17.280000000 juicefs-dev-vol/chunks/0/24/24629_0_29
       29 2025-07-23 12:53:22.341000000 juicefs-dev-vol/chunks/0/24/24630_0_29
       29 2025-07-23 12:53:27.389000000 juicefs-dev-vol/chunks/0/24/24631_0_29
       29 2025-07-23 12:53:32.421000000 juicefs-dev-vol/chunks/0/24/24632_0_29
       29 2025-07-23 12:53:37.456000000 juicefs-dev-vol/chunks/0/24/24633_0_29
       29 2025-07-23 12:53:42.512000000 juicefs-dev-vol/chunks/0/24/24634_0_29
       29 2025-07-23 12:53:47.553000000 juicefs-dev-vol/chunks/0/24/24635_0_29
       29 2025-07-23 12:53:52.593000000 juicefs-dev-vol/chunks/0/24/24636_0_29
       29 2025-07-23 12:53:57.700000000 juicefs-dev-vol/chunks/0/24/24637_0_29
       29 2025-07-23 12:54:02.755000000 juicefs-dev-vol/chunks/0/24/24638_0_29
       29 2025-07-23 12:54:07.796000000 juicefs-dev-vol/chunks/0/24/24639_0_29
       29 2025-07-23 12:54:12.835000000 juicefs-dev-vol/chunks/0/24/24640_0_29
       29 2025-07-23 12:54:17.871000000 juicefs-dev-vol/chunks/0/24/24641_0_29
       29 2025-07-23 12:54:22.912000000 juicefs-dev-vol/chunks/0/24/24642_0_29
       29 2025-07-23 12:54:27.940000000 juicefs-dev-vol/chunks/0/24/24643_0_29
       29 2025-07-23 12:54:32.971000000 juicefs-dev-vol/chunks/0/24/24644_0_29
       29 2025-07-23 12:54:38.044000000 juicefs-dev-vol/chunks/0/24/24645_0_29
       29 2025-07-23 12:54:43.099000000 juicefs-dev-vol/chunks/0/24/24646_0_29
       29 2025-07-23 12:54:48.156000000 juicefs-dev-vol/chunks/0/24/24647_0_29
       29 2025-07-23 12:54:53.263000000 juicefs-dev-vol/chunks/0/24/24648_0_29

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version: 1.3.0+2025-07-03.30190ca1
• Cloud provider or hardware configuration running JuiceFS: N/A
• OS (e.g cat /etc/os-release): Debian GNU/Linux 11 (bullseye)
• Kernel (e.g. uname -a): Linux juicefs-csi-controller-1 5.15.0-83-generic #92-Ubuntu SMP Mon Aug 14 09:30:42 UTC 2023 x86_64 GNU/Linux
• Object storage (cloud provider and region, or self maintained): minio compatible
• Metadata engine info (version, cloud provider managed or self maintained): valkey 8.1.3
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage): Ok

[jiefenghuang]

Could you take a look at the failure logs of your object storage?

[Cenness]

InvalidLocationConstraint: The specified location-constraint is not valid
Problem with region? I don't see an option in docs to overwrite it, but looking at the code it's populated from MINIO_REGION. I'll try setting it.

[Cenness]

Same error.
Api request for location constraint returns null

$ aws s3api --endpoint-url=https://ENDPOINT get-bucket-location --bucket existing-bucket
{
    "LocationConstraint": null
}

[jiefenghuang]

maybe you should check object storage's docs and clarify the mapping between LocationConstraint and endpoints?

example for aws:

• https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucketConfiguration.html?utm_source=chatgpt.com

[Cenness]

maybe you should check object storage's docs and clarify the mapping between LocationConstraint and endpoints?

I did. They don't map to a corresponding region. Docs say its all US, but in reality all endpoints for all AZ return null for that field.