S3 default storage class breaks Minio compatibility

[tycho]

This line:

vaultwarden/src/config.rs

Line 1243 in a0198d8

.default_storage_class("INTELLIGENT_TIERING");

Causes problems if using Minio as the S3 endpoint:

[REQUEST s3.PutObject] [2025-07-28T10:34:52.211] [Client IP: redacted]
PUT /rsa_key.pem
Proto: HTTP/1.1
Host: [redacted]
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Forwarded-For: [redacted]
X-Forwarded-Proto: https
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=vaultwarden/20250728/us-east-1/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class, Signature=4d257b5c8807a7cddf1c2c49749af1f96dce936a0cdf11f96ed363fce5323f81
X-Amz-Date: 20250728T173452Z
X-Amz-Storage-Class: INTELLIGENT_TIERING
X-Real-Ip: [redacted]
Accept-Encoding: gzip, br, zstd, deflate
Content-Length: 1675
<BLOB>
[RESPONSE] [2025-07-28T10:34:52.211] [ Duration 58µs TTFB 50.045µs ↑ 153 B  ↓ 338 B ]
400 Bad Request
Content-Length: 338
Content-Type: application/xml
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin,Accept-Encoding
X-Amz-Id-2: 37b62f02de360c6949659e1d2fb72852efb5d74e5dcbc7d4e665920a1c72a29e
X-Ratelimit-Limit: 20413
X-Xss-Protection: 1; mode=block
Server: MinIO
X-Amz-Request-Id: 18567AC52E4FC471
X-Content-Type-Options: nosniff
X-Ratelimit-Remaining: 20413
Accept-Ranges: bytes
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidStorageClass</Code><Message>Invalid storage class.</Message><Key>rsa_key.pem</Key><BucketName>vaultwarden</BucketName><Resource>/rsa_key.pem</Resource><RequestId>18567AC52E4FC471</RequestId><HostId>37b62f02de360c6949659e1d2fb72852efb5d74e5dcbc7d4e665920a1c72a29e</HostId></Error>

Can that storage class configuration be made user-configurable or even optional, please?

[tycho]

Additionally, this line makes it hard to use other S3-compatible implementations like Garage:

vaultwarden/src/config.rs

Line 1240 in a0198d8

.enable_virtual_host_style()

[BlackDex]

@txase, you might have some ideas here?

[txase]

I wonder if we can look at the bucket config to determine if we should call these methods or not. We definitely want these options if it’s a real aws s3 bucket. If it’s not, then we can skip calling these methods.

I can look into this within the next few days and report back here on what I think we should do.

[rezzorix]

Just wondering... isnt MiniO something to avoid or move away from anyway considering what the developers recently pulled?

Just one example; https://www.reddit.com/r/selfhosted/comments/1kva3pw/avoid_minio_developers_introduce_trojan_horse/

So maybe not worth to spend too many resources and effort on a software that will be not so popular overall anymore?

[Asutorufa]

The aws s3 also support Use Path Style, so consider setting Use Path Style to true.
There are many S3-compatible implementations, such as Cloudflare R2, Cephfs, SeaweedFS, Garage, and minio.

[txase]

AWS has said they are deprecating path style hosting. That said, it’s been so long now that it’s unclear if they really will do it. But it is clear that they suggest virtual host style hosting is the preferred way to operate.

[txase]

I have now reviewed s3 and a few compatible services, including minio and garage. I think we need an extensible solution for enabling s3 features, as each service supports different sets of capabilities. @tycho @BlackDex: Please respond with agreement or concerns with the following proposal. 🙏

Proposal

We currently support a URI for file paths for s3-compatible stores where the URI starts with s3://:

vaultwarden/src/config.rs

Line 1173 in 5d84f17

let operator = if path.starts_with("s3://") {

We can add support for querystring parameters for s3 features, which would be similar to how many database connection strings support parameters. For example, we could have the following for minio:

s3://my-minio-bucket?storage-class=STANDARD

And for garage:

s3://my-garage-bucket?path-style-requests

An example of this parameterization approach can be found in the Go Cloud Development Kit.

Alternative Proposal 1: Provider Parameters

We could set a provider parameter: s3://my-minio-bucket?provider=minio to determine which s3 features to enable. However, I think it is likely easier to maintain per-feature params over time (e.g. if minio began supporting intelligent tiering tomorrow, what should the Vaultwarden provider=minio functionality do?). It also means each service provider must have its own hardcoded set of features, and/or users must map new service providers to other existing service providers already implemented in Vaultwarden that enable a compatible set of features.

Alternative Proposal 2: Provider Protocols

We could use the URI protocol to specify the provider type: minio://my-minio-bucket. But this has the same long-term maintenance issue as the above provider parameter alternative proposal.

Alternative Proposal 3: Dynamic Checks

When we first access an s3-compatible service we could check whether it supports various features. However, this may require multiple requests and the possibility of mutating the underlying s3 store. For example, there is no way to check whether intelligent tiering is available other than to attempt to create an object. This will also increase startup latency, which is a problem when running Vaultwarden in a serverless FaaS (Functions-as-a-Service) service like AWS Lambda.

[dani-garcia]

I think your first proposal looks pretty good and it also mirrors something that OpenDAL itself wants to implement in the future: apache/opendal#5445. That seems like it would resolve this issue for us, but it's not implemented yet. I'd recommend to try to keep it similar to the OpenDAL config naming, so we can migrate over once that's released, so something like might work best:

s3://my-minio-bucket?default_storage_class=STANDARD
s3://my-garage-bucket?enable_virtual_host_style=false

[txase]

@dani-garcia Oh, good find!

This reminds me that @Xuanwo chimed in to offer guidance on integrating OpenDAL in #5626. @Xuanwo: Can you confirm that this is the best approach, and that the specific params and values @dani-garcia provided above would be correct for future OpenDAL compatibility?

[Xuanwo]

Hi, I think the parameters proposed by @dani-garcia are correct and compatible.

[BlackDex]

Looks good to me too. And, it also mimics the same way as is done for Diesel's Database connections.

[txase]

Great!

@tycho: Is this something you would have an interest in drafting a PR for to fix your issue? If you're not able (e.g. you're not familiar enough with Rust), I can probably get to it sometime in the next week. But I don't want to steal away an opportunity for you to have fun getting it working :).

[tycho]

I might be able to, however I've just started a new job so I might not have time for a bit. If someone can get something done sooner, I'd really appreciate if they could work on it!

I think the best path forward is the initial proposal by @txase, with specific functionality overrides in query strings. We could customize things like:

• virtual_host_style, 1 or 0, to enable use of virtual host-based vs. path-based respectively
• storage_class, supporting any option described in the OpenDAL docs, though we can probably just pass it through directly as a string and let OpenDAL decide whether it's valid
• access_key_id and secret_access_key, it might make sense to allow explicitly defining credentials in the connection string, instead of relying on the AWS CLI being configured, especially in a service account like Vaultwarden would typically use (that was a pain point when setting this up for me actually)
• endpoint_uri, another useful one for people that don't have the AWS CLI configured -- when using an S3-compatible self-hosted option, you don't want to use the default https://s3.amazonaws.com endpoint, you want to use a custom one. Right now to do that you have to plug it into the AWS CLI config, so having an explicit option would be good!

So a complete S3 storage URI would probably look something like:

s3://bucket-name?endpoint_uri=https%3A%2F%2Fminio.example.com&access_key_id=SOME_ACCESS_KEY_ID&secret_access_key=SOME_SECRET_ACCESS_KEY&storage_class=STANDARD&virtual_host_style=0

[Xuanwo]

• access_key_id and secret_access_key, it might make sense to allow explicitly defining credentials in the connection string, instead of relying on the AWS CLI being configured, especially in a service account like Vaultwarden would typically use (that was a pain point when setting this up for me actually)

Hi, OpenDAL supports ENV, IMDSv2, and many other credential loaders, so usually you can just use IAM to configure them instead of static keys, which are insecure.