Cannot connect to backblaze b2, got "Unexpected error: S3Error#100 MissingContentLength: Unable to parse ExceptionName: MissingContentLength Message: You must provide the Content-Length HTTP header. for object '_arctic_cfg/cref/*sUt*test'"

[tddschn]

Describe the bug

import arcticdb as adb

db = adb.Arctic(url := get_s3_url())
print(url)
db.create_library("test")

gives error Unexpected error: S3Error#100 MissingContentLength: Unable to parse ExceptionName: MissingContentLength Message: You must provide the Content-Length HTTP header. for object '_arctic_cfg/cref/*sUt*test'

Steps/Code to Reproduce

I'm using a new backblaze bucket created today, which they claim to be 'S3-compatible', and got the keys from https://secure.backblaze.com/app_keys.htm .

Config of the bucket:

Created:March 17, 2025
Bucket ID:<>
Type:Private
File Lifecycle:Keep all versions
Snapshots:0
Current Files:0
Current Size:0 bytes
Endpoint:s3.us-west-000.backblazeb2.com
Encryption:Disabled

.env:

# https://secure.backblaze.com/app_keys.htm
B2_ENDPOINT=https://s3.us-west-000.backblazeb2.com
# redacted keys
B2_KEY_ID=key_id 
B2_ACCESS_KEY=key

# https://github.com/man-group/ArcticDB/blob/8635f8c2fd70ef5693dbe094096d338e62f9ee5b/docs/mkdocs/docs/runtime_config.md?plain=1#L104
# debug logging
ARCTICDB_AWS_LogLevel_int=6

def get_s3_url(
    env_var_prefix: str = "B2", bucket: str = 'my-bucket') -> str:
    import os
    from dotenv import load_dotenv

    load_dotenv()
    _access_key = os.getenv(f"{env_var_prefix}_ACCESS_KEY")
    _key_id = os.getenv(f"{env_var_prefix}_KEY_ID")
    _endpoint = os.getenv(f"{env_var_prefix}_ENDPOINT")
    return f"s3://{_endpoint}:{bucket}?access={_key_id}&secret={_access_key}"

import arcticdb as adb

db = adb.Arctic(url := get_s3_url())
print(url)
db.create_library("test")

Output:

s3://https://s3.us-west-000.backblazeb2.com:my-bucket?access=<>&secret=<> # redacted
20250317 19:08:12.307973 83073196 E arcticdb | Unexpected error: S3Error#100 MissingContentLength: Unable to parse ExceptionName: MissingContentLength Message: You must provide the Content-Length HTTP header. for object '_arctic_cfg/cref/*sUt*test'
Traceback (most recent call last):
  File "./arctic_s3_test.py", line 10, in <module>
    db.create_library("test")
    └ Arctic(config=S3(endpoint=https://s3.us-west-000.backblazeb2.com, bucket=my-bucket))
  File "/Users/tsca/mamba/envs/fin/lib/python3.11/site-packages/arcticdb/arctic.py", line 214, in create_library
    self._library_manager.write_library_config(cfg, lib_mgr_name, self._library_adapter.get_masking_override())
    │                                          │    │             └ Arctic(config=S3(endpoint=https://s3.us-west-000.backblazeb2.com, bucket=my-bucket))
    │                                          │    └ 'test'
    │                                          └ lib_desc {
  name: "test"
  storage_ids: "test1742252892219304960_store"
  version {
    write_options {
      column_group_size...
    └ Arctic(config=S3(endpoint=https://s3.us-west-000.backblazeb2.com, bucket=my-bucket))
arcticdb_ext.exceptions.StorageException: E_UNEXPECTED_S3_ERROR Unexpected error: S3Error#100 MissingContentLength: Unable to parse ExceptionName: MissingContentLength Message: You must provide the Content-Length HTTP header. for object '_arctic_cfg/cref/*sUt*test'

list_libraries() works and returns [], but create_library() doesn't work.

Expected Results

s3://https://s3.us-west-000.backblazeb2.com:my-bucket?access=<>&secret=<> # redacted works seemlessly with Arctic.

OS, Python Version and ArcticDB Version

Python: 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:26:40) [Clang 14.0.6 ]
OS: macOS-15.3.1-arm64-arm-64bit
ArcticDB: 5.2.6

Backend storage used

backblaze b2

Additional Context

I also tried removing https:// from the endpoint env var, and using s3s://, both didn't work either.

The credentials work for other S3 tools like https://github.com/minio/mc .

[phoebusm]

Hi @tddschn my BackBlaze test fails but with a different error. Seems the problem is limited to Mac only as the same test passes on Linux.
I need to dig deeper to understand what has been missed.

Thanks for the reporting the issue

[phoebusm]

The error is due to since aws-sdk-cpp >= 1.11.486, it has turned on checksum integrity check x-amz-checksum-mode: enabled. This feature is not supported in many s3 implementation, include BackBlaze.
ArcticDB pypi build has pinned the sdk version at 1.11.474 but unfortunately such hasn't been done on conda build (Mac)
I will add the missing pin soon.

As a interim solution, please add export AWS_REQUEST_CHECKSUM_CALCULATION=when_required and export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required to disable the checksum check.