Cannot add blobfuse mounted volume

[andrinr]

Description

When trying to add a directory that was mounted using blobfuse2 to the volume argument serving will fail with:

Error response from daemon: error while creating mount source path '/home/azureuser/localfiles/local_datasets/sygaldry-gridded/checkpoints-small-grid-v2-3': mkdir /home/azureuser/localfiles/local_datasets/sygaldry-gridded: file exists

Steps to reproduce

tesseract serve vectoradd-jax  -v "<some-mounted-dir>:/tesseract/checkpoint"

Logs

ptim) (azureml_py38) azureuser@sygaldry8:~/localfiles/sygaldry$ tesseract serve dpp-unet-inference -v "/home/azureuser/localfiles/local_datasets/sygaldry-gridded/checkpoints-small-grid-v2-3:/tesseract/checkpoint"
/anaconda/envs/optim/lib/python3.12/site-packages/requests/__init__.py:86: RequestsDependencyWarning: Unable to find acceptable character detection dependency (chardet or charset_normalizer).
  warnings.warn(
 [i] Waiting for Tesseract containers to start ...
 [-] Command '['docker', 'compose', '-f', '/var/tmp/docker-compose-nm9qqnden9qa.yml5tsk8bxq', '-p', 'tesseract-pqdztn6oqyer', 'up', '-d', '--wait']' returned non-zero exit status 1.
 [-]  Network tesseract-pqdztn6oqyer_multi-tesseract-network  Creating
 Network tesseract-pqdztn6oqyer_multi-tesseract-network  Created
 Container tesseract-pqdztn6oqyer-sha256-md03z07eymkm-1  Creating
 Container tesseract-pqdztn6oqyer-sha256-md03z07eymkm-1  Created
 Container tesseract-pqdztn6oqyer-sha256-md03z07eymkm-1  Starting
Error response from daemon: error while creating mount source path '/home/azureuser/localfiles/local_datasets/sygaldry-gridded/checkpoints-small-grid-v2-3': mkdir /home/azureuser/localfiles/local_datasets/sygaldry-gridded: file exists

 [-] Container 4a0edd9a8ef00c5ecfdaf14cf15d347016fbe13de67a2441921c527890dc8b89 logs:
 [-] 
 [x] Uncaught error
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /anaconda/envs/optim/lib/python3.12/site-packages/tesseract_core/sdk/docker_client.py:684 in up  │
│                                                                                                  │
│   681 │   │   docker_compose = _get_executable("docker-compose")                                 │
│   682 │   │   logger.info("Waiting for Tesseract containers to start ...")                       │
│   683 │   │   try:                                                                               │
│ ❱ 684 │   │   │   _ = subprocess.run(                                                            │
│   685 │   │   │   │   [                                                                          │
│   686 │   │   │   │   │   *docker_compose,                                                       │
│   687 │   │   │   │   │   "-f",                                                                  │
│                                                                                                  │
│ /anaconda/envs/optim/lib/python3.12/subprocess.py:573 in run                                     │
│                                                                                                  │
│    570 │   │   │   raise                                                                         │
│    571 │   │   retcode = process.poll()                                                          │
│    572 │   │   if check and retcode:                                                             │
│ ❱  573 │   │   │   raise CalledProcessError(retcode, process.args,                               │
│    574 │   │   │   │   │   │   │   │   │    output=stdout, stderr=stderr)                        │
│    575 │   return CompletedProcess(process.args, retcode, stdout, stderr)                        │
│    576                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['docker', 'compose', '-f', '/var/tmp/docker-compose-nm9qqnden9qa.yml5tsk8bxq', '-p', 'tesseract-pqdztn6oqyer', 'up', '-d', '--wait']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /anaconda/envs/optim/lib/python3.12/site-packages/tesseract_core/sdk/engine.py:116 in            │
│ wrapper_needs_docker                                                                             │
│                                                                                                  │
│   113 │   │   │   raise UserError(                                                               │
│   114 │   │   │   │   "Could not reach Docker daemon, check if it is running."                   │
│   115 │   │   │   ) from ex                                                                      │
│ ❱ 116 │   │   return func(*args, **kwargs)                                                       │
│   117 │                                                                                          │
│   118 │   return wrapper_needs_docker                                                            │
│   119                                                                                            │
│                                                                                                  │
│ /anaconda/envs/optim/lib/python3.12/site-packages/tesseract_core/sdk/cli.py:495 in serve         │
│                                                                                                  │
│   492 │   │   ports = None                                                                       │
│   493 │                                                                                          │
│   494 │   try:                                                                                   │
│ ❱ 495 │   │   project_id = engine.serve(                                                         │
│   496 │   │   │   image_names,                                                                   │
│   497 │   │   │   host_ip,                                                                       │
│   498 │   │   │   ports,                                                                         │
│                                                                                                  │
│ /anaconda/envs/optim/lib/python3.12/site-packages/tesseract_core/sdk/engine.py:659 in serve      │
│                                                                                                  │
│   656 │   │   compose_file.flush()                                                               │
│   657 │   │                                                                                      │
│   658 │   │   project_name = f"tesseract-{_id_generator()}"                                      │
│ ❱ 659 │   │   if not docker_client.compose.up(compose_file.name, project_name):                  │
│   660 │   │   │   raise RuntimeError("Cannot serve Tesseracts")                                  │
│   661 │   │   return project_name                                                                │
│   662                                                                                            │
│                                                                                                  │
│ /anaconda/envs/optim/lib/python3.12/site-packages/tesseract_core/sdk/docker_client.py:710 in up  │
│                                                                                                  │
│   707 │   │   │   │   container = Containers.get(container_name)                                 │
│   708 │   │   │   │   logger.error(f"Container {container_name} logs:")                          │
│   709 │   │   │   │   logger.error(indent(container.logs(stderr=True).decode(), " > "))          │
│ ❱ 710 │   │   │   raise APIError("Failed to start Tesseract containers.") from ex                │
│   711 │                                                                                          │
│   712 │   @staticmethod                                                                          │
│   713 │   def down(project_id: str) -> bool:                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
APIError: Failed to start Tesseract containers.
 [x] Aborting

OS

Linux

Tesseract version

0.9.1

[dionhaefner]

Not sure if this is supported for bind mounts. @heitorPB any clue as to what's going on?

[heitorPB]

I can't reproduce this particular error:

❯ azure-storage-fuse mount all /tmp/blobs/ --config-file=/tmp/blob.yml --read-only
*** azure-storage-fuse: A new version [2.5.0~preview.1] is available. Consider upgrading to latest version for bug-fixes & new features. ***
Mounting container : redacted to path  /tmp/blobs/redacted

1 of 1 containers were successfully mounted


❯ uv run tesseract serve vectoradd -v "/tmp/blobs/:/tesseract/blobs"
 [i] Waiting for Tesseract containers to start ...
 [i] Container ID: e3a90acd6a26cb75e1692b57346fc62552b0a0e9d5acc2d2e13b5a1549e9b1df
 [i] Name: tesseract-0nzwtkkui10o-sha256-lwrfngm9vjby-1
 [i] Entrypoint: ['tesseract-runtime']
 [i] View Tesseract: http://127.0.0.1:59709/docs
 [i] Tesseract project ID, use it with 'tesseract teardown' command: tesseract-0nzwtkkui10o
{"project_id": "tesseract-0nzwtkkui10o", "containers": [{"name": "tesseract-0nzwtkkui10o-sha256-lwrfngm9vjby-1", "port": "59709", "ip": "127.0.0.1"}]}

But I can't use the files either, the permissions are totally messed up inside the container:

❯ docker exec -it tesseract-0nzwtkkui10o-sha256-lwrfngm9vjby-1 bash
tesseractor@e3a90acd6a26:/tesseract$ ls -l blobs/
ls: cannot access 'blobs/redacted': Permission denied
total 0
d????????? ? ? ? ?            ? redacted
tesseractor@e3a90acd6a26:/tesseract$

Need to troubleshoot further

[pavloboyko]

I tried but failed to reproduce the issue; mounting blobfuse volumes from the host to tesseract works fine on my setup.

It's WSL2 on Windows, Docker Desktop 4.41.2 (191736). Details:

$ uname -a
Linux DESKTOP-5C0SF3L 5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.1 LTS
Release:        24.04
Codename:       noble

$ docker --version
Docker version 28.1.1, build 4eba377

$ blobfuse2 --version
blobfuse2 version 2.5.0

Blobfuse config:

$ cat config.yml 
allow-other: true

logging:
  type: syslog
  level: log_warning

components:
  - libfuse
  - file_cache
  - attr_cache
  - azstorage

libfuse:
  attribute-expiration-sec: 120
  entry-expiration-sec: 120
  negative-entry-expiration-sec: 240

file_cache:
  path: ~/wk/blobfuse/cache
  timeout-sec: 120
  max-size-mb: 4096

attr_cache:
  timeout-sec: 7200

azstorage:
  type: block
  account-name: [redacted]
  account-key: [redacted]
  container: [redacted]
  mode: key

Mounting on the host:

$ sudo blobfuse2 mount /mnt/blobfuse --config-file=config.yml
$ mount
[...]
blobfuse2 on /mnt/wsl/docker-desktop-bind-mounts/Ubuntu/a6fa2e5d06d13fb535bb7c0fdcdea1c2ee3151bf2b5741ad7589171c138907b4 type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
blobfuse2 on /mnt/blobfuse type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
$ echo Message > /mnt/blobfuse/file.txt

Mounting from the host to the container:

$ uv run tesseract serve vectoradd -v "/mnt/blobfuse:/tesseract/blobfuse"
 [i] Waiting for Tesseract to start...
 [i] Serving Tesseract at http://127.0.0.1:58761
 [i] View Tesseract: http://127.0.0.1:58761/docs
 [i] Container ID: 9efb6deef4c1965e4b6eb02517b24df0845e880dfa587c661ca457dbc9a9f77f
 [i] Name: elegant_golick
 [i] Entrypoint: ['tesseract-runtime']
 [i] View Tesseract: http://127.0.0.1:58761/docs
 [i] Tesseract container name, use it with 'tesseract teardown' command: elegant_golick
{"container_name": "elegant_golick", "containers": [{"name": "elegant_golick", "port": "58761", "ip": "127.0.0.1", "networks": {"bridge": {"ip": "172.17.0.3", "port": 8000}}}]}

$ docker exec -it elegant_golick bash
I have no name!@9efb6deef4c1:/tesseract$ cat blobfuse/file.txt 
Message

The volume inside the container is read-only:

$ echo Message2 >> blobfuse/file.txt 
bash: blobfuse/file.txt: Read-only file system

[dionhaefner]

Thanks @pavloboyko for giving this a try! Let's see if @andrinr can still reproduce this. If not, it may have been fixed by one of the many refactors we did recently.