Unable to use HF cache on a readonly filesystem

[stodoran]

Describe the bug

Even if all relevant files are downloaded to the folder specified by HF_HUB_CACHE, the hf_hub_download function will always fail due to these lines.

Reproduction

• Create directory /hf_cache
• Download some asset to said directory, e.g. huggingface-cli download Qwen/Qwen2.5-32B-Instruct
• Change directory /hf_cache to readonly mode
• Try to load the asset, e.g. AutoModel.from_pretrained("Qwen/Qwen2.5-32B-Instruct")

Logs

[rank5]: Traceback (most recent call last):
...
[rank5]:     config_new = AutoConfig.from_pretrained(model_name_or_path, token=access_token)
[rank5]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File ".venv/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1006, in from_pretrained
[rank5]:     config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
[rank5]:                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File ".venv/lib/python3.12/site-packages/transformers/configuration_utils.py", line 570, in get_config_dict
[rank5]:     config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
[rank5]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File ".venv/lib/python3.12/site-packages/transformers/configuration_utils.py", line 629, in _get_config_dict
[rank5]:     resolved_config_file = cached_file(
[rank5]:                            ^^^^^^^^^^^^
[rank5]:   File ".venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 403, in cached_file
[rank5]:     resolved_file = hf_hub_download(
[rank5]:                     ^^^^^^^^^^^^^^^^
[rank5]:   File ".venv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
[rank5]:     return fn(*args, **kwargs)
[rank5]:            ^^^^^^^^^^^^^^^^^^^
[rank5]:   File ".venv/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 860, in hf_hub_download
[rank5]:     return _hf_hub_download_to_cache_dir(
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File ".venv/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 977, in _hf_hub_download_to_cache_dir
[rank5]:     os.makedirs(os.path.dirname(blob_path), exist_ok=True)
[rank5]:   File "<frozen os>", line 215, in makedirs
[rank5]:   File "<frozen os>", line 215, in makedirs
[rank5]:   File "<frozen os>", line 225, in makedirs
[rank5]: OSError: [Errno 30] Read-only file system: '/app/hub_cache'

System info

- huggingface_hub version: 0.28.0
- Platform: Linux-6.8.0-1020-gcp-x86_64-with-glibc2.35
- Python version: 3.12.8
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Has saved token ?: True
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.5.1
- Jinja2: 3.1.5
- Graphviz: N/A
- keras: N/A
- Pydot: 3.0.3
- Pillow: 10.4.0
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: 1.26.4
- pydantic: 2.10.4
- aiohttp: 3.11.11
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /app/hf_cache
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

[Wauplin]

Hi @stodoran, thanks for reporting. Indeed, this looks like a bug that could be avoided. Would you mind opened a PR to add a try/except OSError: pass statement around those two lines ? (and add a comment about why). Thanks!

[mrtpk]

This should be resolved in the latest version:

huggingface-hub==0.35.3
transformers==4.57.1

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

def load_llm(repo_id: str, hf_token: str | None = None):
    """
    Loads a HF causal LLM + tokenizer from a repo/id.
    """

    has_cuda = torch.cuda.is_available()
    dtype = torch.bfloat16 if (has_cuda and torch.cuda.is_bf16_supported()) else torch.float16

    tokenizer = AutoTokenizer.from_pretrained(
        repo_id,
        use_fast=True,
    )

    model = AutoModelForCausalLM.from_pretrained(
        repo_id,
        torch_dtype=dtype if has_cuda else torch.float32,
        device_map="auto" if has_cuda else None,
        low_cpu_mem_usage=True,
    )

    model.eval()
    return model, tokenizer


if __name__ == "__main__":
    repo_id = "Qwen/Qwen2.5-0.5B"

    model, tokenizer = load_llm(repo_id)

    prompt = "Write a short haiku about Dubai."
    inputs = tokenizer(prompt, return_tensors="pt")

    # Move inputs to the right device when using device_map="auto"
    if torch.cuda.is_available():
        inputs = {k: v.to(model.device) for k, v in inputs.items()}

    with torch.inference_mode():
        out = model.generate(
            **inputs,
            max_new_tokens=80,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
        )

    print(tokenizer.decode(out[0], skip_special_tokens=True))