huggingface-cli[hf_xet] error and low download speed (on slow residential connection) #3036
Labels
bug
Something isn't working
Comments
|
Third try, same location, new error: |
|
Hi @aa956, thank you! |
|
No problem, it's not critical at all, just downloading some models and decided to try new hf_xet method. Looks like download continued for some time with aa956@desktop02:~$ HF_XET_NUM_CONCURRENT_RANGE_GETS=4 huggingface-cli download --max-workers 1 bartowski/THUDM_GLM-Z1-32B-0414-GGUF --include "*Q4_K_L*"
Fetching 1 files: 0%| | 0/1 [00:00<?, ?it/s]Downloading 'THUDM_GLM-Z1-32B-0414-Q4_K_L.gguf' to '/mnt/e/cache/huggingface/hub/models--bartowski--THUDM_GLM-Z1-32B-0414-GGUF/blobs/aa976549c95417e1418ae8aaa4deb741fbb4b5fb6f525bfae2c430b787ac7357.incomplete' (resume from 20096519115/20371099776)
Xet Storage is enabled for this repo. Downloading file from Xet Storage..
THUDM_GLM-Z1-32B-0414-Q4_K_L.gguf: 41%|████████████████▌ | 8.42G/20.4G [07:56<11:16, 17.7MB/s]
Fetching 1 files: 0%| | 0/1 [07:56<?, ?it/s]
Traceback (most recent call last):
File "/home/aa956/.local/bin/huggingface-cli", line 10, in <module>
sys.exit(main())
~~~~^^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/commands/huggingface_cli.py", line 57, in main
service.run()
~~~~~~~~~~~^^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/commands/download.py", line 153, in run
print(self._download()) # Print path to downloaded files
~~~~~~~~~~~~~~^^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/commands/download.py", line 187, in _download
return snapshot_download(
repo_id=self.repo_id,
...<10 lines>...
max_workers=self.max_workers,
)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/_snapshot_download.py", line 296, in snapshot_download
thread_map(
~~~~~~~~~~^
_inner_hf_hub_download,
^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
tqdm_class=tqdm_class or hf_tqdm,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/tqdm/std.py", line 1181, in __iter__
for obj in iterable:
^^^^^^^^
File "/home/aa956/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/home/aa956/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
~~~~~~~~~~^^^^^^^^^
File "/home/aa956/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/home/aa956/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/aa956/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/_snapshot_download.py", line 270, in _inner_hf_hub_download
return hf_hub_download(
repo_id,
...<15 lines>...
headers=headers,
)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 961, in hf_hub_download
return _hf_hub_download_to_cache_dir(
# Destination
...<14 lines>...
force_download=force_download,
)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 1112, in _hf_hub_download_to_cache_dir
_download_to_tmp_and_move(
~~~~~~~~~~~~~~~~~~~~~~~~~^
incomplete_path=Path(blob_path + ".incomplete"),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<8 lines>...
xet_file_data=xet_file_data,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 1661, in _download_to_tmp_and_move
xet_get(
~~~~~~~^
incomplete_path=incomplete_path,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
displayed_filename=filename,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 580, in xet_get
download_files(
~~~~~~~~~~~~~~^
pointer_files,
^^^^^^^^^^^^^^
...<3 lines>...
progress_updater=[progress_updater],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
RuntimeError: Data processing error: CAS service error : Error : single flight error: Real call failed: CasObjectError(InternalIOError(Custom { kind: Other, error: reqwest::Error { kind: Decode, source: hyper::Error(Body, Custom { kind: UnexpectedEof, error: IncompleteBody }) } }))
aa956@desktop02:~$ |
|
RuntimeError: Data processing error: CAS service error : Error : single flight error: Real call failed: CasObjectError(InternalIOError(Custom { kind: Other, error: reqwest::Error { kind: Decode, source: hyper::Error(Body, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) } }))For the next try I'll remove |
|
Values 2, 4, 8 tried for $ HF_XET_NUM_CONCURRENT_RANGE_GETS=8 huggingface-cli download bartowski/THUDM_GLM-Z1-32B-0414-GGUF --include "*Q4_K_L*"
[62/62]Fetching 1 files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading 'THUDM_GLM-Z1-32B-0414-Q4_K_L.gguf' to '/mnt/e/cache/huggingface/hub/models--bartowski--THUDM_GLM-Z1-32B-0414-GGUF/blobs/aa976549c95417e1418ae8aaa4deb741fbb4$5fb6f525bfae2c430b787ac7357.incomplete' (resume from 20096519115/20371099776)
Xet Storage is enabled for this repo. Downloading file from Xet Storage..
THUDM_GLM-Z1-32B-0414-Q4_K_L.gguf: 42%|██████████████████████████████████████▉ | 8.54G/20.4G [06:07<08:28, 23.3MB/s]
Fetching 1 files: 0%| | 0/1 [06:07<?, ?it/s]
Traceback (most recent call last):
... same location as previous logs ...
RuntimeError: Data processing error: CAS service error : Error : single flight error: Real call failed: CasObjectError(InternalIOError(Custom { kind: Other, error: reqwest::Error { kind: Decode, source: hyper::Error(Body, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) } }))
$ |
|
I used to get this issue when initially testing xet. In my case, I downloaded a model midway with HTTP then swapped to XET. Deleting the cache dir solved it for me. |
|
Cleared caches, reinstalled I see that Still fails but log shows that one of the errors was handled by retrying: $ uv tool uninstall huggingface-hub
$ rm -rf $HF_HOME/hub/.locks/models--bartowski--THUDM_GLM-Z1-32B-0414-GGUF
$ rm -rf $HF_HOME/hub/models--bartowski--THUDM_GLM-Z1-32B-0414-GGUF
$ rm -rf $HF_HOME/xet
$ uv tool install huggingface-hub[hf_xet]
$ HF_DEBUG=1 huggingface-cli download bartowski/THUDM_GLM-Z1-32B-0414-GGUF --include "*Q4_K_L*"
Fetching 1 files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading 'THUDM_GLM-Z1-32B-0414-Q4_K_L.gguf' to '/mnt/e/cache/huggingface/hub/models--bartowski--THUDM_GLM-Z1-32B-0414-GGUF/blobs/3e9358b3f18ee5943ce43069a477cf3bc5ad998a0cf4a02654452bc852d20524.incomplete'
Xet Storage is enabled for this repo. Downloading file from Xet Storage..
THUDM_GLM-Z1-32B-0414-Q4_K_L.gguf: 27%|███ | 5.40G/20.4G [13:09<1:22:36, 3.02MB/s]
"timestamp":"2025-05-01T06:12:24.182669Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://transfer.xethub.hf.co/xorbs/default/c742d154a2b697560c260af6ad28ff30bbbe68ae9e123df6cf310fe2ad6f47d5?X-Xet-Signed-Range=bytes%3D0-67099508&Expires=1746083194&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly90cmFuc2Zlci54ZXRodWIuaGYuY28veG9yYnMvZGVmYXVsdC9jNzQyZDE1NGEyYjY5NzU2MGMyNjBhZjZhZDI4ZmYzMGJiYmU2OGFlOWUxMjNkZjZjZjMxMGZlMmFkNmY0N2Q1P1gtWGV0LVNpZ25lZC1SYW5nZT1ieXRlcyUzRDAtNjcwOTk1MDgiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE3NDYwODMxOTR9fX1dfQ__&Signature=SYMvTTV-u2Uawi~7GtVKa8qL7-v5p40PNqgm-XyePmaF5ZWHWrbRf-R4nA95CvKksAZlhcPRtK4RcPtYJkUZJ3BM2bnDNNWQhhRn5sK93cXdidn1GMhYz7dO1Zl3Zmta-gtOBQH45P49gRDmN7oTqW3A34DWtS13hUzEI~cR9qK3SJM46Ob5Vf2H1VAY8Ehz7EtNcKi7UQMsbbSPy5R6xXiazNDbe6l5HrUxNjt3VbSec52T6qcQhrqMpxu~F8~NP3dhxs5-H1M9WE07u2gG9OSiDLd9zACuI2JoGQwISw8kbdGpnLzNH4M~g~mUW9QwmjtzzzbbAHFb3xb9t-SM8g__&Key-Pair-Id=K2L8F4GPSG1IFC\", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":164}
{"timestamp":"2025-05-01T06:12:24.182694Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 2.440166319s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.6.1/src/middleware.rs","line_number":166}
THUDM_GLM-Z1-32B-0414-Q4_K_L.gguf: 49%|████████▎ | 10.0G/20.4G [22:05<22:50, 7.56MB/s]
Fetching 1 files: 0%| | 0/1 [22:06<?, ?it/s]
Traceback (most recent call last):
File "/home/aa956/.local/bin/huggingface-cli", line 10, in <module>
sys.exit(main())
~~~~^^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/commands/huggingface_cli.py", line 57, in main
service.run()
~~~~~~~~~~~^^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/commands/download.py", line 153, in run
print(self._download()) # Print path to downloaded files
~~~~~~~~~~~~~~^^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/commands/download.py", line 187, in _download
return snapshot_download(
repo_id=self.repo_id,
...<10 lines>...
max_workers=self.max_workers,
)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/_snapshot_download.py", line 296, in snapshot_download
thread_map(
~~~~~~~~~~^
_inner_hf_hub_download,
^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
tqdm_class=tqdm_class or hf_tqdm,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/tqdm/std.py", line 1181, in __iter__
for obj in iterable:
^^^^^^^^
File "/home/aa956/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/home/aa956/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
~~~~~~~~~~^^^^^^^^^
File "/home/aa956/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/home/aa956/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/aa956/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/_snapshot_download.py", line 270, in _inner_hf_hub_download
return hf_hub_download(
repo_id,
...<15 lines>...
headers=headers,
)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 961, in hf_hub_download
return _hf_hub_download_to_cache_dir(
# Destination
...<14 lines>...
force_download=force_download,
)
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 1112, in _hf_hub_download_to_cache_dir
_download_to_tmp_and_move(
~~~~~~~~~~~~~~~~~~~~~~~~~^
incomplete_path=Path(blob_path + ".incomplete"),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<8 lines>...
xet_file_data=xet_file_data,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 1661, in _download_to_tmp_and_move
xet_get(
~~~~~~~^
incomplete_path=incomplete_path,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
displayed_filename=filename,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/aa956/.local/share/uv/tools/huggingface-hub/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 580, in xet_get
download_files(
~~~~~~~~~~~~~~^
pointer_files,
^^^^^^^^^^^^^^
...<3 lines>...
progress_updater=[progress_updater],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
RuntimeError: Data processing error: CAS service error : Error : single flight error: Real call failed: CasObjectError(InternalIOError(Custom { kind: Other, error: reqwest::Error { kind: Decode, source: hyper::Error(Body, Custom { kind: UnexpectedEof, error: IncompleteBody }) } })) |
|
System info for the last log: $ huggingface-cli env
Copy-and-paste the text below in your GitHub issue.
- huggingface_hub version: 0.30.2
- Platform: Linux-6.1.0-34-amd64-x86_64-with-glibc2.36
- Python version: 3.13.1
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /mnt/e/cache/huggingface/token
- Has saved token ?: True
- Who am I ?: **redacted**
- Configured git credential helpers:
- FastAI: N/A
- Tensorflow: N/A
- Torch: N/A
- Jinja2: N/A
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: N/A
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: N/A
- pydantic: N/A
- aiohttp: N/A
- hf_xet: 1.1.0
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /mnt/e/cache/huggingface/hub
- HF_ASSETS_CACHE: /mnt/e/cache/huggingface/assets
- HF_TOKEN_PATH: /mnt/e/cache/huggingface/token
- HF_STORED_TOKENS_PATH: /mnt/e/cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Tried the hf_xet today for the first time:
Notes:
Reproduction
Logs
System info
The text was updated successfully, but these errors were encountered: