-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Description
Is there an existing issue for this problem?
- I have searched the existing issues
Install method
Invoke's Launcher
Operating system
Linux
GPU vendor
Nvidia (CUDA)
GPU model
RTX 4070
GPU VRAM
12GB
Version number
v6.10.0rc1
Browser
No response
System Information
No response
What happened
Lost connection for about two hours yesterday - iSP issues. Attempted to generate images with ZImage-Turbo models and got the next error:
[2025-12-28 12:01:50,052]::[InvokeAI]::INFO --> Executing queue item 8, session 1a506751-93c2-426d-b9f5-a18c8541f21c
[2025-12-28 12:02:02,915]::[Qwen3EncoderGGUFLoader]::INFO --> Detected llama.cpp GGUF format, converting keys to PyTorch format
[2025-12-28 12:02:02,922]::[Qwen3EncoderGGUFLoader]::INFO --> Qwen3 GGUF Encoder config detected: layers=36, hidden=2560, heads=32, kv_heads=8, intermediate=9728, head_dim=128
[2025-12-28 12:02:04,022]::[Qwen3EncoderGGUFLoader]::INFO --> Dequantized embed_tokens weight for embedding lookups
[2025-12-28 12:02:04,022]::[Qwen3EncoderGGUFLoader]::INFO --> Tied lm_head.weight to embed_tokens.weight (GGUF tied embeddings)
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 001cb3c6-3b74-4a6d-b9e7-7e7f5d55b83b)')' thrown while requesting HEAD https://huggingface.co/Qwen/Qwen3-4B/resolve/main/tokenizer_config.json
Retrying in 1s [Retry 1/5].
'(MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Qwen/Qwen3-4B/resolve/main/tokenizer_config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f02b01bb1a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: ce6972a5-2a09-4198-b06b-6e0e3a62bc79)')' thrown while requesting HEAD https://huggingface.co/Qwen/Qwen3-4B/resolve/main/tokenizer_config.json
Retrying in 2s [Retry 2/5].
'(MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Qwen/Qwen3-4B/resolve/main/tokenizer_config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0294323c80>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: b1d05fec-2a7c-4efc-8801-62a662d2b9f1)')' thrown while requesting HEAD https://huggingface.co/Qwen/Qwen3-4B/resolve/main/tokenizer_config.json
Retrying in 4s [Retry 3/5].
'(MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Qwen/Qwen3-4B/resolve/main/tokenizer_config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f019b832600>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: 3ea263cb-974a-47af-9b0a-74c9ab276341)')' thrown while requesting HEAD https://huggingface.co/Qwen/Qwen3-4B/resolve/main/tokenizer_config.json
Retrying in 8s [Retry 4/5].
'(MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Qwen/Qwen3-4B/resolve/main/tokenizer_config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f019b830b90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: 9c9a8af7-973e-4854-8f2c-5c76234f7d51)')' thrown while requesting HEAD https://huggingface.co/Qwen/Qwen3-4B/resolve/main/tokenizer_config.json
Retrying in 8s [Retry 5/5].
'(MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Qwen/Qwen3-4B/resolve/main/tokenizer_config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f019b831010>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: edd13311-2a77-421f-b306-2d78ddd89c14)')' thrown while requesting HEAD https://huggingface.co/Qwen/Qwen3-4B/resolve/main/tokenizer_config.json
[2025-12-28 12:02:37,366]::[InvokeAI]::ERROR --> Error while invoking session 1a506751-93c2-426d-b9f5-a18c8541f21c, invocation 3f63be96-d98d-40fb-af35-add46f0f0950 (z_image_text_encoder): (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/Qwen/Qwen3-4B (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0294610560>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: 734302ca-eb97-485e-b032-a5692be2782b)')
[2025-12-28 12:02:37,366]::[InvokeAI]::ERROR --> Traceback (most recent call last):
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 72, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/lib/python3.12/socket.py", line 978, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f0294610560>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/Qwen/Qwen3-4B (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0294610560>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 130, in run_node
output = invocation.invoke_internal(context=context, services=self._services)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/invokeai/app/invocations/baseinvocation.py", line 244, in invoke_internal
output = self.invoke(context)
^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/invokeai/app/invocations/z_image_text_encoder.py", line 59, in invoke
prompt_embeds = self._encode_prompt(context, max_seq_len=Z_IMAGE_MAX_SEQ_LEN)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/invokeai/app/invocations/z_image_text_encoder.py", line 75, in _encode_prompt
tokenizer_info = context.models.load(self.qwen3_encoder.tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/invokeai/app/services/shared/invocation_context.py", line 392, in load
return self._services.model_manager.load.load_model(model, submodel_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/invokeai/app/services/model_load/model_load_default.py", line 71, in load_model
).load_model(model_config, submodel_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/invokeai/backend/model_manager/load/load_default.py", line 57, in load_model
cache_record = self._load_and_cache(model_config, submodel_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/invokeai/backend/model_manager/load/load_default.py", line 78, in _load_and_cache
loaded_model = self._load_model(config, submodel_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/invokeai/backend/model_manager/load/model_loaders/z_image.py", line 689, in _load_model
return AutoTokenizer.from_pretrained(self.DEFAULT_TOKENIZER_SOURCE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 1156, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2395, in _from_pretrained
tokenizer = cls._patch_mistral_regex(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2438, in _patch_mistral_regex
if _is_local or is_base_mistral(pretrained_model_name_or_path):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2432, in is_base_mistral
model = model_info(model_id)
^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2660, in model_info
r = get_session().get(path, headers=headers, timeout=timeout, params=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 95, in send
return super().send(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/user/Invoke/.venv/lib/python3.12/site-packages/requests/adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/Qwen/Qwen3-4B (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0294610560>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: 734302ca-eb97-485e-b032-a5692be2782b)')
[2025-12-28 12:02:37,399]::[InvokeAI]::INFO --> Graph stats: 1a506751-93c2-426d-b9f5-a18c8541f21c
Node Calls Seconds VRAM Used
string 1 0.001s 6.445G
integer 1 0.001s 6.445G
core_metadata 1 0.001s 6.445G
z_image_model_loader 1 0.001s 6.445G
z_image_text_encoder 1 47.287s 6.445G
TOTAL GRAPH EXECUTION TIME: 47.290s
TOTAL GRAPH WALL TIME: 47.292s
RAM used by InvokeAI process: 5.73G (-9.414G)
RAM used to load models: 4.23G
VRAM in use: 0.172G
RAM cache statistics:
Model cache hits: 1
Model cache misses: 2
Models cached: 2
Models cleared from cache: 2
Cache high water mark: 4.38/0.00G
What you expected to happen
ZImage-Turbo models to generate images without trying to connect to huggingface.co each time a prompt is changed or modified.
How to reproduce the problem
Load a ZImage-Turbo model and generate an image. Disconnect from the internet, modify the prompt and attempt to generate another image.
Additional context
Only happens when using ZImage-Turbo models, Flux models can gen offline.
Discord username
No response