Trained model fails to setup with cog.server.exceptions.FatalWorkerException

[makkor-git]

I tried training a model as suggested in the README to add custom HF weights, and while training succeded and produced a model, the model itself fails to setup with the following exception:

Downloading user weights from: <URLFile at 0x7d1da7a00720 for 'https://replicate.delivery/xezq/[REDACTED]/weights.tar'>
⏳ Downloading weights.tar to downloaded_user_models
Traceback (most recent call last):
File "/root/.pyenv/versions/3.12.6/lib/python3.12/site-packages/cog/server/worker.py", line 717, in _handle_setup_error
yield
File "/root/.pyenv/versions/3.12.6/lib/python3.12/site-packages/cog/server/worker.py", line 533, in _setup
self._predictor.setup(weights=weights)  # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/predict.py", line 37, in setup
self.handle_user_weights(weights)
File "/src/predict.py", line 44, in handle_user_weights
WeightsDownloader.download("weights.tar", weights, config["USER_WEIGHTS_PATH"])
File "/src/weights_downloader.py", line 76, in download
subprocess.check_call(
File "/root/.pyenv/versions/3.12.6/lib/python3.12/subprocess.py", line 408, in check_call
retcode = call(*popenargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.6/lib/python3.12/subprocess.py", line 389, in call
with Popen(*popenargs, **kwargs) as p:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.6/lib/python3.12/subprocess.py", line 1026, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/root/.pyenv/versions/3.12.6/lib/python3.12/subprocess.py", line 1885, in _execute_child
self.pid = _fork_exec(
^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not URLFile
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.12.6/lib/python3.12/site-packages/cog/server/runner.py", line 257, in _handle_done
    f.result()
  File "/root/.pyenv/versions/3.12.6/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/3.12.6/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
cog.server.exceptions.FatalWorkerException: Predictor errored during setup: expected str, bytes or os.PathLike object, not URLFile

[gokhancelikkaya]

is there any solution to this. I trained a new model with including a custom model from Civit.ai but it does not start and my trained version gets disabled automatically.

[makkor-git]

is there any solution to this. I trained a new model with including a custom model from Civit.ai but it does not start and my trained version gets disabled automatically.

I made a quick workaround in predict.py (in handle_user_weights method, lines 44-45):

# If weights is a URLFile, extract the URL string
if hasattr(weights, "url"):
    weights_url = "https://replicate.delivery/" + weights.url
else:
    weights_url = weights

print(f"Downloading user weights from: {weights_url}")
WeightsDownloader.download("weights.tar", weights_url, config["USER_WEIGHTS_PATH"])

It works for me, can't guarantee how much of a real solution it is, but still.

[gokhancelikkaya]

thanks for sharing this fix which seems valid. But could you please also tell me how I can use the code change for the replicate model https://replicate.com/fofr/any-comfyui-workflow/train

[makkor-git]

thanks for sharing this fix which seems valid. But could you please also tell me how I can use the code change for the replicate model https://replicate.com/fofr/any-comfyui-workflow/train

The only way to do that is to fork this repo, add this fix there and make your own custom Replicate model from it as described in "Create and deploy a fork using Cog" section of the README.

[fofr]

I've added this fix in e7f2cb7
I'm still investigating why this regressed, my guess is some upstream cog changes