Seeing threading errors since last pull

[Koneko349]

Seeing these errors since the fixes yesterday.

Exception in thread Thread-159:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/mnt/data/AI/dream-factory/scripts/sdi.py", line 98, in run
[cuda:0] >>> received hypernetwork query response: SD indicates 0 hypernetworks available for use...
self.callback(response)
File "/mnt/data/AI/dream-factory/scripts/sdi.py", line 515, in sampler_response
[cuda:0] >>> received model query response: SD indicates 0 models available for use...
samplers.append(i['name'])
TypeError: string indices must be integers
[cuda:0] >>> received LoRA query response: SD indicates 0 LoRAs available for use...
Exception in thread Thread-163:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/mnt/data/AI/dream-factory/scripts/sdi.py", line 158, in run
self.callback(response)
File "/mnt/data/AI/dream-factory/scripts/sdi.py", line 678, in script_response
Exception in thread Thread-164:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
for i in r['txt2img']:
KeyError: 'txt2img'
self.run()
File "/mnt/data/AI/dream-factory/scripts/sdi.py", line 170, in run
self.callback(response)
File "/mnt/data/AI/dream-factory/scripts/sdi.py", line 696, in upscaler_response
upscalers.append(i['name'])
TypeError: string indices must be integers

[Koneko349]

I should also note that the errors are intermittent

[rbbrdckybk]

It looks like something happened with your Auto1111 instance where the initial API calls to fetch models, LoRAs, etc all failed. I haven't seen that happen on my end ever so I'm not sure what might cause it. I'm assuming if you restart everything it should work properly.

[Koneko349]

It seems to persist through restarts, I can try to reinstall Auto1111, what gets me is that its intermittent. Loading Auto1111 itself seems to work just fine. I will reinstall and let you know.

[Koneko349]

reloaded auto1111 and pulled the new version of dream factory, but the issue persists. I will continue to try various fixes and report back if i am able to resolve the issue

[Koneko349]

Doing a little digging on this... updated some code

# handle server sampler response
def sampler_response(self, response):
    try:
        r = response.json()
    except ValueError as e:
        self.log(f'Error decoding JSON: {e}')
        return

    samplers = []
    sampler_str = ''
    if isinstance(r, list):  # Add this check
        for i in r:
            try:  # Add a try/except block to handle dictionaries without 'name'
                samplers.append(i['name'])
                sampler_str += '   - ' + i['name'] + '\n'
            except TypeError:
                self.log(f'Error: expected a dict, got {i}')
            except KeyError:
                self.log(f'Error: dict does not have "name" key: {i}')
    else:
        self.log(f'Error: expected a list, got {r}')

    self.busy = False

and now seeing this error in the startup logging, [cuda:0] >>> Error: expected a list, got {'detail': 'Not Found'}

[Koneko349]

Narrowing this down, it looks like sometimes (often) it takes longer than what is expected for the /sdapi/v1/samplers API and I assume others as well, to be ready. I can curl the endpoint as dream factory is starting and get the detail: not found response but a few minutes later the endpoint will begin to return correctly. I am going to guess this is due to my models being on a NAS and having to load before the API begins to respond.

[rbbrdckybk]

That makes sense. If you want a short-term fix, you can open scripts/sdi.py and add this right before line 276:

time.sleep(5)

That will make DF wait 5 seconds before proceeding after the Auto1111 API first becomes available. Increase the time if 5 secs isn't enough. Leave this issue open and I'll get a config.txt option for a startup delay added as soon as I can.

[Koneko349]

I think i rigged up a wait and retry for the server initialization. I will make a pull request for you to review

[Koneko349]

Looks like other APIs may finish getting ready after the samplers API does, but the code fix looks to have reduced the error rate significantly. The other api that caused it to fail was sd-models.