Problems running llama.cpp vulkan (linux) on LM Studio along with LLM greater than 38GB in size or about 55B parameters and higher

[erickwill]

The latest updates made to LM Studio binaries, llama.cpp vulkan runtime, is not loading properly LLM as large as 50B/70B parameters (about 35~40GB) on AMD RX580 GPU. Leading to the message:

Previous models ran perfectly.

Here are a few logs when it was working properly and when it stop to work:
Vulkan while working: vulkan-working.txt

Recent vulkan not working: vulkan-not-working.txt

Attached the llama.cpp vulkan working perfectly:
llama.cpp-linux-x86_64-vulkan-avx2-1.4.0.zip

I already discussed with LM Studio devs, that suspect that the issue is related to llama.cpp vulkan. Could you please help? Any advice is appreciated.

[0cc4m]

Can you elaborate on how it is failing? I don't see any immediate issue in the logs you provided.

[erickwill]

Thanks, appreciated that! I'll test it out

[erickwill]

It's basically just cloning this repo and running:

cmake -B build -DGGML_VULKAN=1 -DCMAKE_BUILD_TYPE=Release .
cmake --build build -j

Afterwards you can find the example binaries in build/bin/. You can try llama-cli with your model.

I did it and it did work perfectly. Still, LM Studio compiled runtime, based in Vulkan Llama.cpp should work too and the recent ones, does not. I'm running out of ideas. Do you have any suggestion on what could be wrong on LM Studio runtime? Because the log shows a potential stack overflow on the runtime that does not work, but it does not happen using the one compiled from the source.
Thank you very much for your help.

[0cc4m]

Sadly, without knowing what version they are using and how they are using it, we can only speculate.

Can you show me what you mean by "potential stack overflow"? I still don't see anything unexpected in the log.

[erickwill]

Sadly, without knowing what version they are using and how they are using it, we can only speculate.

Can you show me what you mean by "potential stack overflow"? I still don't see anything unexpected in the log.

Yes, there is a stack overflow, but not in llama.cpp. I was pointed to this: lmstudio-ai/lmstudio-bug-tracker#285
"The issue is related to limitations with chromium's memory allocator. We are tracking the issue here lmstudio-ai/lmstudio-bug-tracker#285"
Hopefully a fix will be implemented anytime soon.
Thanks for your support