How to ensure Khoj is using GPU on device

[mzeidhassan]

Hi team,
I am using "khoj_0.10.1_amd64.exe" at the moment, but it takes a long time to get responses during chat.
Is there a way to ensure that Khoj is utilizing my GPU to speed up things a bit?

Thanks

[1am]

It would be great to see how to increase. Using Khoj on i7 32GB Ram offline with llama-2-7b-chat.ggmlv3.q4_K_S.bin and it takes >200sec to get a response which could probably reduced. Still super happy about Khoj though :)

[sammcj]

I think Khoj might actually not support GPUs at all looking at #388 and #390, it seems the project is waiting on this upstream PR: nomic-ai/gpt4all#746

[Hunanbean-Collective]

that is unfortunate, considering LocalGPT works on CUDA already

[sabaimran]

Hey all, thanks a lot for the supporting details/documentation. And sorry about the slow load times on your personal devices 🤕.

Just confirming a few things here:

1. As far as GPU support, we have verified that this works with MPS (Apple Mac devices), but it seems like it's not working for CUDA. We are indeed hoping to get support via GPT4All, but may need to look into alternatives if this doesn't get unlocked soon. (Added cuda and opencl support nomic-ai/gpt4all#746), (Ability to use GPU in GUI version? nomic-ai/gpt4all#508).
2. Khoj with the GUI mode does not use the GPU for offline chat. I haven't been able to figure out why this is. We're getting rid of the GUI mode, so I probably won't look into this much further.

If it helps at all, we're probably going to host an instance of Llama 70B. I know it breaks the secure relationship on your local network, but it might help for the near term as improvements are made for GPU-support in the libraries we use.

[Hunanbean-Collective]

That makes it all pretty pointless.

[debanjum]

That makes it all pretty pointless.

@Hunanbean-Collective please keep your comments constructive (and nice)

[Hunanbean-Collective]

That is meant as constructive. the point of use of the project is to keep our private databases private.. unless i misunderstand, this is the exact opposite of that.

…

On Fri, Oct 6, 2023 at 11:49 AM Debanjum ***@***.***> wrote: That makes it all pretty pointless. @Hunanbean-Collective <https://github.com/Hunanbean-Collective> please keep your comments constructive (and nice) — Reply to this email directly, view it on GitHub <#418 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARD3WIIBNH32C5WKFRR2PH3X6BHDHAVCNFSM6AAAAAA3EPM7XGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJRGI3DMNBRGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[debanjum]

Thanks for clarifying your expectations.

1. Khoj is meant to be a great open-source, personal AI copilot
2. Khoj being completely private is one of the options it supports but it is not the only one. For example, Khoj chat works with OpenAI's Chat model. The Khoj inference server will expose more powerful open-source models that cannot run on consumer hardware like the Llama 2 70B
3. Once the underlying dependencies gain GPU support for offline chat models, we'll add support for it in Khoj as well. See Improve Offline Chat Model Experience #494

[debanjum]

Closing this issue as Khoj uses GPU where GPU support is available by the underlying library used (GPT4All). This currently includes MacOS and now Vulcan supported GPU + OS with #494