Custom CUDA Image for GPT4All GPU and CPU Support #1721
Closed
dpsalvatierra
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I went down the rabbit hole on trying to find ways to fully leverage the capabilities of GPT4All, specifically in terms of GPU via FastAPI/API. At this time, we only have CPU support using the tiangolo/uvicorn-gunicorn:python3.11 image and huggingface TGI image which really isn't using gpt4all.
While working with the Nvidia CUDA image, I encountered several limitations due to outdated components and lack of GLIBCXX_3.4.29 support, since the maintainer has stopped building the CudaGL images since last year - reason? on hold to improve CI/CD system : https://gitlab.com/nvidia/container-images/cudagl
To address this, I've taken the initiative to build a custom image with Nomic Vulkan support. The image is built on CUDA version 12.3.1 with Ubuntu 22.04, specifically tailored for x86_64 architecture with CUDA GL support.
I've temporarily hosted this image on my Docker Hub (dsalvat1/cudagl) for testing and review. I would greatly appreciate your feedback on this solution and any suggestions for improvement.
Additionally, we should discuss the long-term plan for maintaining this image. Whether it remains hosted on my Docker Hub, wait for the maintainer to optimize the image or we transition to an official repository, I'm open to suggestions and willing to assist in its upkeep.
Finally, I am also including my branch which has the updated docker compose file and other improvements such as enabling streaming for Chat Completion.
Guess the ask is to use the branch I created called "fastapi-dev" and then merge once the maintainers are satisfied.
Todos:
https://github.com/dpsalvatierra/gpt4all/tree/fastapi-dev
Beta Was this translation helpful? Give feedback.
All reactions