-
Notifications
You must be signed in to change notification settings - Fork 5
Closed
Description
The folks at @menloresearch have recently updated their builds to include CUDA and Vulkan with common CPU instruction sets. Along with Metal builds, this means that all common GPUs are now supported with optimized CPU offload on Windows and Linux.
It may be helpful to target these binaries rather than upstream llama.cpp, as they are well-tested and widely deployed in Jan. The inclusion of the GGML_CPU_ALL_VARIANTS=ON flag means that the GPU and non-GPU builds are both available with optimizations for most common CPU types.
The current Menlo release of LlamaCPP, and the first to include these optimized builds, is b6765. The new Windows and Linux builds are those with common_cpus in the filenames.
Metadata
Metadata
Assignees
Labels
No labels