Skip to content

Idea: Target Jan's llama.cpp build #20

@iwr-redmond

Description

@iwr-redmond

The folks at @menloresearch have recently updated their builds to include CUDA and Vulkan with common CPU instruction sets. Along with Metal builds, this means that all common GPUs are now supported with optimized CPU offload on Windows and Linux.

It may be helpful to target these binaries rather than upstream llama.cpp, as they are well-tested and widely deployed in Jan. The inclusion of the GGML_CPU_ALL_VARIANTS=ON flag means that the GPU and non-GPU builds are both available with optimizations for most common CPU types.

The current Menlo release of LlamaCPP, and the first to include these optimized builds, is b6765. The new Windows and Linux builds are those with common_cpus in the filenames.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions