Convention for gguf model location?

[mofosyne]

Does it make sense to have a convention for model locations for gguf files?

(Idea originally from Mozilla-Ocho/llamafile#178 but adapted for GGUF context)

Installing A GGUF And Making It Accessible To Other Local Applications

For optimal discovery and integration with local application scripts/programs, we recommend the following search paths:

• System-wide Paths:

  • /usr/share/gguf (Linux/MacOS/BSD): Ideal for developers creating packages, commonly accessed via package managers like apt get install in Debian-based Linux OSes.
  • /opt/gguf (Linux/MacOS/BSD): Positioned in the /opt directory, suitable for installers downloaded directly from the web.
  • C:\gguf (Windows): A direct path for Windows systems.

• User-specific Path:

  • ~/.gguf (Linux/MacOS/BSD): Located in the user's home directory, facilitating user-specific configurations in line with Unix-like conventions.

For applications or scripts referencing the GGUF path, setting the environment variable $GGUF_PATH to a singular path can enhance configuration simplicity and system consistency.

Python Example

For your convenience, here is a python example using the above path conventions

#!/usr/bin/python3
import os

def search_gguf_files():
    search_paths = [
        os.getenv("GGUF_PATH"),        # User-defined model repository location (cross-platform)
        os.path.expanduser("~/.gguf"), # User-specific models (Linux/MacOS/BSD)
        "/usr/share/gguf",             # System-wide location for package manager installed models (Linux/MacOS/BSD)
        "/opt/gguf",                   # System-wide location for manually downloaded models (Linux/MacOS/BSD)
        "C:\\gguf"                     # System-wide location for manually downloaded models (Windows)
    ]

    return [
        os.path.join(root, name)
        for path in search_paths if path and os.path.exists(path)
        for root, _, files in os.walk(path)
        for name in files
    ]

# Get all valid GGUF paths and their files
print(f"GGUF paths: {search_gguf_files()}")

---

edit: lowercase gguf path name as per @teleprint-me 's recommendation

[teleprint-me]

Typically, path names are lowercase in unix-like systems because they're case sensitive. This is just a nitpick though.

[mofosyne]

Found that there is already people using $GGUF_PATH as environmental variable in this thing called hyperdemocracy (unsure what this repo is about but it's definitely an AI based project with reference to langchain).

• https://github.com/QuanMPhm/hyperdemocracy/blob/65a4bb4cdaa76e808e3e5b580a69a8d8c73313f9/legisqa/legisqa/streamlit_app.py#L25 - hyperdemocracy
• https://github.com/andrewshaodev/hyperdemocracy/blob/a2ed51adaa12faf0156ab0360810b613e716a42e/legisqa/legisqa/streamlit_app.py#L27 - hyperdemocracy

env_gguf_path = os.getenv("GGUF_PATH")

So at least that's one datapoint that's it's already a notation that in use that can be formalised.

[ggerganov]

Does it make sense to have a convention for model locations for gguf files?

Don't think there is need to introduce default location for a file format. llama.cpp for example is a project that uses GGUF files and can have it's own default location to store model files (ggml-org/llama.cpp#7252). Another project that also uses GGUF files (not necessarily LLM) could have a different location than llama.cpp

[mofosyne]

Ah i see. That's fair enough, I guess this is good enough as an ad hoc thing then. Well at least this is here if anyone searches for it.