Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[draft] add llamafile 🦙📁 #866

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

[draft] add llamafile 🦙📁 #866

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3fa5005
add gguf snippet for llamafile
not-lain Aug 26, 2024
321cb48
update comment
not-lain Aug 26, 2024
1ca85cd
update code snippet for 🦙📁
not-lain Aug 26, 2024
deaf038
Merge branch 'main' into llamafile
Vaibhavs10 Aug 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions packages/tasks/src/local-apps.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,29 @@ const snippetLlamacpp = (model: ModelData, filepath?: string): LocalAppSnippet[]
];
};

const snippetLlamafileGGUF = (model: ModelData, filepath?: string): LocalAppSnippet[] => {
const command = (binary: string) =>
[
"# Load and run the model :",
`wget https://huggingface.co/${model.id}/resolve/main/`.concat(`${filepath ?? "{{GGUF_FILE}}"}`), // could not figure out how to do it without concat
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`wget https://huggingface.co/${model.id}/resolve/main/`.concat(`${filepath ?? "{{GGUF_FILE}}"}`), // could not figure out how to do it without concat
`wget https://huggingface.co/${model.id}/resolve/main/${filepath ?? '{{GGUF_FILE}}'}`,

Would something like that work? (not sure)

`chmod +x ${binary}`,
`${binary} -m ${filepath?? "{{GGUF_FILE}}"} -p 'You are a helpful assistant' `, // will this create a second dropdown ?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`${binary} -m ${filepath?? "{{GGUF_FILE}}"} -p 'You are a helpful assistant' `, // will this create a second dropdown ?
`${binary} -m ${filepath ?? "{{GGUF_FILE}}"} -p 'You are a helpful assistant'`, // will this create a second dropdown ?

].join("\n");
return [
{
title: "Use pre-built binary",
Copy link
Member

@ngxson ngxson Aug 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this step is correct in llamafile, because the downloaded model file is already a pre-built binary.

The release binary from https://github.com/Mozilla-Ocho/llamafile/releases is used in Creating llamafiles section of the README. So this step should be named Create your own llamafile (or we could also remove this section, and just leave a link to README)

setup: [
// prettier-ignore
"# Download pre-built binary from:",
"# https://github.com/Mozilla-Ocho/llamafile/releases",
"# Example : ",
"# wget https://github.com/Mozilla-Ocho/llamafile/releases/download/0.8.13/llamafile-0.8.13",
].join("\n"),
content: command("./llamafile-0.8.13")
}
];
};

const snippetLocalAI = (model: ModelData, filepath?: string): LocalAppSnippet[] => {
const command = (binary: string) =>
["# Load and run the model:", `${binary} huggingface://${model.id}/${filepath ?? "{{GGUF_FILE}}"}`].join("\n");
Expand Down Expand Up @@ -146,6 +169,15 @@ export const LOCAL_APPS = {
displayOnModelPage: isLlamaCppGgufModel,
snippet: snippetLlamacpp,
},
// llamafile uses .llamafile and .gguf files
// update this later to handle .llamafile
llamafile : {
prettyLabel : "llamafile",
docsUrl : "https://github.com/Mozilla-Ocho/llamafile",
mainTask : "text-generation",
displayOnModelPage : isLlamaCppGgufModel, // update this later to include .llamafile
snippet: snippetLlamafileGGUF ,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
snippet: snippetLlamafileGGUF ,
snippet: snippetLlamafileGGUF,

},
lmstudio: {
prettyLabel: "LM Studio",
docsUrl: "https://lmstudio.ai",
Expand Down
Loading