Local Inference (Ollama) and Offline Functionality Not Working as Expected

[0utKast]

We attempted to set up Hugging Face AI Sheets to perform inference using local Large Language Models (LLMs) via Ollama
(specifically llama3 and phi3:mini), aiming for data privacy and potential offline use. However, our extensive testing
revealed that the application does not behave as expected based on the CUSTOMIZATION.md documentation.

Observed Behavior:

1. Hard Dependency on Internet Connection (Offline Failure):
2. Inference Not Routed to Local Ollama (Even When Online):
3. UI Misleading/Problematic:
4. HF_TOKEN Requirement:
5. Clone the AI Sheets repository: git clone https://github.com/huggingface/aisheets.git
   HF_TOKEN="hf_YOUR_VALID_HUGGINGFACE_TOKEN" MODEL_ENDPOINT_URL="http://localhost:11434" MODEL_ENDPOINT_NAME="llama3" # Or "phi3:mini" for testing the other model
6. Build the application: pnpm run build
7. Start the server: pnpm serve (ensure internet connection is active for this step).
8. In the AI Sheets UI, load a CSV file (e.g., reviews.csv with texto_resena in column2).
9. Add a new column using a prompt (e.g., sentiment analysis on {{column2}}).
10. Observe: Check the ollama serve terminal. No new log entries will appear, indicating no request reached the local server.
11. Test Offline: Disconnect from the internet. Restart pnpm serve. The application will fail to load with an ENOTFOUND
    huggingface.co error.

Expected Behavior:

• The application should be able to load and function without an internet connection when configured for local LLMs.
• Misleading documentation regarding local/offline capabilities.
• Operating System: Windows (win32)

[frascuchon]

Hi @0utKast,

The latest changes to improve the model selection removed the custom model selection, ignoring the custom setup when generating cells. We'll bring this back as soon as possible.

Meanwhile, you can try using the stable branch:

git fetch origin
git checkout stable
pnpm i
pnpm build
pnpm serve

cc @damianpumar, @Amelie-V

[GeekLad]

I tried running the stable branch, but I get a 500 error when trying to run without an Internet connection due to failure to connect to huggingface.co

[frascuchon]

Hey @0utKast, @GeekLad

I've been working on this, and custom models and offline mode should work now. Pick the latest changes from main and give it a try.

Note that with offline mode, I needed to define the model endpoint URL as http://127.0.0.1... instead of http://localhost... using the ollama server. Just in case you have problems.