AI Local Plugin

A local, privacy-first AI model runner with gRPC and REST APIs.

---

Prerequisites

• Python 3.11+ (Download)
• Docker Desktop (Download)
• curl (Download)
• grpcurl (Download)

---

Quick Installation (Windows)

1. Clone the repository:

   git clone https://github.com/Egregore-co/ai-local-plugin.git
   cd ai-local-plugin

2. Run the installer:

   install.bat

---

Manual Installation

1. Create and activate a virtual environment:

   python -m venv .venv
   .venv\Scripts\activate

2. Install dependencies:

   pip install -r requirements.txt

3. Start the server:

   python -m src.main

---

Usage Examples

Testing with Postman

If you encounter issues with PowerShell or curl on Windows (such as JSON decode errors), you can use Postman to test the API endpoints easily:

How to test endpoints with Postman:

1. Completion (/v1/completions)

1. Open Postman and click New > HTTP Request.
2. Set the request type to POST.
3. Enter the URL: http://localhost:8000/v1/completions
4. Go to the Headers tab and add:
   • Key: Content-Type Value: application/json
5. Go to the Body tab, select raw, and choose JSON from the dropdown.
6. Paste the following JSON:

   {
     "model": "llama2-7b-chat",
     "prompt": "Hello"
   }

7. Click Send.
8. You should receive a JSON response with a completion result or a mock response if the model is not running.

2. Chat Completion (/v1/chat/completions)

1. Open Postman and click New > HTTP Request.
2. Set the request type to POST.
3. Enter the URL: http://localhost:8000/v1/chat/completions
4. Go to the Headers tab and add:
   • Key: Content-Type Value: application/json
5. Go to the Body tab, select raw, and choose JSON from the dropdown.
6. Paste the following JSON:

   {
     "model": "llama2-7b-chat",
     "messages": [
       {"role": "user", "content": "Hi"}
     ]
   }

7. Click Send.
8. You should receive a JSON response with a chat completion result or a mock response if the model is not running.

3. List Models (/v1/models)

1. Open Postman and click New > HTTP Request.
2. Set the request type to GET.
3. Enter the URL: http://localhost:8000/v1/models
4. Click Send.
5. You should receive a JSON response listing all available models.

Note: If you see errors about JSON decoding or content length when using PowerShell, this is a shell quoting issue. Postman avoids these problems.

1. List Models (REST)

Invoke-RestMethod -Uri "http://localhost:8000/v1/models" -Method Get

2. Chat Completion (REST, OpenAI-compatible)

Invoke-RestMethod -Uri "http://localhost:8000/v1/chat/completions" -Method Post -ContentType "application/json" -Body '{"model": "llama2-7b-chat", "messages": [{"role": "user", "content": "Hi"}]}'

3. Completion (REST, OpenAI-compatible)

Invoke-RestMethod -Uri "http://localhost:8000/v1/completions" -Method Post -ContentType "application/json" -Body '{"model": "llama2-7b-chat", "prompt": "Hello"}'

4. List Models (gRPC)

grpcurl -plaintext -d '{}' localhost:50051 ModelService/ListModels

5. Install a Model (gRPC)

grpcurl -plaintext -d '{\"item_id\": \"llama2-7b-chat\", \"version\": \"latest\"}' localhost:50051 install_service.InstallService/Install

---

Troubleshooting

• Unicode errors: Run PowerShell as Administrator and set $env:PYTHONIOENCODING="utf-8"
• Docker not found: Ensure Docker Desktop is running
• Port in use: Change the port in src/main.py or stop the conflicting process
• Model not found: Check model name and installation status

---

Development

• Run tests: pytest
• Lint: black . && isort . && mypy src/
• Coverage: pytest --cov=src

---

License

MIT# ai-local-plugin