Add Ollama as a supported provider

[ldmosquera]

As mentioned in issue #2, Ollama support is valuable for users interested in self-hosted/offline inference.

This PR adds initial support for Ollama including chat completions, streaming and embeddings. No tool support yet, needs further investigation.

PR is a rough draft and will likely take some back and forth to get it merge-ready; more comments to follow.

Closes #2

[ldmosquera]

This was done copying and adapting from the existing Gemini provider. Tool usage and media was excised for now; I still need to research what Ollama offers and what overlaps within this project. Also, capabilities.rb is full of placeholder stuff until we figure out sane defaults since Ollama allows running arbitrary models instead of a well known list.

---

Commit 9b387c1 disables all providers by default in models.refresh! unless they are explicitly configured.

The intention is to be able to work on Ollama (or any other specific single provider) without being forced to provide valid keys for EVERY provider, also incurring live calls. In particular, because Ollama doesn't come with any default models and so models.refresh! is mandatory before any usage to populate available models on the server.

As mentioned in the commit message, this might be too intrusive for the intended usage of this project, so an alternative is to only do this when given a specific env var that puts ruby-llm into such a "default offline" mode.

This might interfere with tests and/or the models update rake task, none of which I touched yet since they also require valid keys for all providers.

---

As for tests, I'd like some guidance into how to implement unit testing and eventually integration testing specifically for Ollama, in a way that does not require configuring and using all APIs (and attending cost).

Since Ollama does not come with default models, I suggest a separate test suite that first ensures that a tiny model is downloaded into the Ollama server via its API.

[jm3]

was just about to file an issue for this — awesome!

[crmne]

Two new features to consider in your provider implementation:

1. Model Aliases: Please add entries for your provider in aliases.json:

   "claude-3-5-sonnet": {
     "anthropic": "claude-3-5-sonnet-20241022",
     "your-provider": "your-provider-specific-id"
   }

2. Provider Selection: Users will be able to specify your provider:

   chat = RubyLLM.chat(model: 'claude-3-5-sonnet', provider: 'your-provider')

Docs: https://rubyllm.com/guides/models#using-model-aliases

[ldmosquera]

In general, I understand this can be used to specify a provider when the model is served by more than one provider.

But for that, we have to know for sure that the provider supports that model, and yet in the case of Ollama, there are no guarantees that a given server will have any models installed, thus we can't hardcode any static assumptions into the gem. Rather, the list of models would only be available after querying the server at runtime.

I realize this deviates from the general pattern of the gem which aims to provide a good out of the box experience for well known API providers, but by definition Ollama is not that. It can be used with arbitrary models even beyond Ollama's official library - you can pull from HuggingFace or even create your own models with arbitrary names.

I still think supporting Ollama within reasonable limits is a valuable addition to the gem; we just have to figure out those limits.

[ldmosquera]

Commit 9b387c1 disables all providers by default in models.refresh! unless they are explicitly configured.
As mentioned in the commit message, this might be too intrusive for the intended usage of this project, so an alternative is to only do this when given a specific env var that puts ruby-llm into such a "default offline" mode.

Any guidance on this?

As for tests, I'd like some guidance into how to implement unit testing and eventually integration testing specifically for Ollama, in a way that does not require configuring and using all APIs (and attending cost).

And on this?

I didn't consider it originally, but I see now that the test suite has no support for being run without valid keys for all providers, which is a barrier for me.

Is there any chance that the test suite can be mocked as much as possible (requests for metadata, etc) such that only tests that NEED to run API inference fail? This is the only way I can see to mitigate the key requirement problem while still running as much of the test suite as possible.

Another option is to create an entirely separate suit of tests just for the Ollama provider, but the reduced integration test coverage is obviously problematic even on paper.

[tpaulshippy]

Because of the recently added VCR support I was able to get the tests to pass (with empty strings in all the API key environment variables). Wish the tests did not require these, but it's an easy fix.

[ldmosquera]

Excellent! I somehow missed that commit earlier.

I'm now unblocked and will work on tests for this provider.

[ldmosquera]

Tests added, but a few provisional hacks are still in place and will likely need revisiting:

• 37c5fd5
• 11eedd8 (new since force push)
• 4f48aae (new since force push)

Taking a step back, all of these hacks have to do with the requirement for valid API keys when using models.refresh!. This is a blocker for developers without keys for everything (an onus which will only grow with each provider contributed in), and it also creates friction for privacy minded users who want to use Ollama locally with ZERO traffic to cloud providers.

It's a bit of a rough marriage, but we can make it work :D We just need to figure out a sane way to conditionally turn off all but one provider.

---

All current tests pass when relying on VCR cassettes.

[crmne]

Valid point @ldmosquera! I'm working on a solution, will commit it on main soon.

[crmne]

Added configuration requirements handling in 75f99a1

Each provider now specifies what configuration is required via a simple configuration_requirements method (you will need to implement this in your main provider file) that returns an array of config keys as symbols. The Provider module uses this to:

1. Determine if a provider is properly configured
2. Throw an error if you're trying to use that provider without configuration
3. Include ready-to-paste configuration code in the error message
4. Skip unconfigured providers during model refresh while preserving their models

Example of the new error messages:

RubyLLM::ConfigurationError: anthropic provider is not configured. Add this to your initialization:

RubyLLM.configure do |config|
  config.anthropic_api_key = ENV['ANTHROPIC_API_KEY']
end

[ldmosquera]

Smashing 😎

All major hacks dropped, save for RubyLLM::Provider.disable_all_providers (see below).

[crmne]

The implementation path is straightforward - Ollama is just another provider that needs to:

1. Implement the standard provider interface
2. Override specific methods where needed
3. Use the existing test infrastructure
4. Get documented like other providers

I think the only change w.r.t. to other providers is that in the documentation we should add that refreshing models should be done prior to using the API.

Happy to help out. The goal is making Ollama work cleanly within RubyLLM's existing patterns.

[ldmosquera]

Merged main and solved conflicts.

Echoing the commit message in ba56fa2 for visibility:

All cassettes for Ollama are up to date; I manually copied the new
Bedrock responses for model listings.

VCR data will be synthetic until someone can run tests with all
providers properly enabled, including a local Ollama server having run
the rake task to get test models.

[frimmy]

@crmne @ldmosquera just dropping appreciation! 🙇 🙏

[crmne]

@ldmosquera I'd love to see Ollama support merged soon. Do you think you can resolve the conflicts? Once you're ready set this PR as ready for review and I'll take a look.

[ldmosquera]

Ready for review; I imagine there will be some more back and forth before it's mergeable. Apologies and thanks in advance 😬

[ldmosquera]

Merged main; diff to yesterday here.

[khasinski]

Just curious - couldn't it be easier to use OpenAI's compatibility layer in Ollama? https://ollama.com/blog/openai-compatibility

The same worked for OpenRouter, so maybe it's also the case here.

[crmne]

Hey @khasinski I'm pondering the same. The reason why I preferred a true Ollama implementation are summed up perfectly by Ollama themselves:

This is initial experimental support for the OpenAI API. Future improvements under consideration include:

Embeddings API
Function calling
Vision support

However, we could simply test these assumptions and roll it out as OpenAI compatible if everything works.

Case in point OpenRouter claims that they don't support PDFs in their documentation, but you can check out the tests I just committed in RubyLLM main since they do work!

[ldmosquera]

At this point the PR has full coverage for the Ollama API, including repeatable tests plus a rake task that automates downloading the models needed for those tests.

If we were to scale back to simply using Ollama's OpenAI compatibility layer, some Ollama-specific code for parsing responses could be dropped, but other code for querying models would still need to remain. Tests would need to be rethought.

That, along with the fact the Ollama's OAI compatibility is not yet complete as mentioned above, makes think we should continue targeting for Ollama API at least for now. I can commit to watch for incoming Ollama-related issues and provide support.

---

This could be revisited in the future; over time it would be great if we could have generic live model querying capabilities, so that OpenAI compatibility can be used to point at any compatible self-hosted engines like Ollama, KoboldCPP, LlamaCPP, TabbyAPI and so on.

The messy thing is still dealing with uncertain self-hosted model capabitilies - this is inherent, as people can stand up models with any context size, with or without media projector in the case of GGUF based vision models, etc.