Node request: OllamaInference (local LLM via Ollama, no API key needed)

[dhruvmohantewari]

Proposed node name

OllamaInference

What should the node do?

Calls a self-hosted Ollama instance via its OpenAI-compatible endpoint, as a local/no-cost alternative to LlmInference (which only supports OpenRouter). Self-contained, no separate config node, since there's no API key to externalise (Ollama needs no auth). Configurable base URL defaults to http://127.0.0.1:11434/v1 for the common localhost case.

Inputs and outputs

inputs:
  - prompt: String (required)
  - systemPrompt: String (optional)
config fields:
  - model: String (required) — Ollama model tag, e.g. qwen2.5-coder:7b
  - baseUrl: String (optional, default http://127.0.0.1:11434/v1)
  - temperature, maxTokens, topP: Number (optional)
  - parseJson: Boolean (optional)
outputs:
  - response: MustOverride (String, or Dict when parseJson is set)

Use case

Running a self-hosted personal AI server (Ollama + local models) and want to use Weft for orchestration without routing every LLM call through a paid provider. Right now LlmInference only supports OpenRouter, so there's no way to wire a graph node directly to a local model, this closes that gap for anyone running Ollama, which is one of the most common local-LLM setups.

Current workaround (if any)

ExecPython with a manual Python requests call to Ollama's API works, but it's untyped, doesn't get the dashboard's node UI/validation, and has to be re-written per project instead of being a reusable typed node.

[dhruvmohantewari]

I already have this built and tested locally (compiles, registers, verified live against a real model)

[barriegith]

Strong +1 on this. One suggestion: rather than an Ollama-specific node, a pluggable LLM provider interface where one of the options is a local subprocess/CLI command (prompt in on stdin, completion out on stdout) would cover Ollama, llama.cpp and, importantly, users on flat-fee CLI subscriptions, all in one stroke and with no OpenRouter API key.

Right now every LLM node bills per token through OpenRouter, which quietly rules Weft out for a large set of self-hosters and cost-capped teams. A generic "command provider" (plus the Ollama one requested here) would open the door for them without changing the language at all. Happy to help spec the provider interface if useful.