Local & remote model servers

Polyglot’s openai-compatible provider talks to anything that implements the OpenAI chat-completions API. You set three things: the base URL, the model name that server expects, and (for remote/hosted endpoints) an API key.

export POLYGLOT_PROVIDER=openai-compatible
export POLYGLOT_MODEL=<model the server exposes>
export POLYGLOT_BASE_URL=<http://host:port/v1>
export POLYGLOT_API_KEY=<only if the endpoint requires one>

Or the equivalent in settings.json.

Per-server notes

Server	Base URL (default)	Notes
Ollama	http://localhost:11434/v1	Used if POLYGLOT_BASE_URL is unset. ollama pull <model> first. Recent versions support structured output.
LM Studio	http://localhost:1234/v1	Start the server from the Developer tab. Supports structured output.
llama.cpp server	http://localhost:8080/v1	Run llama-server. Supports GBNF grammars → structured output.
vLLM	http://localhost:8000/v1	--api-key if you set one. Supports guided decoding → structured output.
TGI / others	varies	Anything OpenAI-compatible works for the default free-text tool protocol; structured output depends on response_format support.

Context window

A local server with a small context window doesn’t refuse a long prompt: it quietly drops the start of it and answers from what’s left. For an agent, the start is its instructions and tool descriptions, so it forgets how to call tools, with no error anywhere. Ollama is the usual case: unless told otherwise it runs a model with a small default window (4,096 tokens on recent versions, 2,048 on older ones), whatever the model supports, and an agent’s prompt passes that within the first few turns.

Polyglot watches for it. Each turn it compares the size of the prompt it sent with the prompt tokens the server says it read, and when the server read clearly less, it warns once:

The model server read about 2,050 of the ~5,169 prompt tokens sent: its context window is probably
cutting off the start of the prompt (instructions and tool docs), so answers and tool calls will
suffer. Raise the server's context window (Ollama: OLLAMA_CONTEXT_LENGTH, or num_ctx in a Modelfile;
llama.cpp: -c; vLLM: --max-model-len).

In headless runs the same warning goes to stderr. To fix it, give the server a larger window:

Server	How
Ollama	Start it with OLLAMA_CONTEXT_LENGTH=32768 ollama serve, or set PARAMETER num_ctx 32768 in a Modelfile and ollama create a variant
llama.cpp	llama-server -c 32768
vLLM	--max-model-len 32768
LM Studio	The context length setting when loading the model

32,768 suits most coding agents; a larger window uses more memory. To see the window a server actually runs, use --probe: for Ollama it reads the window of the loaded model from Ollama’s own API, since Ollama’s OpenAI-compatible endpoint doesn’t report it.

Remote servers

Point POLYGLOT_BASE_URL at the remote host and set POLYGLOT_API_KEY if it’s protected. Note that with a remote endpoint your prompts and tool output leave the machine - see Data handling. Run /status to confirm whether the active endpoint is local or remote.

Choosing a model

Any instruction-tuned model can hold a conversation, but agent work needs one that follows the tool-call format reliably. Coding-tuned models in the ~7B+ range (Qwen2.5-Coder, DeepSeek-Coder, etc.) are a reasonable floor; smaller or non-coding models often need structured output to be usable. See the compatibility matrix.