Local & remote model servers

Polyglot’s openai-compatible provider talks to anything that implements the OpenAI chat-completions API. You set three things: the base URL, the model name that server expects, and (for remote/hosted endpoints) an API key.

export POLYGLOT_PROVIDER=openai-compatible
export POLYGLOT_MODEL=<model the server exposes>
export POLYGLOT_BASE_URL=<http://host:port/v1>
export POLYGLOT_API_KEY=<only if the endpoint requires one>

Or the equivalent in settings.json.

Per-server notes

Server	Base URL (default)	Notes
Ollama	http://localhost:11434/v1	Used if POLYGLOT_BASE_URL is unset. ollama pull <model> first. Recent versions support structured output.
LM Studio	http://localhost:1234/v1	Start the server from the Developer tab. Supports structured output.
llama.cpp server	http://localhost:8080/v1	Run llama-server. Supports GBNF grammars → structured output.
vLLM	http://localhost:8000/v1	--api-key if you set one. Supports guided decoding → structured output.
TGI / others	varies	Anything OpenAI-compatible works for the default free-text tool protocol; structured output depends on response_format support.

Remote servers

Point POLYGLOT_BASE_URL at the remote host and set POLYGLOT_API_KEY if it’s protected. Note that with a remote endpoint your prompts and tool output leave the machine - see Data handling. Run /status to confirm whether the active endpoint is local or remote.

Choosing a model

Any instruction-tuned model can hold a conversation, but agent work needs one that follows the tool-call format reliably. Coding-tuned models in the ~7B+ range (Qwen2.5-Coder, DeepSeek-Coder, etc.) are a reasonable floor; smaller or non-coding models often need structured output to be usable. See the compatibility matrix.