Your local coding agent might be doing nothing at all

Updated 1 October 2026: the larger follow-up - six agents, seven local models, 30 runs each, every run public - is here.

Updated 29 September 2026: we re-ran every agent on these models with a 32k context window. Our harness had been using Ollama’s default 4k context, which cut off the longer agent prompts (Goose’s and Hermes’s). pi’s and Hermes’s results didn’t change. Goose’s did, in both modes - the table and the Goose section below show the new numbers.

The setup

You run a coding agent against a local model - say qwen2.5-coder on Ollama, because it’s a solid coder and it fits on your GPU. You give it a task. It thinks for a bit, prints a plausible-looking summary of what it did, and stops.

Except it didn’t do anything. No files changed. The agent never called a single tool.

This isn’t a rare failure, and it isn’t only the small models. We saw it on the 7B and the 14B. It’s the default behaviour of a broad class of open-weight models with a broad class of agent runtimes, and most of the time nobody notices - because the model’s description of the work it didn’t do reads just fine.

Why it happens

Agents call tools. On a hosted model like Claude or GPT, the model emits tool calls through a dedicated channel - structured, unambiguous, easy to execute. Open-weight models are trained to do the same thing, but the training is thinner and less consistent, and the smaller the model the more it wobbles.

So qwen2.5-coder:7b, asked to edit a file, will often produce this - as ordinary text, in the middle of its reply:

{
  "name": "edit",
  "arguments": { "path": "math.mjs", "edits": [ ... ] }
}

That’s the right intent in slightly the wrong place. A runtime that only looks at the native tool-call channel sees nothing there, treats the whole reply as a final answer, and ends the turn. Task over. Zero tools called.

The benchmark

We ran six small coding tasks - add a CLI subcommand, fix an off-by-one, rename a function across two files, trace a runtime error to its cause - through four agents on the same local models:

• Polyglot, which describes tools in the prompt and parses tool calls back out of the model’s text, with a repair pass for the malformed ones.
• pi, an excellent CLI agent that uses native function calling.
• Hermes Agent (Nous Research), which advertises “11 tool-call parsers.”
• Goose (Linux Foundation’s Agentic AI Foundation, originally built at Block), the largest project in this comparison by a wide margin - and the only other one here that ships a real, documented fix for this exact failure mode.

qwen2.5-coder - capable coder, shaky native tool-calling

Both the 7B and the 14B:

pi and Hermes made zero tool calls on every run of every task - on both models. Not because the tasks were hard, but because the model never used the channel they were listening on. The 14B even hallucinated tool names into its text ({"name":"skills_list","arguments":{}}) - still text, still ignored.

Polyglot recognised the text-JSON shape (it has a regression test for exactly this) and ran the calls. 39% on the 7B isn’t impressive on its own - a 7B still writes plenty of broken edits - but the agent did the work instead of narrating a fiction, and on the 14B it’s up near 80%.

The Hermes result surprised us, given the “11 tool-call parsers” line. Reading the hermes-agent source: those parsers live at the model-training and serving layer (the hermes-function-calling datasets, and a vLLM-side parser for Hermes-format models) - not in the agent runtime. The runtime’s live loop reads the native tool_calls field only. A helper called strip_think_blocks actively deletes text-emitted <tool_call> blocks from the model’s output as noise, and there’s an explicit code comment treating in-context tool-call syntax as “data - do not re-emit it as a tool call.” So Hermes is genuinely strong for a Nous model behind a Nous endpoint; point it at your own local Qwen and it behaves like pi. It’s a deliberate design choice, and it’s the opposite of ours.

Goose is the one that actually surprised us. It’s backed by the Linux Foundation’s Agentic AI Foundation - platinum members include AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, and Microsoft - and, unlike pi or Hermes, it ships a real, documented fix aimed at exactly this problem: GOOSE_TOOLSHIM, which routes a non-tool-calling model’s output through a second, interpretive model to convert it into a proper call. We turned it on and reran everything. In default mode, Goose does attempt calls but completes nothing on either model. With the toolshim on, it completed 4/18 on the 7B and 13/18 on the 14B. On the 7B that’s well short of Polyglot; on the 14B it’s close. A second model interpreting the first model’s text is the same idea as Polyglot’s repair, done with an extra model instead of a parser - and on the 14B it works.

qwen3-coder (30B) - strong native tool-calling

When the model’s native function calling is solid, pi and Polyglot are both strong. (Even here, one of pi’s misses was qwen3-coder emitting a tool call as text - the channel slips occasionally on every model; it’s just rare enough not to matter.)

Hermes scored 1/18 here, but that number is confounded: its restricted toolset carries a background-process manager and a search tool whose argument shapes trip the model, and its default 26-tool surface times a local model out completely. It’s a persistent-assistant platform, not a focused coding agent - so we’re not reporting that as a clean parsing result.

The picture

The gap lives in one band: models good enough to do the work, but not reliable enough at emitting tool calls through the local server. That’s not a fringe - it’s the 7B and the 14B, and it keeps being repopulated: every non-flagship open-weight release, every 4-bit quant, every model you run yourself instead of paying per token.

What Polyglot does

Polyglot never assumes the model will format a tool call correctly. It:

1. Teaches the grammar in the system prompt.
2. Parses tool calls out of the streamed text - the taught <tool_call> tags, and the OpenAI-style {name, arguments} blob the model reaches for when it forgets.
3. Repairs the near-misses - trailing commas, single quotes, a fenced code block wrapped around the arguments, a tool name that’s one character off - and runs the call anyway.
4. Shows you every repair. A ↺ repaired marker on the card, the raw model output one keypress away, a per-model tally in /reliability. A parser fix can’t quietly paper over a model getting worse.

Same parser, same executor, under every provider - so your agent loop behaves the same on a flaky local model as it does on Claude.

Methodology

• Tasks. Six small, realistic coding jobs: add a CLI subcommand, fix an off-by-one bug, answer a question from a file without editing anything, remove dead code, rename a function and fix its caller across two files, and trace a runtime error to its cause across two files. Each seeded into a fresh working directory per run.
• Models. All local, all via Ollama, all on the same machine, one at a time: qwen2.5-coder:7b and :14b (the size most people actually run), qwen3-coder 30B-A3B (strong native tool-calling, as a control), llama3.2:3b (a floor check).
• Context window. Every model runs with a 32k context (num_ctx 32768), so no agent’s prompt is truncated. The results above were re-run this way on 29 September.
• Agents. Polyglot, pi (@earendil-works/pi-coding-agent), Hermes Agent (hermes-agent, Nous Research), and Goose (aaif-goose/goose, Linux Foundation) - each pointed at the identical model, identical tasks, identical machine, no changes to the tasks between tools. Goose was also tested with its GOOSE_TOOLSHIM mechanism explicitly enabled, using llama3.2:3b as the interpreter model (Goose’s own documented default, mistral-nemo, wasn’t already on our machine).
• Scoring. Automated, not eyeballed: a task counts as done only if the final file contents (or, for read-only tasks, the reported answer) satisfy a fixed pass/fail check, applied the same way to every agent.
• Trials. 3 runs per task per model per agent - weak models are noisy enough that a single run isn’t representative.
• Controlled for. Same hardware, same local endpoint (Ollama’s OpenAI-compatible API) for every agent under test, same task wording.

We’re keeping the full harness and raw transcripts internal for now, but every number above should reproduce within a run or two if you point these four agents at the same setup yourself.

Honest caveats

• Hermes’s qwen3-coder number is confounded by its runtime design, not just parsing - we’ve flagged it rather than lead with it.
• On a genuinely weak model (3B) nothing saves you - all agents mostly fail.
• Polyglot’s 7B number (39%) is comparative, not a claim that it makes a 7B good.
• Ollama’s OpenAI-compatible endpoint is the realistic local setup, but it isn’t necessarily every tool at its absolute best.
• Goose’s toolshim result used a substituted interpreter model (llama3.2:3b, not their documented default mistral-nemo) - worth re-running with the real default before treating that specific number as final. The default (no-toolshim) result doesn’t depend on this and needs no such caveat. This also isn’t a Goose-reviewed benchmark - it’s our own good-faith run of their public CLI, once.

Try it

Polyglot is free and open source. npm install -g @usepolyglot/cli, point it at whatever’s already running on your GPU, and see for yourself whether your agent is doing the work or just narrating it. Get started →