Enforce the tool-call contract, asking servers only where it works #45

2026-09-24T22:54:50Z

dev-agent commented

2026-09-24 22:54:50 +00:00

The third 4090-readiness change: enforced tool calls.

What was measured first

Through LiteLLM, with a prompt that invites chatting and tool_choice: required:

Alias	Server	Result	Declared in LiteLLM
`local-fast` (Qwen3-8B-AWQ)	vLLM 0.29	calls `finish` every time	`supports_tool_choice: true`
`local-coder` (Qwen3-14B)	llama.cpp	can't stop; writes until the output ceiling (a known Qwen3-14B bug)	`false`
`claude-haiku-4-5`	Anthropic	calls `finish` every time	`true`

So enforcement is requested per model. The loop holds the contract itself everywhere.

Changes

tool_choice only where declared. The catalog reads supports_tool_choice. BudgetedClient sends "required" (normalized route) or {"type": "any"} (Anthropic passthrough) only to models declared true, and only when tools are offered.
An explicit finish tool, now the only way to end a milestone. Prose no longer means "done".
- Finishing with nothing changed is refused once, then escalates.
- A prose reply gets one nudge, then escalates.
- A call to a tool that wasn't offered counts toward failure_limit. Measured: a model on a server that doesn't enforce tool_choice called chat, say_hello and respond.
- Calls after finish are recorded as not run.
Fallback detection. Clients record which model actually answered, and ModelFacts.answered() recognises a proxy fallback. Such a fallback once credited Claude's answers to a local model that couldn't call tools at all.
The context probe now reports a fallback instead of trusting it (a bug from #44). Rerun against both local models, it reports refuses over-long prompts, which is correct.
Capability matrix:
- every answer must come from the model asked;
- a new test that models call only offered tools;
- a new test that a declared supports_tool_choice is true.
Spec A15 now says how constrained decoding is requested, and what backs it up where it's unavailable.
The tuning guide gets a section on enforced tool calls and a warning about LiteLLM fallbacks.

Tests

Unit:
- finish, the refusal and escalation, prose, invented tools, calls after finish;
- tool_choice rendering on both routes, and the wrapper asking only declared models;
- catalog parsing, fallback recognition, and the probe's fallback verdict.
Live capability matrix on local-fast, local-coder and claude-haiku-4-5: 20 passed, 1 skipped. The skip is local-coder's tool_choice test, as declared. Every answer came from the model asked.
scripts/check.sh passes: 377 unit tests, and the integration suite with 18 passed and 7 skipped. The live milestone now ends with the model calling finish.

🤖 Generated with Claude Code

The third 4090-readiness change: enforced tool calls. ## What was measured first Through LiteLLM, with a prompt that invites chatting and `tool_choice: required`: | Alias | Server | Result | Declared in LiteLLM | |---|---|---|---| | `local-fast` (Qwen3-8B-AWQ) | vLLM 0.29 | calls `finish` every time | `supports_tool_choice: true` | | `local-coder` (Qwen3-14B) | llama.cpp | can't stop; writes until the output ceiling (a known Qwen3-14B bug) | `false` | | `claude-haiku-4-5` | Anthropic | calls `finish` every time | `true` | So enforcement is requested per model. The loop holds the contract itself everywhere. ## Changes - **`tool_choice` only where declared.** The catalog reads `supports_tool_choice`. `BudgetedClient` sends `"required"` (normalized route) or `{"type": "any"}` (Anthropic passthrough) only to models declared `true`, and only when tools are offered. - **An explicit `finish` tool**, now the only way to end a milestone. Prose no longer means "done". - Finishing with nothing changed is refused once, then escalates. - A prose reply gets one nudge, then escalates. - **A call to a tool that wasn't offered counts toward `failure_limit`.** Measured: a model on a server that doesn't enforce `tool_choice` called `chat`, `say_hello` and `respond`. - Calls after `finish` are recorded as not run. - **Fallback detection.** Clients record which model actually answered, and `ModelFacts.answered()` recognises a proxy fallback. Such a fallback once credited Claude's answers to a local model that couldn't call tools at all. - **The context probe** now reports a fallback instead of trusting it (a bug from #44). Rerun against both local models, it reports **refuses** over-long prompts, which is correct. - **Capability matrix:** - every answer must come from the model asked; - a new test that models call only offered tools; - a new test that a declared `supports_tool_choice` is true. - **Spec A15** now says how constrained decoding is requested, and what backs it up where it's unavailable. - **The tuning guide** gets a section on enforced tool calls and a warning about LiteLLM fallbacks. ## Tests - Unit: - `finish`, the refusal and escalation, prose, invented tools, calls after `finish`; - `tool_choice` rendering on both routes, and the wrapper asking only declared models; - catalog parsing, fallback recognition, and the probe's fallback verdict. - **Live capability matrix** on `local-fast`, `local-coder` and `claude-haiku-4-5`: 20 passed, 1 skipped. The skip is `local-coder`'s `tool_choice` test, as declared. Every answer came from the model asked. - `scripts/check.sh` passes: 377 unit tests, and the integration suite with 18 passed and 7 skipped. The live milestone now ends with the model calling `finish`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-24 22:54:50 +00:00

Enforce the tool-call contract, asking servers only where it works

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 56s

Details

b934c46714

The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.

- The catalog reads supports_tool_choice from LiteLLM's model_info;
  BudgetedClient sends tool_choice only to models declared true, as
  "required" on the normalized route and {"type": "any"} on the
  Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
  milestone. Finishing with nothing changed is refused once; a prose
  reply gets one nudge, then escalates; a call to a tool not offered
  counts toward failure_limit, like a failed command; calls after
  finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
  recognises a proxy fallback, which once credited Claude's answers to
  a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
  both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
  that models call only offered tools, and that a declared
  supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
  declared and backed up

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dongar approved these changes

2026-09-25 00:05:42 +00:00

dongar merged commit b2bbf70b7b into main

2026-09-25 00:05:47 +00:00

dongar deleted branch enforce-tool-calls

2026-09-25 00:05:47 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#45