Enforce the tool-call contract, asking servers only where it works

The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.

- The catalog reads supports_tool_choice from LiteLLM's model_info;
  BudgetedClient sends tool_choice only to models declared true, as
  "required" on the normalized route and {"type": "any"} on the
  Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
  milestone. Finishing with nothing changed is refused once; a prose
  reply gets one nudge, then escalates; a call to a tool not offered
  counts toward failure_limit, like a failed command; calls after
  finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
  recognises a proxy fallback, which once credited Claude's answers to
  a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
  both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
  that models call only offered tools, and that a declared
  supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
  declared and backed up

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/Tuning-Guide.md

@@ -282,6 +282,48 @@ log.
 - **Deciding:** if milestones stop with "cannot fit", look first at
   `tools.max_output_chars` and the model's window, not at these.
 
+### Enforced tool calls — `supports_tool_choice`, declared in LiteLLM
+
+The agent's loops depend on every turn being a well-formed call to a real
+tool. A model only *writes text*: a "tool call" is text in a format the model
+was trained to produce, which the server recognises and turns into a
+structured call. By default (`tool_choice: auto`) the server just parses
+whatever comes out, so prose, broken JSON and calls to invented tools are all
+possible.
+
+With `tool_choice: required`, a server that supports it **constrains
+decoding**: at every step it rules out any token that couldn't lead to a
+valid call to one of the offered tools, with arguments matching its schema.
+The model still chooses the tool and the arguments; it can't produce anything
+else. That makes it a property of the **model server** (and sometimes of the
+model), not something a prompt can buy.
+
+So the agent asks for it only where it is known to work: models whose LiteLLM
+entry declares `supports_tool_choice: true`. Measured on the 4070:
+
+| Alias | Server | `required` on a chat prompt | Declare |
+|---|---|---|---|
+| `local-fast` (Qwen3-8B-AWQ) | vLLM 0.29 | calls `finish` every time | `true` |
+| `local-coder` (Qwen3-14B) | llama.cpp | can't stop; writes until the output ceiling (a known Qwen3-14B bug) | `false` |
+| `claude-haiku-4-5` | Anthropic | calls `finish` every time | `true` (LiteLLM already says so) |
+
+Asking a model that can't honour it can be worse than not asking: the
+Qwen3-14B turn above costs a full output ceiling of generation, where a plain
+prose reply would have cost a few hundred tokens.
+
+Where it isn't enforced, the executor holds the contract itself:
+
+- **`finish` is the only way to end a milestone.** It is refused once if
+  nothing has been changed yet.
+- **A prose reply is a violation.** The model is told once; a second escalates.
+- **A call to a tool that wasn't offered counts as a failure** toward
+  `failure_limit`, like a failed command, so an inventing model escalates in a
+  couple of turns instead of using up `max_turns`.
+
+The capability matrix checks the declaration: a model declared `true` that
+answers a required call in prose fails
+`test_supports_tool_choice_is_declared_truthfully`.
+
 ### Declaring a local model's limits
 
 LiteLLM knows commercial models' limits. For a local model, declare them in
@@ -292,8 +334,10 @@ model_list:
   - model_name: local-fast
     litellm_params: {model: openai/qwen3-8b-awq, api_base: http://inference-host:8080/v1}
     model_info:
-      max_input_tokens: 32768   # the server's per-request context
-      max_output_tokens: 4096   # the most one response may use
+      max_input_tokens: 32768       # the server's per-request context
+      max_output_tokens: 4096       # the most one response may use
+      supports_function_calling: true
+      supports_tool_choice: true    # only once measured; see above
 ```
 
 `max_input_tokens` is the *whole* window, prompt plus response. The agent
@@ -332,7 +376,7 @@ Upstream choices change what's right downstream, so tune in this order and
 change one thing at a time.
 
 1. **Model and quantization.** Start with whether a candidate holds the
-   tool-call contract at all:
+   tool-call contract at all, and whether its server enforces `tool_choice`:
 
        uv run pytest -m integration tests/integration/test_model_capability.py \
          --capability-models=alias-a,alias-b
@@ -342,6 +386,14 @@ change one thing at a time.
    each one actually points at in LiteLLM (`/model/info`, the
    `litellm_params.model` field). Two aliases believed to be the same model
    once weren't, and that cost a lot of debugging.
+
+   **Make sure LiteLLM has no fallback for a model you are measuring.** A
+   fallback answers a failed request with another model, under that model's
+   name, in a response that otherwise looks normal. One made a local model
+   that couldn't call tools at all look like it "held the tool contract 20 of
+   20": every answer was Claude's. The capability matrix and the probe now
+   check which model answered and fail on a mismatch, but the fallback also
+   quietly turns local work into frontier spending in production.
 2. **Context length and KV-cache precision** on the server. Take the largest
    context that fits beside the weights with 1–2 GB of headroom. Try an 8-bit
    KV cache before giving up context.
@@ -369,8 +421,5 @@ section for each:
 - **Compaction.** A model-written summary replacing old turns, instead of
   dropping them. The log already has an event for it; it needs a summarizing
   model and a policy for what is safe to summarize.
-- **Enforced tool calls.** An explicit `finish` tool, so the server can be told
-  a tool call is required and hold the model to the contract, where today it
-  can fall back to prose.
 - **A benchmark script** that runs the capability matrix and representative
   milestones across candidate models and prints a comparison table.