Enforce the tool-call contract, asking servers only where it works
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 56s

The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.

- The catalog reads supports_tool_choice from LiteLLM's model_info;
  BudgetedClient sends tool_choice only to models declared true, as
  "required" on the normalized route and {"type": "any"} on the
  Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
  milestone. Finishing with nothing changed is refused once; a prose
  reply gets one nudge, then escalates; a call to a tool not offered
  counts toward failure_limit, like a failed command; calls after
  finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
  recognises a proxy fallback, which once credited Claude's answers to
  a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
  both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
  that models call only offered tools, and that a declared
  supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
  declared and backed up

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-09-24 22:54:49 +00:00
1 parent 389b71a5d1
commit b934c46714
16 files changed
+568 -86

No files matched your search

+16
View File
@@ -54,11 +54,26 @@ class ModelFacts:
max_input_tokens: int | None = None
# The most one response may use (model_info.max_output_tokens).
max_output_tokens: int | None = None
# Whether its server enforces tool_choice (model_info.supports_tool_choice).
# Only a declared True counts: a model that says nothing is not asked.
supports_tool_choice: bool | None = None
@property
def native_anthropic(self) -> bool:
return self.provider == ANTHROPIC_PROVIDER
def answered(self, reported: str | None) -> bool:
"""Whether a response reporting this model name came from this model.
A proxy fallback answers under another model's name. LiteLLM reports
the alias; a provider may report its own name for the backing model,
sometimes with a date suffix ("claude-haiku-4-5-20251001").
"""
if not reported:
return True # nothing reported, nothing contradicted
own_name = self.backing.partition("/")[2] or self.backing
return reported in (self.alias, self.backing) or reported.startswith(own_name)
class ModelCatalog:
"""What one LiteLLM key can see, as of the moment it was asked."""
@@ -87,6 +102,7 @@ class ModelCatalog:
supports_tools=info.get("supports_function_calling"),
max_input_tokens=_positive_int(info.get("max_input_tokens")),
max_output_tokens=_positive_int(info.get("max_output_tokens")),
supports_tool_choice=info.get("supports_tool_choice"),
)
return cls(facts)