Enforce the tool-call contract, asking servers only where it works
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 56s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 56s
The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.
- The catalog reads supports_tool_choice from LiteLLM's model_info;
BudgetedClient sends tool_choice only to models declared true, as
"required" on the normalized route and {"type": "any"} on the
Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
milestone. Finishing with nothing changed is refused once; a prose
reply gets one nudge, then escalates; a call to a tool not offered
counts toward failure_limit, like a failed command; calls after
finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
recognises a proxy fallback, which once credited Claude's answers to
a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
that models call only offered tools, and that a declared
supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
declared and backed up
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
389b71a5d1
commit
b934c46714
16 files changed
+568
-86
No files matched your search
@@ -54,11 +54,26 @@ class ModelFacts:
|
||||
max_input_tokens: int | None = None
|
||||
# The most one response may use (model_info.max_output_tokens).
|
||||
max_output_tokens: int | None = None
|
||||
# Whether its server enforces tool_choice (model_info.supports_tool_choice).
|
||||
# Only a declared True counts: a model that says nothing is not asked.
|
||||
supports_tool_choice: bool | None = None
|
||||
|
||||
@property
|
||||
def native_anthropic(self) -> bool:
|
||||
return self.provider == ANTHROPIC_PROVIDER
|
||||
|
||||
def answered(self, reported: str | None) -> bool:
|
||||
"""Whether a response reporting this model name came from this model.
|
||||
|
||||
A proxy fallback answers under another model's name. LiteLLM reports
|
||||
the alias; a provider may report its own name for the backing model,
|
||||
sometimes with a date suffix ("claude-haiku-4-5-20251001").
|
||||
"""
|
||||
if not reported:
|
||||
return True # nothing reported, nothing contradicted
|
||||
own_name = self.backing.partition("/")[2] or self.backing
|
||||
return reported in (self.alias, self.backing) or reported.startswith(own_name)
|
||||
|
||||
|
||||
class ModelCatalog:
|
||||
"""What one LiteLLM key can see, as of the moment it was asked."""
|
||||
@@ -87,6 +102,7 @@ class ModelCatalog:
|
||||
supports_tools=info.get("supports_function_calling"),
|
||||
max_input_tokens=_positive_int(info.get("max_input_tokens")),
|
||||
max_output_tokens=_positive_int(info.get("max_output_tokens")),
|
||||
supports_tool_choice=info.get("supports_tool_choice"),
|
||||
)
|
||||
return cls(facts)
|
||||
|
||||
|
||||
Reference in new issue
Block a user