Enforce the tool-call contract, asking servers only where it works
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 56s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 56s
The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.
- The catalog reads supports_tool_choice from LiteLLM's model_info;
BudgetedClient sends tool_choice only to models declared true, as
"required" on the normalized route and {"type": "any"} on the
Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
milestone. Finishing with nothing changed is refused once; a prose
reply gets one nudge, then escalates; a call to a tool not offered
counts toward failure_limit, like a failed command; calls after
finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
recognises a proxy fallback, which once credited Claude's answers to
a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
that models call only offered tools, and that a declared
supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
declared and backed up
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
389b71a5d1
commit
b934c46714
16 files changed
+568
-86
No files matched your search
@@ -42,7 +42,9 @@ def test_probe_calibrates_then_sends_half_again_the_window() -> None:
|
||||
words = len(json.loads(request.content)["messages"][0]["content"].split())
|
||||
sizes.append(words * 2)
|
||||
reported = min(words * 2, 4096)
|
||||
return httpx.Response(200, json={"usage": {"prompt_tokens": reported}})
|
||||
return httpx.Response(
|
||||
200, json={"model": "llama3.2", "usage": {"prompt_tokens": reported}}
|
||||
)
|
||||
|
||||
with httpx.Client(
|
||||
base_url="http://litellm", transport=httpx.MockTransport(handler)
|
||||
@@ -51,3 +53,37 @@ def test_probe_calibrates_then_sends_half_again_the_window() -> None:
|
||||
|
||||
assert result.verdict == "matches"
|
||||
assert sizes[1] >= 4096 * 1.4 # the probe really exceeded the window
|
||||
|
||||
|
||||
def test_a_fallback_answer_is_never_credited_to_the_model() -> None:
|
||||
"""Measured: LiteLLM answered an oversized prompt to local-coder with
|
||||
claude-sonnet-5, and the probe called the refusing server roomy."""
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
words = len(json.loads(request.content)["messages"][0]["content"].split())
|
||||
model = "local-coder" if words < 1000 else "claude-sonnet-5"
|
||||
return httpx.Response(
|
||||
200, json={"model": model, "usage": {"prompt_tokens": words * 2}}
|
||||
)
|
||||
|
||||
with httpx.Client(
|
||||
base_url="http://litellm", transport=httpx.MockTransport(handler)
|
||||
) as http:
|
||||
result = probe(http, "local-coder", declared=32768, backing="openai/qwen3-14b")
|
||||
|
||||
assert result.verdict == "fallback"
|
||||
assert "claude-sonnet-5 answered instead of local-coder" in result.message
|
||||
|
||||
|
||||
def test_a_provider_reporting_its_own_model_name_is_not_a_fallback() -> None:
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(
|
||||
200, json={"model": "qwen3-14b", "usage": {"prompt_tokens": 400}}
|
||||
)
|
||||
|
||||
with httpx.Client(
|
||||
base_url="http://litellm", transport=httpx.MockTransport(handler)
|
||||
) as http:
|
||||
result = probe(http, "local-coder", declared=32768, backing="openai/qwen3-14b")
|
||||
|
||||
assert result.verdict != "fallback"
|
||||
Reference in new issue
Block a user