The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.
- The catalog reads supports_tool_choice from LiteLLM's model_info;
BudgetedClient sends tool_choice only to models declared true, as
"required" on the normalized route and {"type": "any"} on the
Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
milestone. Finishing with nothing changed is refused once; a prose
reply gets one nudge, then escalates; a call to a tool not offered
counts toward failure_limit, like a failed command; calls after
finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
recognises a proxy fallback, which once credited Claude's answers to
a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
that models call only offered tools, and that a declared
supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
declared and backed up
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Completes step 7.
- expert.py: one stateless call for a question needing no repo access.
Structured output is contracted at the prompt level, since vendors
differ on JSON enforcement, so the reply is parsed and validated
locally with one retry — tolerating a fenced code block, because
models wrap JSON habitually, but not missing fields
- The per-task budget is counted from the canonical log rather than
memory, so it survives a restart and cannot hand a circling executor
a fresh allowance. Exhaustion raises rather than answering: a caller
out of consults should get on with the work or escalate
- Consults are recorded as delegation_started/completed with role
consultant. A16 asks for a "consult/delegation eval set", and
reusing the composite events gets consult evaluation for free rather
than inventing a parallel record
- evals.py pairs starts with completions into eval records, with
verdicts kept outside the canonical log: the log records what
happened, a verdict is a later judgement about it, and judgements
get revised. Schema v3 adds the outcomes table, so a v1 store now
traverses two migrations — a better exercise of the hook than one
- ask_expert is offered to the executor as a tool only when a desk is
configured, in addition to the observable escalation triggers and
never instead of them (A12). Consulting is not mutating, so a
completion claim backed only by consults is still challenged
Found live: passing an empty message list and putting the question in
the system prompt fails outright — the Messages API requires at least
one non-system message. The question now travels as a synthesised user
message that is never appended to the log, which keeps the consult
stateless while remaining a valid request.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous fix challenged a completion claim made before any tool
call, but treated every tool as work — so a model that ran one
list_dir and then declared the milestone done was believed, and CI
caught the same empty milestone again.
Reading a file or listing a directory is the model orienting itself.
Only write_file and run_command now count, so a completion claim must
be backed by something that changed or verified the tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI caught the executor recording an empty milestone as a success: the
model answered with prose on its first turn, and the loop reads "no
tool call" as "the plan is done". It had therefore finished a
milestone having touched nothing, and the milestone would have gone
on to publish with no changes.
A completion claim is now challenged once when no tool has been
called yet — the same shape as the planner's nudge — and a model that
persists escalates instead of reporting success. After the first tool
call the claim is taken at face value, as before.
The live milestone test failed roughly one run in five on this; four
consecutive runs pass now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- tools.py: four flat tools (read_file, write_file, list_dir,
run_command) per A15's reduced manifest. Deliberately unsandboxed:
the VM is the trust boundary (B3) and tool use is auto-approved
inside an approved unit of work (A4), so path confinement would add
failure modes without adding safety. What is enforced is
operational — timeout with process-group kill so a backgrounded
server cannot outlive its command, output truncation so a build log
cannot eat the context window, closed stdin so prompts fail fast,
and a scrubbed environment so project code never sees the agent's
credentials
- executor.py: renders the log, calls the executor role with the
reduced manifest, dispatches tool calls, and checkpoints every
result (A13). Escalation is observable, never self-assessed (A12):
consecutive command failures and a hard turn cap. Failed reads do
not count — looking around is not failing
- Until delegation exists (step 7), escalation gates to the human
with the failures attached
- control.py: start milestone now runs the executor; a stopped
milestone gates to AWAIT_INPUT
Two bugs found by running it live against real models:
- A turn-ending gate tool (submit_plan) left a tool_call with no
tool_result, which providers reject outright, so every planning
turn after the first failed with a 400. The planner now records the
gate as the call's outcome, which is also the truthful transcript
- The renderer now emits a placeholder for any tool_call lacking a
result, so a crash between call and result (A3) cannot poison the
transcript permanently
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dongarandClaude Opus 5.5