Completes step 7.
- expert.py: one stateless call for a question needing no repo access.
Structured output is contracted at the prompt level, since vendors
differ on JSON enforcement, so the reply is parsed and validated
locally with one retry — tolerating a fenced code block, because
models wrap JSON habitually, but not missing fields
- The per-task budget is counted from the canonical log rather than
memory, so it survives a restart and cannot hand a circling executor
a fresh allowance. Exhaustion raises rather than answering: a caller
out of consults should get on with the work or escalate
- Consults are recorded as delegation_started/completed with role
consultant. A16 asks for a "consult/delegation eval set", and
reusing the composite events gets consult evaluation for free rather
than inventing a parallel record
- evals.py pairs starts with completions into eval records, with
verdicts kept outside the canonical log: the log records what
happened, a verdict is a later judgement about it, and judgements
get revised. Schema v3 adds the outcomes table, so a v1 store now
traverses two migrations — a better exercise of the hook than one
- ask_expert is offered to the executor as a tool only when a desk is
configured, in addition to the observable escalation triggers and
never instead of them (A12). Consulting is not mutating, so a
completion claim backed only by consults is still challenged
Found live: passing an empty message list and putting the question in
the system prompt fails outright — the Messages API requires at least
one non-system message. The question now travels as a synthesised user
message that is never appended to the log, which keeps the consult
stateless while remaining a valid request.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A frontier model receives a scoped sub-task, works autonomously with
full tool access, and hands back; the local loop reads the handback
note and the file state, never the reasoning.
- Branch at delegation start, and abort discards it. That required
committing each turn's work onto the branch: without commits,
deleting a branch discards nothing, so "abort discards the work"
was a comforting phrase rather than a fact. The caller's own
in-flight changes are committed to their branch first, so an abort
cannot destroy work nobody delegated
- Hard caps on turns and wall clock; a breach is a defined failure
return, and its result is not usable, so the caller gates rather
than resuming on a half-edited tree
- Every event is checkpointed as it arrives, so a crash mid-delegation
leaves the branch at a known commit and a truthful transcript
- Handback is validated locally: blank fields or a non-boolean success
flag are refused and the delegation continues rather than accepting
a useless note
Two failure modes found by running it against a real model:
- A success claim with nothing changed was accepted. The handback plus
file state is the entire interface back, so a note describing work
that does not exist is worse than no note. Success now requires a
successful mutating tool call; reporting failure still does not,
since "this is impossible" is a legitimate finding
- The delegate invented the absolute path /workspace, wrote there, ran
its tests there, and handed back truthfully — with the repo empty
and git seeing nothing. Writes outside the workspace are now refused
with a message naming the real root, and both prompts state where
the workspace is instead of leaving the model to guess. This is not
sandboxing: work outside the repo is invisible to git, so the
milestone would publish an empty PR
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous fix challenged a completion claim made before any tool
call, but treated every tool as work — so a model that ran one
list_dir and then declared the milestone done was believed, and CI
caught the same empty milestone again.
Reading a file or listing a directory is the model orienting itself.
Only write_file and run_command now count, so a completion claim must
be backed by something that changed or verified the tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI caught the executor recording an empty milestone as a success: the
model answered with prose on its first turn, and the loop reads "no
tool call" as "the plan is done". It had therefore finished a
milestone having touched nothing, and the milestone would have gone
on to publish with no changes.
A completion claim is now challenged once when no tool has been
called yet — the same shape as the planner's nudge — and a model that
persists escalates instead of reporting success. After the first tool
call the claim is taken at face value, as before.
The live milestone test failed roughly one run in five on this; four
consecutive runs pass now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- tools.py: four flat tools (read_file, write_file, list_dir,
run_command) per A15's reduced manifest. Deliberately unsandboxed:
the VM is the trust boundary (B3) and tool use is auto-approved
inside an approved unit of work (A4), so path confinement would add
failure modes without adding safety. What is enforced is
operational — timeout with process-group kill so a backgrounded
server cannot outlive its command, output truncation so a build log
cannot eat the context window, closed stdin so prompts fail fast,
and a scrubbed environment so project code never sees the agent's
credentials
- executor.py: renders the log, calls the executor role with the
reduced manifest, dispatches tool calls, and checkpoints every
result (A13). Escalation is observable, never self-assessed (A12):
consecutive command failures and a hard turn cap. Failed reads do
not count — looking around is not failing
- Until delegation exists (step 7), escalation gates to the human
with the failures attached
- control.py: start milestone now runs the executor; a stopped
milestone gates to AWAIT_INPUT
Two bugs found by running it live against real models:
- A turn-ending gate tool (submit_plan) left a tool_call with no
tool_result, which providers reject outright, so every planning
turn after the first failed with a 400. The planner now records the
gate as the call's outcome, which is also the truthful transcript
- The renderer now emits a placeholder for any tool_call lacking a
result, so a crash between call and result (A3) cannot poison the
transcript permanently
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dongarandClaude Opus 5