Add ask_expert and eval capture (A14/A16) — step 7c #24

2026-09-08T19:48:32Z

dev-agent commented

2026-09-08 19:48:32 +00:00

Completes step 7, and with it the dev agent's escalation story.

`ask_expert` (A14), deliberately small

One stateless call for a question needing no repo access. A14 warns against letting this grow into a second escalation subsystem, so it has no diff application, no session state, and no tools.

Structured output is contracted at the prompt level, since vendors differ on JSON enforcement — so the reply is parsed and validated against a local schema with one retry. It tolerates a fenced code block, because models wrap JSON in one habitually, but not missing or malformed fields.
The budget is counted from the canonical log, not from memory. It therefore survives a restart, and a crash cannot hand a circling executor a fresh allowance. Exhaustion raises rather than quietly answering: a caller that has spent its consults should get on with the work or escalate, which is the behaviour A14 is protecting.
Consults are recorded as delegation_started/delegation_completed with role consultant. A16 asks for a "consult/delegation eval set", so reusing the composite events gets consult evaluation for free instead of inventing a parallel record.

Eval capture (A16)

evals.py pairs starts with completions into records, exports JSON lines, and summarises by model. Verdicts live outside the canonical log: the log records what happened, a verdict is a later judgement about it, and judgements get revised — so a good can become a bad without rewriting history. Schema v3 adds the outcomes table, which means a v1 store now traverses two migrations; the existing migration test was updated to assert that, making it a better exercise of the hook than it was.

An unpaired start (process died mid-delegation) is skipped rather than reported half-formed — a partial record would pollute the very set A16 exists to make trustworthy.

Executor integration

ask_expert is offered as a tool only when a desk is configured, in addition to the observable escalation triggers and never instead of them (A12). Two details worth noting: consulting is not mutating, so a completion claim backed only by consults is still challenged by the check added in 7b; and an exhausted budget comes back as a failed tool result rather than an exception, so the model learns it must stop asking rather than crashing the milestone.

Found live

Passing an empty message list and putting the question in the system prompt fails outright — the Messages API requires at least one non-system message, so the "stateless" consult was unsendable. The question now travels as a synthesised user message that is never appended to the log, which keeps the consult stateless while being a valid request. The unit test got better as a result: it now asserts the caller's repo context is absent and the question is present, rather than that nothing was sent at all.

23 new unit tests, 2 live. 245 unit + 17 integration green locally.

🤖 Generated with Claude Code

Completes step 7, and with it the dev agent's escalation story. ### `ask_expert` (A14), deliberately small One stateless call for a question needing no repo access. A14 warns against letting this grow into a second escalation subsystem, so it has no diff application, no session state, and no tools. - **Structured output is contracted at the prompt level**, since vendors differ on JSON enforcement — so the reply is parsed and validated against a local schema with one retry. It tolerates a fenced code block, because models wrap JSON in one habitually, but not missing or malformed fields. - **The budget is counted from the canonical log, not from memory.** It therefore survives a restart, and a crash cannot hand a circling executor a fresh allowance. Exhaustion *raises* rather than quietly answering: a caller that has spent its consults should get on with the work or escalate, which is the behaviour A14 is protecting. - **Consults are recorded as `delegation_started`/`delegation_completed` with role `consultant`.** A16 asks for a "consult/delegation eval set", so reusing the composite events gets consult evaluation for free instead of inventing a parallel record. ### Eval capture (A16) `evals.py` pairs starts with completions into records, exports JSON lines, and summarises by model. **Verdicts live outside the canonical log**: the log records what happened, a verdict is a later judgement about it, and judgements get revised — so a `good` can become a `bad` without rewriting history. Schema v3 adds the outcomes table, which means a v1 store now traverses *two* migrations; the existing migration test was updated to assert that, making it a better exercise of the hook than it was. An unpaired start (process died mid-delegation) is skipped rather than reported half-formed — a partial record would pollute the very set A16 exists to make trustworthy. ### Executor integration `ask_expert` is offered as a tool only when a desk is configured, **in addition to** the observable escalation triggers and never instead of them (A12). Two details worth noting: consulting is not mutating, so a completion claim backed only by consults is still challenged by the check added in 7b; and an exhausted budget comes back as a failed tool result rather than an exception, so the model learns it must stop asking rather than crashing the milestone. ### Found live Passing an empty message list and putting the question in the system prompt fails outright — the Messages API requires at least one non-system message, so the "stateless" consult was unsendable. The question now travels as a synthesised user message that is never appended to the log, which keeps the consult stateless while being a valid request. The unit test got better as a result: it now asserts the caller's repo context is absent *and* the question is present, rather than that nothing was sent at all. 23 new unit tests, 2 live. 245 unit + 17 integration green locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-08 19:48:32 +00:00

Add ask_expert and eval capture (A14/A16) — step 7c

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m7s

Details

2b7651fb37

Completes step 7.

- expert.py: one stateless call for a question needing no repo access.
  Structured output is contracted at the prompt level, since vendors
  differ on JSON enforcement, so the reply is parsed and validated
  locally with one retry — tolerating a fenced code block, because
  models wrap JSON habitually, but not missing fields
- The per-task budget is counted from the canonical log rather than
  memory, so it survives a restart and cannot hand a circling executor
  a fresh allowance. Exhaustion raises rather than answering: a caller
  out of consults should get on with the work or escalate
- Consults are recorded as delegation_started/completed with role
  consultant. A16 asks for a "consult/delegation eval set", and
  reusing the composite events gets consult evaluation for free rather
  than inventing a parallel record
- evals.py pairs starts with completions into eval records, with
  verdicts kept outside the canonical log: the log records what
  happened, a verdict is a later judgement about it, and judgements
  get revised. Schema v3 adds the outcomes table, so a v1 store now
  traverses two migrations — a better exercise of the hook than one
- ask_expert is offered to the executor as a tool only when a desk is
  configured, in addition to the observable escalation triggers and
  never instead of them (A12). Consulting is not mutating, so a
  completion claim backed only by consults is still challenged

Found live: passing an empty message list and putting the question in
the system prompt fails outright — the Messages API requires at least
one non-system message. The question now travels as a synthesised user
message that is never appended to the log, which keeps the consult
stateless while remaining a valid request.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dongar approved these changes

2026-09-08 19:53:26 +00:00

dongar merged commit 608a3a5bfb into main

2026-09-08 19:53:30 +00:00

dongar deleted branch ask-expert

2026-09-08 19:53:30 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#24

Add ask_expert and eval capture (A14/A16) — step 7c #24

ask_expert (A14), deliberately small

Eval capture (A16)

Executor integration

Found live

`ask_expert` (A14), deliberately small