Add ask_expert and eval capture (A14/A16) — step 7c
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m7s

Completes step 7.

- expert.py: one stateless call for a question needing no repo access.
  Structured output is contracted at the prompt level, since vendors
  differ on JSON enforcement, so the reply is parsed and validated
  locally with one retry — tolerating a fenced code block, because
  models wrap JSON habitually, but not missing fields
- The per-task budget is counted from the canonical log rather than
  memory, so it survives a restart and cannot hand a circling executor
  a fresh allowance. Exhaustion raises rather than answering: a caller
  out of consults should get on with the work or escalate
- Consults are recorded as delegation_started/completed with role
  consultant. A16 asks for a "consult/delegation eval set", and
  reusing the composite events gets consult evaluation for free rather
  than inventing a parallel record
- evals.py pairs starts with completions into eval records, with
  verdicts kept outside the canonical log: the log records what
  happened, a verdict is a later judgement about it, and judgements
  get revised. Schema v3 adds the outcomes table, so a v1 store now
  traverses two migrations — a better exercise of the hook than one
- ask_expert is offered to the executor as a tool only when a desk is
  configured, in addition to the observable escalation triggers and
  never instead of them (A12). Consulting is not mutating, so a
  completion claim backed only by consults is still challenged

Found live: passing an empty message list and putting the question in
the system prompt fails outright — the Messages API requires at least
one non-system message. The question now travels as a synthesised user
message that is never appended to the log, which keeps the consult
stateless while remaining a valid request.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5 committed 2026-09-08 19:48:13 +00:00
1 parent dd22eda528
commit 2b7651fb37
9 files changed
+846 -8

No files matched your search

+5 -2
View File
@@ -16,7 +16,7 @@ from nisse_dev.canonical import (
from_payload,
mint_call_id,
)
from nisse_dev.state import StateStore
from nisse_dev.state import CURRENT_SCHEMA_VERSION, StateStore
ALL_EVENTS = [
UserMessage(text="use demo", source="matrix:$abc"),
@@ -116,9 +116,12 @@ def test_default_migration_upgrades_a_real_v1_store(tmp_path: Path) -> None:
conn.close()
with StateStore(db) as store:
assert store.schema_version == 2
assert store.schema_version == CURRENT_SCHEMA_VERSION
state = store.get_state()
assert state is not None
assert state.state == "IDLE" # v1 data intact
store.append_event(UserMessage(text="hello", source="m:1"))
assert len(store.read_events()) == 1
# Two migrations were traversed, so both later tables must exist.
store.mark_outcome(1, "good", "arrived via v1 -> v3")
assert store.outcomes()[1] == ("good", "arrived via v1 -> v3")