dongarandClaude Opus 5.5
83804fe02a
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m14s
Development Plan M1.1, as discussed. - Schema v4: a work_items table (id, scope, title, status, outcome, position, data) and an agent row whose active_item points at one (NULL: idle). Canonical events and intents carry their work item, and events their actor. Older stores are refused with a clear message: there was no production data, so no migration (the mechanism stays). - The log is reached through a Transcript bound to one work item and one actor: store.transcript(item, actor) reads only that item and writes with both. The planner, executor, delegation and expert desk get one per call from the orchestrator instead of holding the store, so a component can neither read another task's events (the planner read every earlier task's, #78) nor write without attribution. store.all_events() is kept for readers that span items (evaluation capture, dumps). Intents are begun through the transcript or with work_item=, so each belongs to its item. - The orchestrator opens a work item on 'use', closes it with its outcome (completed or abandoned), and carries what the operator said while idle into the new item as copies with their origin, so a task sent before 'use' is still planned from (D1). control.py keeps its state machine through get_state()/transition(), now over the active work item, until M2.6 retires it. - nisse_dev.dump shows work items and each event's item and actor. Tests: test_work_items.py (scoping, attribution, intent ownership, restart, outcomes, carried messages, a second task never seeing the first); the suites updated for the per-call transcript. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
89 lines
3.1 KiB
Python
89 lines
3.1 KiB
Python
"""Delegation and consult evaluation capture (architecture L9).
|
|
|
|
Each ``delegation_started`` / ``delegation_completed`` pair is already a clean
|
|
input-output record; this pairs them up and attaches a verdict, so "best model
|
|
this week" can be decided by re-running a captured set rather than by benchmark
|
|
chatter. Without it, swapping the frontier tier becomes something to avoid,
|
|
because the effect is unmeasurable.
|
|
|
|
Verdicts live outside the canonical log: the log records what happened, a
|
|
verdict is a later judgement about it, and judgements get revised.
|
|
"""
|
|
|
|
import json
|
|
from dataclasses import asdict, dataclass
|
|
from typing import Any
|
|
|
|
from nisse_dev.canonical import DelegationCompleted, DelegationStarted
|
|
from nisse_dev.state import StateStore
|
|
|
|
UNMARKED = "unmarked"
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class EvalRecord:
|
|
event_id: int # of the completion, which is what a verdict attaches to
|
|
role: str
|
|
model: str
|
|
brief: str
|
|
caps: dict[str, Any]
|
|
handback: str
|
|
artifacts: list[str]
|
|
tokens: int
|
|
duration_s: float
|
|
exit_status: str
|
|
verdict: str = UNMARKED
|
|
note: str = ""
|
|
|
|
|
|
def collect(store: StateStore, after: int = 0) -> list[EvalRecord]:
|
|
"""Pair each start with the completion that follows it.
|
|
|
|
A start with no completion is skipped rather than reported half-formed:
|
|
it means the process died mid-delegation, which L8 handles, and an
|
|
unfinished record would pollute the eval set.
|
|
"""
|
|
verdicts = store.outcomes()
|
|
records: list[EvalRecord] = []
|
|
pending: tuple[int, DelegationStarted] | None = None
|
|
for stored in store.all_events(after=after):
|
|
if isinstance(stored.event, DelegationStarted):
|
|
pending = (stored.id, stored.event)
|
|
elif isinstance(stored.event, DelegationCompleted) and pending is not None:
|
|
_, started = pending
|
|
verdict, note = verdicts.get(stored.id, (UNMARKED, ""))
|
|
records.append(
|
|
EvalRecord(
|
|
event_id=stored.id,
|
|
role=started.role,
|
|
model=started.model,
|
|
brief=started.brief,
|
|
caps=started.caps,
|
|
handback=stored.event.handback,
|
|
artifacts=stored.event.artifacts,
|
|
tokens=stored.event.tokens,
|
|
duration_s=stored.event.duration_s,
|
|
exit_status=stored.event.exit_status,
|
|
verdict=verdict,
|
|
note=note,
|
|
)
|
|
)
|
|
pending = None
|
|
return records
|
|
|
|
|
|
def to_jsonl(records: list[EvalRecord]) -> str:
|
|
"""JSON lines, so a captured set can be diffed and re-run per record."""
|
|
return "\n".join(json.dumps(asdict(r)) for r in records)
|
|
|
|
|
|
def summarise(records: list[EvalRecord]) -> dict[str, Any]:
|
|
by_model: dict[str, dict[str, int]] = {}
|
|
for record in records:
|
|
bucket = by_model.setdefault(
|
|
record.model, {"total": 0, "good": 0, "bad": 0, UNMARKED: 0}
|
|
)
|
|
bucket["total"] += 1
|
|
bucket[record.verdict if record.verdict in ("good", "bad") else UNMARKED] += 1
|
|
return by_model
|