dongarandClaude Opus 5.5
089af0c391
What the fast path calls unknown is judged by an independent model call on the item's classifier role (L4), before it runs. - nisse_dev.loop.classifier: it sees the action and the agreed scope (project, repository, issue number, the restatement the operator said go to), built in code; never the transcript, the issue's text, tool output or the driver's reasoning. Its verdict is a tool call: safe runs; in_scope runs and is reported in the thread; needs_discussion asks, as a dangerous action does. An error, no verdict or an unknown one fails closed. Verdicts are cached per item, agreed revision and action. A helper is refused what needs discussion. - action_verdict gains source, model and the classifier's exact input; details shows the source. An allow after the classifier asked is an override. scripts/classifier-evals.py exports the judgements from a store, read-only; tests/integration/classifier_cases.yaml is the seed set, run per model with --capability-models. - The fast path's delete rule: deleting on the build VM is contained; only what breaks the host (system and home directories) stays dangerous, since in tests the build VM is the test machine. - The classifier role is the local model in the example config. - The property walk gains a classifier answering at random, junk and failures included; dangerous_only_when_allowed now covers its verdicts (checked against a mutant that runs them all). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
62 lines
1.9 KiB
Python
62 lines
1.9 KiB
Python
"""Export the action classifier's judgements from a state store (L4, M3.2).
|
|
|
|
uv run python scripts/classifier-evals.py STATE_DB [...] > cases.jsonl
|
|
uv run python scripts/classifier-evals.py STATE_DB --overridden
|
|
|
|
One JSON line per distinct judgement: exactly what the classifier was shown,
|
|
its verdict and reason, the model, what happened (ran, asked, allowed,
|
|
refused), and whether the operator overrode it. Overridden cases are the
|
|
ones worth adding to tests/integration/classifier_cases.yaml, with the
|
|
verdict the operator's choice implies.
|
|
|
|
The store is opened read-only, so a live agent's database is safe to read.
|
|
"""
|
|
|
|
import argparse
|
|
import json
|
|
import sqlite3
|
|
import sys
|
|
from dataclasses import asdict
|
|
from pathlib import Path
|
|
|
|
from nisse_dev.canonical import StoredEvent, from_payload
|
|
from nisse_dev.evals import classifier_cases
|
|
|
|
|
|
def events(path: Path) -> list[StoredEvent]:
|
|
conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
|
|
rows = conn.execute(
|
|
"SELECT id, ts, type, payload, work_item, actor FROM canonical_events "
|
|
"WHERE type = 'action_verdict' ORDER BY id"
|
|
).fetchall()
|
|
conn.close()
|
|
return [
|
|
StoredEvent(
|
|
id=row[0],
|
|
ts=row[1],
|
|
event=from_payload(row[2], json.loads(row[3])),
|
|
work_item=row[4],
|
|
actor=row[5],
|
|
)
|
|
for row in rows
|
|
]
|
|
|
|
|
|
def main() -> int:
|
|
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
|
parser.add_argument("stores", nargs="+", type=Path)
|
|
parser.add_argument(
|
|
"--overridden", action="store_true", help="only cases the operator overrode"
|
|
)
|
|
args = parser.parse_args()
|
|
for store in args.stores:
|
|
for case in classifier_cases(events(store)):
|
|
if args.overridden and not case.overridden:
|
|
continue
|
|
print(json.dumps({"store": str(store), **asdict(case)}))
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|