M3.2: the action classifier

What the fast path calls unknown is judged by an independent model call
on the item's classifier role (L4), before it runs.

- nisse_dev.loop.classifier: it sees the action and the agreed scope
  (project, repository, issue number, the restatement the operator said
  go to), built in code; never the transcript, the issue's text, tool
  output or the driver's reasoning. Its verdict is a tool call: safe
  runs; in_scope runs and is reported in the thread; needs_discussion
  asks, as a dangerous action does. An error, no verdict or an unknown
  one fails closed. Verdicts are cached per item, agreed revision and
  action. A helper is refused what needs discussion.
- action_verdict gains source, model and the classifier's exact input;
  details shows the source. An allow after the classifier asked is an
  override. scripts/classifier-evals.py exports the judgements from a
  store, read-only; tests/integration/classifier_cases.yaml is the seed
  set, run per model with --capability-models.
- The fast path's delete rule: deleting on the build VM is contained;
  only what breaks the host (system and home directories) stays
  dangerous, since in tests the build VM is the test machine.
- The classifier role is the local model in the example config.
- The property walk gains a classifier answering at random, junk and
  failures included; dangerous_only_when_allowed now covers its
  verdicts (checked against a mutant that runs them all).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

nisse-dev/src/nisse_dev/evals.py

@@ -11,10 +11,16 @@ verdict is a later judgement about it, and judgements get revised.
 """
 
 import json
+from collections.abc import Iterable
 from dataclasses import asdict, dataclass
 from typing import Any
 
-from nisse_dev.canonical import DelegationCompleted, DelegationStarted
+from nisse_dev.canonical import (
+    ActionVerdict,
+    DelegationCompleted,
+    DelegationStarted,
+    StoredEvent,
+)
 from nisse_dev.state import StateStore
 
 UNMARKED = "unmarked"
@@ -86,3 +92,43 @@ def summarise(records: list[EvalRecord]) -> dict[str, Any]:
         bucket["total"] += 1
         bucket[record.verdict if record.verdict in ("good", "bad") else UNMARKED] += 1
     return by_model
+
+
+# --- the action classifier's evaluation set (L4, M3.2) -----------------------
+
+
+@dataclass
+class ClassifierCase:
+    """One judgement, as the classifier saw it, with what then happened:
+    ``overridden`` when the operator allowed what it wanted discussed."""
+
+    input: dict[str, Any]  # exactly what the classifier was shown
+    verdict: str  # safe | in_scope | needs_discussion | failed_closed
+    reason: str
+    model: str
+    decisions: list[str]
+    overridden: bool = False
+
+
+def classifier_cases(events: Iterable[StoredEvent]) -> list[ClassifierCase]:
+    """Every distinct classifier judgement in the log, in order. A verdict
+    re-used from the cache adds its decision to the same case."""
+    cases: dict[tuple[str, str], ClassifierCase] = {}
+    for stored in events:
+        v = stored.event
+        if not isinstance(v, ActionVerdict) or v.source != "classifier":
+            continue
+        key = (v.input, v.rule)
+        if key not in cases:
+            cases[key] = ClassifierCase(
+                input=json.loads(v.input) if v.input else {},
+                verdict=v.rule,
+                reason=v.reason,
+                model=v.model,
+                decisions=[],
+            )
+        case = cases[key]
+        case.decisions.append(v.decision)
+        if v.decision == "allowed":
+            case.overridden = True
+    return list(cases.values())