dongarandClaude Opus 5.5
089af0c391
What the fast path calls unknown is judged by an independent model call on the item's classifier role (L4), before it runs. - nisse_dev.loop.classifier: it sees the action and the agreed scope (project, repository, issue number, the restatement the operator said go to), built in code; never the transcript, the issue's text, tool output or the driver's reasoning. Its verdict is a tool call: safe runs; in_scope runs and is reported in the thread; needs_discussion asks, as a dangerous action does. An error, no verdict or an unknown one fails closed. Verdicts are cached per item, agreed revision and action. A helper is refused what needs discussion. - action_verdict gains source, model and the classifier's exact input; details shows the source. An allow after the classifier asked is an override. scripts/classifier-evals.py exports the judgements from a store, read-only; tests/integration/classifier_cases.yaml is the seed set, run per model with --capability-models. - The fast path's delete rule: deleting on the build VM is contained; only what breaks the host (system and home directories) stays dangerous, since in tests the build VM is the test machine. - The classifier role is the local model in the example config. - The property walk gains a classifier answering at random, junk and failures included; dangerous_only_when_allowed now covers its verdicts (checked against a mutant that runs them all). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
56 lines
1.8 KiB
Python
56 lines
1.8 KiB
Python
"""The action classifier against its evaluation set (L4, M3.2), per model.
|
|
|
|
Opt in by naming the models, as for the capability matrix:
|
|
|
|
uv run pytest -m integration tests/integration/test_classifier_cases.py \
|
|
--capability-models=qwen3.8-27b-nothink
|
|
|
|
Skipped otherwise. Each case passes when the model's verdict is one of those
|
|
the case allows; a case that fails is a model to watch, or a case to rethink.
|
|
"""
|
|
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import pytest
|
|
import yaml
|
|
|
|
from nisse_core.actions import Action
|
|
from nisse_dev.llm import LiteLLMClient
|
|
from nisse_dev.loop.classifier import ActionClassifier, Scope
|
|
|
|
pytestmark = pytest.mark.integration
|
|
|
|
CASES: list[dict[str, Any]] = yaml.safe_load(
|
|
(Path(__file__).parent / "classifier_cases.yaml").read_text()
|
|
)
|
|
|
|
|
|
def pytest_generate_tests(metafunc: pytest.Metafunc) -> None:
|
|
if "model" not in metafunc.fixturenames:
|
|
return
|
|
raw = metafunc.config.getoption("--capability-models")
|
|
models = [m.strip() for m in raw.split(",") if m.strip()]
|
|
if not models:
|
|
metafunc.parametrize(
|
|
"model",
|
|
[
|
|
pytest.param(
|
|
None, marks=pytest.mark.skip(reason="pass --capability-models")
|
|
)
|
|
],
|
|
)
|
|
return
|
|
metafunc.parametrize("model", models)
|
|
|
|
|
|
@pytest.mark.parametrize("case", CASES, ids=[c["name"] for c in CASES])
|
|
def test_the_classifier_judges_the_case(
|
|
model: str, case: dict[str, Any], litellm: LiteLLMClient
|
|
) -> None:
|
|
classifier = ActionClassifier(litellm, model)
|
|
scope = Scope(project="demo", repo="dev/demo", issue=7, agreed=case["agreed"])
|
|
action = Action("run_command", {"command": case["command"]})
|
|
verdict, _ = classifier.judge(action, scope, "/srv/work/demo")
|
|
assert verdict.rule in case["expect"], f"{model}: {verdict.rule}: {verdict.reason}"
|