dongarandClaude Opus 5.5
6ae2375194
nisse-dev now runs only the conversational loop. The gated state machine it was built alongside is gone: - Removed control.py, planner.py, executor.py, approvals.py, gates.py (command_form moves to loop/reader.py), publish.py and poller.py, the old assembly in agent.py (its shared helpers move to loop/build.py), the store's state-machine view (get_state, transition), their tests, the old end-to-end and live planning/milestone/gate tests, and specs/orchestrator.qnt with its scenarios. - Roles are driver, helper, expert and classifier; driver is required. loop:, roles.planner, roles.executor and tuning.executor are refused with what to use instead; tuning.driver takes the turn cap and failure limit. - The benchmark and the capability matrix measure the driver: the loop's own system prompt and tools, the task's plan as the agreed restatement. Task plans end with finish_milestone, so earlier runs show as (old); stored runs' executor tuning reads as the driver's. - A delegation on the loop records what its helper's alias resolved to (L9), as the old path's did. - The dump shows a work item's revision and what it waits for. - Behaviour the old tests covered and the loop shares stays covered: driver runs that fail on a model call, stop mid-turn, reset the failure count; go-aheads ignoring case and punctuation; a request too big to fit is never sent; the assembly tests, now on build_loop_agent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
18 lines
912 B
Python
18 lines
912 B
Python
"""The driver benchmark: canned milestones, run repeatedly per model.
|
|
|
|
Choosing a driver model, and later tuning its knobs, should rest on
|
|
measured success rates, not on one run or on reputation. Each task is a fixed
|
|
approved plan over a seed project, with a hidden check the model never sees;
|
|
success means the check passes, not that the model said it finished. Runs use
|
|
the agent's own client wiring (context budget, declared limits, tool_choice
|
|
only where declared), so what is measured is what the agent would do.
|
|
|
|
Results are stored, one JSON line per run, with everything needed to compare
|
|
them later: the model's identity and declared facts, a note on how it was
|
|
served, the task's content hash, the agent's commit, and its tuning. A new
|
|
model then needs only its own runs to be compared with everything measured
|
|
before it.
|
|
|
|
See scripts/benchmark.py, and docs/Tuning-Guide.md for how to read results.
|
|
"""
|