M2.8: switch over to the new loop
nisse-dev now runs only the conversational loop. The gated state machine it was built alongside is gone: - Removed control.py, planner.py, executor.py, approvals.py, gates.py (command_form moves to loop/reader.py), publish.py and poller.py, the old assembly in agent.py (its shared helpers move to loop/build.py), the store's state-machine view (get_state, transition), their tests, the old end-to-end and live planning/milestone/gate tests, and specs/orchestrator.qnt with its scenarios. - Roles are driver, helper, expert and classifier; driver is required. loop:, roles.planner, roles.executor and tuning.executor are refused with what to use instead; tuning.driver takes the turn cap and failure limit. - The benchmark and the capability matrix measure the driver: the loop's own system prompt and tools, the task's plan as the agreed restatement. Task plans end with finish_milestone, so earlier runs show as (old); stored runs' executor tuning reads as the driver's. - A delegation on the loop records what its helper's alias resolved to (L9), as the old path's did. - The dump shows a work item's revision and what it waits for. - Behaviour the old tests covered and the loop shares stays covered: driver runs that fail on a model call, stop mid-turn, reset the failure count; go-aheads ignoring case and punctuation; a request too big to fit is never sent; the assembly tests, now on build_loop_agent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
1 parent
09232dcbdd
commit
6ae2375194
89 files changed
+951
-8592
No files matched your search
@@ -1,4 +1,4 @@
|
||||
"""Benchmark executor models on canned milestones, and compare the results.
|
||||
"""Benchmark driver models on canned milestones, and compare the results.
|
||||
|
||||
uv run python scripts/benchmark.py run --models A,B [--runs 5] [--tasks T,...]
|
||||
[--note "vLLM 0.29, AWQ, fp8 KV"] [--config PATH]
|
||||
@@ -14,13 +14,13 @@ import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from nisse_dev.agent import build_host, make_llm_clients
|
||||
from nisse_dev.bench.results import append, load, render, results_file, summarise
|
||||
from nisse_dev.bench.runner import agent_commit, run_task
|
||||
from nisse_dev.bench.tasks import load_tasks
|
||||
from nisse_dev.catalog import ModelCatalog
|
||||
from nisse_dev.config import AgentConfig, ConfigError
|
||||
from nisse_dev.envfile import load_env_file
|
||||
from nisse_dev.loop.build import build_host, make_llm_clients
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
RESULTS = REPO / "bench" / "results"
|
||||
@@ -46,7 +46,8 @@ def run(args: argparse.Namespace) -> int:
|
||||
catalog = ModelCatalog.discover(config.llm_base_url, config.llm_token)
|
||||
catalog.require_limits({m: m for m in models})
|
||||
normalized, _passthrough = make_llm_clients(config, catalog)
|
||||
host = build_host(config) # runs happen on the build VM, as the agent's do
|
||||
# Runs happen on the first project's build VM, as the agent's do.
|
||||
host = build_host(config, config.projects[0])
|
||||
commit = agent_commit(REPO)
|
||||
|
||||
for model in models:
|
||||
@@ -54,7 +55,7 @@ def run(args: argparse.Namespace) -> int:
|
||||
out = results_file(args.results, model)
|
||||
for task in tasks:
|
||||
for n in range(args.runs):
|
||||
# The executor always takes the normalized route in the agent,
|
||||
# The driver always takes the normalized route in the agent,
|
||||
# so it does here too.
|
||||
record = run_task(
|
||||
task,
|
||||
|
||||
Reference in new issue
Block a user