M2.8: switch over to the new loop
nisse-dev now runs only the conversational loop. The gated state machine it was built alongside is gone: - Removed control.py, planner.py, executor.py, approvals.py, gates.py (command_form moves to loop/reader.py), publish.py and poller.py, the old assembly in agent.py (its shared helpers move to loop/build.py), the store's state-machine view (get_state, transition), their tests, the old end-to-end and live planning/milestone/gate tests, and specs/orchestrator.qnt with its scenarios. - Roles are driver, helper, expert and classifier; driver is required. loop:, roles.planner, roles.executor and tuning.executor are refused with what to use instead; tuning.driver takes the turn cap and failure limit. - The benchmark and the capability matrix measure the driver: the loop's own system prompt and tools, the task's plan as the agreed restatement. Task plans end with finish_milestone, so earlier runs show as (old); stored runs' executor tuning reads as the driver's. - A delegation on the loop records what its helper's alias resolved to (L9), as the old path's did. - The dump shows a work item's revision and what it waits for. - Behaviour the old tests covered and the loop shares stays covered: driver runs that fail on a model call, stop mid-turn, reset the failure count; go-aheads ignoring case and punctuation; a request too big to fit is never sent; the assembly tests, now on build_loop_agent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
1 parent
09232dcbdd
commit
6ae2375194
89 files changed
+951
-8592
No files matched your search
+44
-36
@@ -127,9 +127,10 @@ a local model can be held to less without affecting the frontier roles.
|
||||
response inside the context window. With a 32k context and a 16k ceiling,
|
||||
only 16k is left for the prompt. Some servers reject any request where
|
||||
prompt plus ceiling exceeds the context.
|
||||
- **Too low:** truncated plans and tool calls.
|
||||
- **Deciding:** executor turns are short, so 2–4k is plenty for a local
|
||||
executor. Frontier planning wants the large default. Leave this high, and
|
||||
- **Too low:** truncated restatements and tool calls.
|
||||
- **Deciding:** a working turn is short, so 2–4k is plenty for a local
|
||||
model doing a milestone's work. A frontier driver restating and designing
|
||||
wants the large default. Leave this high, and
|
||||
set a local model's smaller ceiling as its `max_output_tokens` in LiteLLM.
|
||||
|
||||
### `llm.request_timeout_s` — default 600
|
||||
@@ -137,8 +138,8 @@ a local model can be held to less without affecting the frontier roles.
|
||||
How long one model request may take before it is abandoned. It applies to
|
||||
every request, on both routes.
|
||||
|
||||
- **Too low:** a slow model's long answer is cut off, and the executor
|
||||
escalates work that was going well. At 120 s, the old fixed value, a dense
|
||||
- **Too low:** a slow model's long answer is cut off, and the driver's run
|
||||
fails on work that was going well. At 120 s, the old fixed value, a dense
|
||||
27B model on the RTX 4090 timed out writing one module in one turn.
|
||||
- **Too high:** a hung connection holds the milestone for that long before
|
||||
anything notices.
|
||||
@@ -147,10 +148,12 @@ every request, on both routes.
|
||||
expect, which is the model's output ceiling divided by its generation
|
||||
speed.
|
||||
|
||||
### `tuning.executor.max_turns` — default 40
|
||||
### `tuning.driver.max_turns` — default 40
|
||||
|
||||
How many model calls one milestone may make before the executor stops with
|
||||
`capped` and asks you what to do.
|
||||
How many model calls one driver run may make before it stops as `capped` and
|
||||
asks you what to do. A run lasts until the driver waits (for you, a review or
|
||||
a merge), so a long milestone is usually one run. Before M2.8 this was
|
||||
`tuning.executor.max_turns`.
|
||||
|
||||
- **Too low:** milestones that were making progress stop short. You'll see
|
||||
`capped` outcomes on work that was going well.
|
||||
@@ -167,12 +170,14 @@ How many model calls one milestone may make before the executor stops with
|
||||
`long-milestone`, one of which had in fact finished the work. It is a guard
|
||||
against a model trying forever, not a budget, so it was raised to 40.
|
||||
|
||||
### `tuning.executor.failure_limit` — default 2
|
||||
### `tuning.driver.failure_limit` — default 2
|
||||
|
||||
How many consecutive failed `run_command` calls trigger escalation: the
|
||||
sub-task is handed to a frontier model as a delegation (architecture L9). A
|
||||
successful command resets the count. Failed file reads don't count, since
|
||||
reading is looking around, not doing the work.
|
||||
How many consecutive failed `run_command` calls mean the driver is stuck: the
|
||||
first time in a milestone, a helper takes the problem as a delegation
|
||||
(architecture L9); the second time, the item waits on you. A successful
|
||||
command resets the count. Failed file reads don't count, since reading is
|
||||
looking around, not doing the work. Before M2.8 this was
|
||||
`tuning.executor.failure_limit`.
|
||||
|
||||
- **Why it's a count and not a judgment:** small models are poorly calibrated
|
||||
about their own competence. Asked "are you stuck?", they say no until far
|
||||
@@ -194,13 +199,13 @@ How many `ask_expert` calls one milestone may make. `ask_expert` is a single
|
||||
stateless question to a stronger model, with no access to the repository
|
||||
(architecture L9), for things like "is this API deprecated?".
|
||||
|
||||
- **Too low:** the executor can't get unstuck on a knowledge question it
|
||||
- **Too low:** the driver can't get unstuck on a knowledge question it
|
||||
can't answer from the code.
|
||||
- **Too high:** a confused executor asks variations of the same question
|
||||
- **Too high:** a confused driver asks variations of the same question
|
||||
instead of doing the work or failing. That's the circling L9 warns about.
|
||||
- **Deciding:** count consults per milestone, and check whether the answers
|
||||
changed what the executor did next. If it regularly exhausts the budget
|
||||
without progress, the budget is fine and the model or the plan is the
|
||||
changed what the driver did next. If it regularly exhausts the budget
|
||||
without progress, the budget is fine and the model or the brief is the
|
||||
problem.
|
||||
|
||||
### `tuning.delegation` — hard caps on one delegation
|
||||
@@ -282,10 +287,9 @@ Tokens are estimated from characters (pessimistically, at 3 per token) and
|
||||
the estimate is corrected from the prompt sizes the server reports. The
|
||||
correction only ever makes the estimate larger.
|
||||
|
||||
The executor also sends less in the first place: the approved milestone plan
|
||||
goes in its system prompt, which is never cut, and it sees only what happened
|
||||
since that plan. The planning conversation and earlier milestones stay in the
|
||||
log.
|
||||
Each role also sends less in the first place: it sees its own work item's
|
||||
transcript (L6), with other roles' work reduced to what it needs (a helper's
|
||||
handback is one report to the driver).
|
||||
|
||||
| Knob | Default | |
|
||||
|---|---|---|
|
||||
@@ -331,14 +335,15 @@ Asking a model that can't honour it can be worse than not asking: the
|
||||
Qwen3-14B turn above costs a full output ceiling of generation, where a plain
|
||||
prose reply would have cost a few hundred tokens.
|
||||
|
||||
Where it isn't enforced, the executor holds the contract itself:
|
||||
Where it isn't enforced, the driver's run holds the contract itself:
|
||||
|
||||
- **`finish` is the only way to end a milestone.** It is refused once if
|
||||
nothing has been changed yet.
|
||||
- **A prose reply is a violation.** The model is told once; a second escalates.
|
||||
- **A call to a tool that wasn't offered counts as a failure** toward
|
||||
`failure_limit`, like a failed command, so an inventing model escalates in a
|
||||
couple of turns instead of using up `max_turns`.
|
||||
- **A waiting tool is the only way to end a run** (`finish_milestone`,
|
||||
`ask_human`, and the like); the harness, not the model, then publishes or
|
||||
asks.
|
||||
- **A prose reply is a violation.** The model is told once; a second ends
|
||||
the run, and the item waits on you.
|
||||
- **A call to a tool that wasn't offered is answered, not run:** the result
|
||||
names the tools there are, so the model can put it right on its next turn.
|
||||
|
||||
The capability matrix checks the declaration: a model declared `true` that
|
||||
answers a required call in prose fails
|
||||
@@ -412,14 +417,17 @@ done.
|
||||
The first four were the original set. By September 2026 all seven newer local
|
||||
models passed all of them, so the last four were added to separate models that
|
||||
can hold the tool contract from models that can do a milestone's real work.
|
||||
Their plans stay explicit, as the planner is told to write them; what makes
|
||||
Their plans stay explicit, as an agreed restatement is; what makes
|
||||
them harder is the work itself: tracing a cause, searching a codebase,
|
||||
reading a spec closely, and holding a plan over many turns.
|
||||
|
||||
It runs the executor exactly as the agent does (the same client wiring,
|
||||
context budget and `tool_choice` handling, and your config's tuning knobs),
|
||||
with **delegation and `ask_expert` turned off**. An escalation is a failure of
|
||||
the model under test, not a frontier model's success.
|
||||
It runs the loop's own driver exactly as the agent does: the same system
|
||||
prompt, tools, client wiring, context budget, `tool_choice` handling and your
|
||||
config's tuning knobs. The task's plan is given as the agreed restatement, and
|
||||
finishing the milestone ends the run. **Delegation and `ask_expert` are
|
||||
turned off**: an escalation is a failure of the model under test, not a
|
||||
frontier model's success. Until M2.8 it ran the old loop's executor, so runs
|
||||
from before then are against older task versions and show as `(old)`.
|
||||
|
||||
**Evaluating a new model** is one command, once its LiteLLM entry declares its
|
||||
limits:
|
||||
@@ -456,7 +464,7 @@ served differently (different note) are kept apart.
|
||||
`failure_limit: 2` mid-recovery. That says as much about the knob as about
|
||||
the model. Tuning a knob and re-running is a fair comparison, because the
|
||||
tuning is recorded with every run.
|
||||
- **Cost** is per run, straight from LiteLLM, so a frontier model as executor
|
||||
- **Cost** is per run, straight from LiteLLM, so a frontier model as driver
|
||||
can be compared on success rate *and* dollars.
|
||||
|
||||
---
|
||||
@@ -511,8 +519,8 @@ Later work that will add knobs, each with its own section here:
|
||||
- **Compaction.** A model-written summary replacing old turns, instead of
|
||||
dropping them. The log already has an event for it; it needs a summarizing
|
||||
model and a policy for what is safe to summarize.
|
||||
- **Benchmarking the other roles:** planner and delegate, with delegation
|
||||
turned back on for comparisons across frontier tiers.
|
||||
- **Benchmarking the other roles:** the helper, with delegation turned back
|
||||
on for comparisons across frontier tiers.
|
||||
|
||||
### `tuning.loop.bug_grace_s` — default 600
|
||||
|
||||
|
||||
Reference in new issue
Block a user