M2.8: switch over to the new loop
nisse-dev now runs only the conversational loop. The gated state machine it was built alongside is gone: - Removed control.py, planner.py, executor.py, approvals.py, gates.py (command_form moves to loop/reader.py), publish.py and poller.py, the old assembly in agent.py (its shared helpers move to loop/build.py), the store's state-machine view (get_state, transition), their tests, the old end-to-end and live planning/milestone/gate tests, and specs/orchestrator.qnt with its scenarios. - Roles are driver, helper, expert and classifier; driver is required. loop:, roles.planner, roles.executor and tuning.executor are refused with what to use instead; tuning.driver takes the turn cap and failure limit. - The benchmark and the capability matrix measure the driver: the loop's own system prompt and tools, the task's plan as the agreed restatement. Task plans end with finish_milestone, so earlier runs show as (old); stored runs' executor tuning reads as the driver's. - A delegation on the loop records what its helper's alias resolved to (L9), as the old path's did. - The dump shows a work item's revision and what it waits for. - Behaviour the old tests covered and the loop shares stays covered: driver runs that fail on a model call, stop mid-turn, reset the failure count; go-aheads ignoring case and punctuation; a request too big to fit is never sent; the assembly tests, now on build_loop_agent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
1 parent
09232dcbdd
commit
6ae2375194
89 files changed
+951
-8592
No files matched your search
+28
-18
@@ -24,8 +24,8 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
|
||||
| Section | Status | Notes |
|
||||
|---|---|---|
|
||||
| **L1** Work model | mostly done | Work items with an active pointer (M1.1); issue intake (M2.1); queue, parking and blocked items (M2.2) on the new loop; no grace period yet |
|
||||
| **L2** Agent loop | partial | `control.py`, a fixed state machine with no-stranding, `stop`, `abandon`. Replaced by the new loop (Phase 2) |
|
||||
| **L3** Authority | partial | Revision binding and arrival stamping (`gates.py`); scope is still plan approval, not the issue and its conversation |
|
||||
| **L2** Agent loop | mostly done | The loop (Phase 2), the only one since the switch-over (M2.8): driver runs, waits, `stop`, `abandon`, question runs |
|
||||
| **L3** Authority | partial | Go-aheads bound to the restatement's revision, on the issue and its conversation (M2.1); no action classes yet (M3.1) |
|
||||
| **L4** Classifier | none | |
|
||||
| **L5** Conversation | partial | Matrix transport, mentions, lenient commands; on the new loop, a thread per item, named and ambiguous commands, plain-language go-aheads (M2.3); little explanation |
|
||||
| **L6** Context | partial | Context fitting, per-role views and orchestrator facts (M1.2); no scope or agent memory |
|
||||
@@ -41,7 +41,7 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
|
||||
| **D5** Review | partial | PR conversation comments only (#69), watch stall (#70) |
|
||||
| **D7–D8** Deployment, projects | partial | Runs from a venv as a systemd unit on this box; project entries in config (M1.4); egress not enforced |
|
||||
| **O**, **X** | none | No container image yet |
|
||||
| `specs/` | done for now | `orchestrator.qnt` (current loop), `loop.qnt` (the architecture, #91) |
|
||||
| `specs/` | done for now | `loop.qnt` (the architecture, #91); `orchestrator.qnt` went with `control.py` (M2.8) |
|
||||
|
||||
---
|
||||
|
||||
@@ -51,8 +51,9 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
|
||||
Authelia. The infrastructure repository exists in Gitea.
|
||||
- **Local inference:** llama-swap on an RTX 4090 host (320 W cap). The benchmark
|
||||
(PR #55) chose `qwen3.8-27b` for local work.
|
||||
- **Models by role today:** executor `qwen3.8-27b`, planner and helper
|
||||
`claude-opus-5-5`, expert `claude-haiku-4-5`. Phase 2 moves to driver and helpers.
|
||||
- **Models by role today:** driver and helper `claude-opus-5-5`, expert
|
||||
`claude-haiku-4-5`. `qwen3.8-27b` has had no role since the executor's went
|
||||
(M2.8); the driver benchmark measures where it fits.
|
||||
- **Bring-up** runs on this development VM (`scripts/bringup.sh`), against test
|
||||
resources: a test repository, the test Matrix room, a budget-capped LiteLLM key;
|
||||
configuration in `/root/nisse-e2e/`.
|
||||
@@ -85,11 +86,9 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
|
||||
(`-k`) or diagnosed (`--lf`); `scripts/e2e.sh` reports what a run spent. Manual
|
||||
live runs are for the new loop's first runs and phase ends. Findings are filed as
|
||||
issues and either fixed before moving on or deliberately deferred.
|
||||
- **The Quint models move with the loop:** `orchestrator.qnt` while `control.py`
|
||||
runs, `loop.qnt` for the new loop, whose properties become tests as each piece
|
||||
lands (Phase 2).
|
||||
- **`control.py` gets fixes only,** and only for what blocks a live run. New
|
||||
behaviour goes into the new loop.
|
||||
- **The Quint model moves with the loop:** `loop.qnt` models the architecture,
|
||||
and its properties are tests (the property walk), changed in the same PR as a
|
||||
rule they cover.
|
||||
- **Bring-up runs on this box as root,** against test resources only, with today's
|
||||
gates on until the classifier and trust dial are proven (Phase 3).
|
||||
- **This plan is revised as we go,** in the same PR as the work that changes it.
|
||||
@@ -230,8 +229,8 @@ deletes afterwards.
|
||||
**M2.6 Roles and capability profiles.** *Pre-approved.* **Done** (this PR).
|
||||
`driver`, `helper`, `expert`, `classifier` (L9); escalation triggers; delegation and
|
||||
`ask_expert` on the new loop.
|
||||
- `roles.driver` (unset: the planner's model) and `roles.classifier` (unset: the
|
||||
expert's), pinned per item; the driver and the go-ahead reader run on the item's.
|
||||
- `roles.driver` (required since M2.8; until then, unset meant the planner's
|
||||
model) and `roles.classifier` (unset: the expert's), pinned per item; the driver and the go-ahead reader run on the item's.
|
||||
- `profiles:` in configuration (tools, strengths, weaknesses), used in briefings:
|
||||
the driver is told what its helper and expert are good at, the helper who will
|
||||
read its handback; never a model's name.
|
||||
@@ -264,9 +263,8 @@ in prose, asking mid-work (L5, #42, #77).
|
||||
defaults, secrets by name with their visibility, egress additions.
|
||||
- Acceptance: the bring-up configuration is one project entry; a second project
|
||||
needs only a second entry.
|
||||
- `projects: {name: entry}`. The old loop runs exactly one project and refuses
|
||||
more. On the new loop each project has its own clone, workspace, push credential
|
||||
and milestones. Issues are named `project#14` when two projects could both have
|
||||
- `projects: {name: entry}`. Each project has its own clone, workspace, push
|
||||
credential and milestones. Issues are named `project#14` when two projects could both have
|
||||
`#14`, and a bare `#14` that matches two is asked about. Switching projects parks
|
||||
the one worked on last.
|
||||
- Secrets are by `${VAR}` reference only, `commands` or `harness`. `commands`
|
||||
@@ -295,8 +293,20 @@ the release's own tests over it).
|
||||
- The live bring-up's switch to the image waits for the next live test (it needs
|
||||
its own test repository too); `bringup.sh plan` shows what it will run.
|
||||
|
||||
**M2.8 Switch over.** *Pre-approved, after acceptance.* `nisse-dev` runs the new
|
||||
loop; `control.py`, its tests and `orchestrator.qnt` are removed.
|
||||
**M2.8 Switch over.** *Pre-approved, after acceptance.* **Done** (this PR, after
|
||||
the live session's fixes in #119 and #122). `nisse-dev` runs only the loop.
|
||||
- Removed: `control.py`, `planner.py`, `executor.py`, `approvals.py`, `gates.py`,
|
||||
`publish.py`, `poller.py`, the old assembly (`agent.py`, folded into
|
||||
`loop/build.py`), the state store's state-machine view, their tests, the old
|
||||
end-to-end and live planning tests, and `orchestrator.qnt` with its scenarios.
|
||||
- Configuration: `loop:`, `roles.planner`, `roles.executor` and
|
||||
`tuning.executor` are refused with what to use instead; `roles.driver` is
|
||||
required, and `tuning.driver` takes the driver's turn cap and failure limit.
|
||||
- The benchmark and the capability matrix measure the driver: the loop's own
|
||||
system prompt and tools, the task's plan as the agreed restatement. Task plans
|
||||
end with `finish_milestone`, so earlier runs show as `(old)`.
|
||||
- A delegation on the loop records what its helper's alias resolved to (L9), as
|
||||
the old path's did.
|
||||
|
||||
### Phase 3 — Authority
|
||||
|
||||
@@ -392,7 +402,7 @@ if the implementation differs.
|
||||
lines, then drop the oldest events with a note saying so; never the system
|
||||
prompt, the current plan or the most recent turns. View labels:
|
||||
`[orchestrator]`, `[operator, Matrix]`, `[operator, comment on PR #N]`,
|
||||
`[helper report]`, `[executor]` narration.
|
||||
`[helper report]`, `[driver]` narration.
|
||||
- **L7 rendering:** content is stored as bytes or local paths and uploaded or
|
||||
inlined at render time, never as provider handles. Token budgeting and
|
||||
prompt-cache breakpoints are render-time concerns, never stored. Compaction
|
||||
|
||||
+44
-36
@@ -127,9 +127,10 @@ a local model can be held to less without affecting the frontier roles.
|
||||
response inside the context window. With a 32k context and a 16k ceiling,
|
||||
only 16k is left for the prompt. Some servers reject any request where
|
||||
prompt plus ceiling exceeds the context.
|
||||
- **Too low:** truncated plans and tool calls.
|
||||
- **Deciding:** executor turns are short, so 2–4k is plenty for a local
|
||||
executor. Frontier planning wants the large default. Leave this high, and
|
||||
- **Too low:** truncated restatements and tool calls.
|
||||
- **Deciding:** a working turn is short, so 2–4k is plenty for a local
|
||||
model doing a milestone's work. A frontier driver restating and designing
|
||||
wants the large default. Leave this high, and
|
||||
set a local model's smaller ceiling as its `max_output_tokens` in LiteLLM.
|
||||
|
||||
### `llm.request_timeout_s` — default 600
|
||||
@@ -137,8 +138,8 @@ a local model can be held to less without affecting the frontier roles.
|
||||
How long one model request may take before it is abandoned. It applies to
|
||||
every request, on both routes.
|
||||
|
||||
- **Too low:** a slow model's long answer is cut off, and the executor
|
||||
escalates work that was going well. At 120 s, the old fixed value, a dense
|
||||
- **Too low:** a slow model's long answer is cut off, and the driver's run
|
||||
fails on work that was going well. At 120 s, the old fixed value, a dense
|
||||
27B model on the RTX 4090 timed out writing one module in one turn.
|
||||
- **Too high:** a hung connection holds the milestone for that long before
|
||||
anything notices.
|
||||
@@ -147,10 +148,12 @@ every request, on both routes.
|
||||
expect, which is the model's output ceiling divided by its generation
|
||||
speed.
|
||||
|
||||
### `tuning.executor.max_turns` — default 40
|
||||
### `tuning.driver.max_turns` — default 40
|
||||
|
||||
How many model calls one milestone may make before the executor stops with
|
||||
`capped` and asks you what to do.
|
||||
How many model calls one driver run may make before it stops as `capped` and
|
||||
asks you what to do. A run lasts until the driver waits (for you, a review or
|
||||
a merge), so a long milestone is usually one run. Before M2.8 this was
|
||||
`tuning.executor.max_turns`.
|
||||
|
||||
- **Too low:** milestones that were making progress stop short. You'll see
|
||||
`capped` outcomes on work that was going well.
|
||||
@@ -167,12 +170,14 @@ How many model calls one milestone may make before the executor stops with
|
||||
`long-milestone`, one of which had in fact finished the work. It is a guard
|
||||
against a model trying forever, not a budget, so it was raised to 40.
|
||||
|
||||
### `tuning.executor.failure_limit` — default 2
|
||||
### `tuning.driver.failure_limit` — default 2
|
||||
|
||||
How many consecutive failed `run_command` calls trigger escalation: the
|
||||
sub-task is handed to a frontier model as a delegation (architecture L9). A
|
||||
successful command resets the count. Failed file reads don't count, since
|
||||
reading is looking around, not doing the work.
|
||||
How many consecutive failed `run_command` calls mean the driver is stuck: the
|
||||
first time in a milestone, a helper takes the problem as a delegation
|
||||
(architecture L9); the second time, the item waits on you. A successful
|
||||
command resets the count. Failed file reads don't count, since reading is
|
||||
looking around, not doing the work. Before M2.8 this was
|
||||
`tuning.executor.failure_limit`.
|
||||
|
||||
- **Why it's a count and not a judgment:** small models are poorly calibrated
|
||||
about their own competence. Asked "are you stuck?", they say no until far
|
||||
@@ -194,13 +199,13 @@ How many `ask_expert` calls one milestone may make. `ask_expert` is a single
|
||||
stateless question to a stronger model, with no access to the repository
|
||||
(architecture L9), for things like "is this API deprecated?".
|
||||
|
||||
- **Too low:** the executor can't get unstuck on a knowledge question it
|
||||
- **Too low:** the driver can't get unstuck on a knowledge question it
|
||||
can't answer from the code.
|
||||
- **Too high:** a confused executor asks variations of the same question
|
||||
- **Too high:** a confused driver asks variations of the same question
|
||||
instead of doing the work or failing. That's the circling L9 warns about.
|
||||
- **Deciding:** count consults per milestone, and check whether the answers
|
||||
changed what the executor did next. If it regularly exhausts the budget
|
||||
without progress, the budget is fine and the model or the plan is the
|
||||
changed what the driver did next. If it regularly exhausts the budget
|
||||
without progress, the budget is fine and the model or the brief is the
|
||||
problem.
|
||||
|
||||
### `tuning.delegation` — hard caps on one delegation
|
||||
@@ -282,10 +287,9 @@ Tokens are estimated from characters (pessimistically, at 3 per token) and
|
||||
the estimate is corrected from the prompt sizes the server reports. The
|
||||
correction only ever makes the estimate larger.
|
||||
|
||||
The executor also sends less in the first place: the approved milestone plan
|
||||
goes in its system prompt, which is never cut, and it sees only what happened
|
||||
since that plan. The planning conversation and earlier milestones stay in the
|
||||
log.
|
||||
Each role also sends less in the first place: it sees its own work item's
|
||||
transcript (L6), with other roles' work reduced to what it needs (a helper's
|
||||
handback is one report to the driver).
|
||||
|
||||
| Knob | Default | |
|
||||
|---|---|---|
|
||||
@@ -331,14 +335,15 @@ Asking a model that can't honour it can be worse than not asking: the
|
||||
Qwen3-14B turn above costs a full output ceiling of generation, where a plain
|
||||
prose reply would have cost a few hundred tokens.
|
||||
|
||||
Where it isn't enforced, the executor holds the contract itself:
|
||||
Where it isn't enforced, the driver's run holds the contract itself:
|
||||
|
||||
- **`finish` is the only way to end a milestone.** It is refused once if
|
||||
nothing has been changed yet.
|
||||
- **A prose reply is a violation.** The model is told once; a second escalates.
|
||||
- **A call to a tool that wasn't offered counts as a failure** toward
|
||||
`failure_limit`, like a failed command, so an inventing model escalates in a
|
||||
couple of turns instead of using up `max_turns`.
|
||||
- **A waiting tool is the only way to end a run** (`finish_milestone`,
|
||||
`ask_human`, and the like); the harness, not the model, then publishes or
|
||||
asks.
|
||||
- **A prose reply is a violation.** The model is told once; a second ends
|
||||
the run, and the item waits on you.
|
||||
- **A call to a tool that wasn't offered is answered, not run:** the result
|
||||
names the tools there are, so the model can put it right on its next turn.
|
||||
|
||||
The capability matrix checks the declaration: a model declared `true` that
|
||||
answers a required call in prose fails
|
||||
@@ -412,14 +417,17 @@ done.
|
||||
The first four were the original set. By September 2026 all seven newer local
|
||||
models passed all of them, so the last four were added to separate models that
|
||||
can hold the tool contract from models that can do a milestone's real work.
|
||||
Their plans stay explicit, as the planner is told to write them; what makes
|
||||
Their plans stay explicit, as an agreed restatement is; what makes
|
||||
them harder is the work itself: tracing a cause, searching a codebase,
|
||||
reading a spec closely, and holding a plan over many turns.
|
||||
|
||||
It runs the executor exactly as the agent does (the same client wiring,
|
||||
context budget and `tool_choice` handling, and your config's tuning knobs),
|
||||
with **delegation and `ask_expert` turned off**. An escalation is a failure of
|
||||
the model under test, not a frontier model's success.
|
||||
It runs the loop's own driver exactly as the agent does: the same system
|
||||
prompt, tools, client wiring, context budget, `tool_choice` handling and your
|
||||
config's tuning knobs. The task's plan is given as the agreed restatement, and
|
||||
finishing the milestone ends the run. **Delegation and `ask_expert` are
|
||||
turned off**: an escalation is a failure of the model under test, not a
|
||||
frontier model's success. Until M2.8 it ran the old loop's executor, so runs
|
||||
from before then are against older task versions and show as `(old)`.
|
||||
|
||||
**Evaluating a new model** is one command, once its LiteLLM entry declares its
|
||||
limits:
|
||||
@@ -456,7 +464,7 @@ served differently (different note) are kept apart.
|
||||
`failure_limit: 2` mid-recovery. That says as much about the knob as about
|
||||
the model. Tuning a knob and re-running is a fair comparison, because the
|
||||
tuning is recorded with every run.
|
||||
- **Cost** is per run, straight from LiteLLM, so a frontier model as executor
|
||||
- **Cost** is per run, straight from LiteLLM, so a frontier model as driver
|
||||
can be compared on success rate *and* dollars.
|
||||
|
||||
---
|
||||
@@ -511,8 +519,8 @@ Later work that will add knobs, each with its own section here:
|
||||
- **Compaction.** A model-written summary replacing old turns, instead of
|
||||
dropping them. The log already has an event for it; it needs a summarizing
|
||||
model and a policy for what is safe to summarize.
|
||||
- **Benchmarking the other roles:** planner and delegate, with delegation
|
||||
turned back on for comparisons across frontier tiers.
|
||||
- **Benchmarking the other roles:** the helper, with delegation turned back
|
||||
on for comparisons across frontier tiers.
|
||||
|
||||
### `tuning.loop.bug_grace_s` — default 600
|
||||
|
||||
|
||||
Reference in new issue
Block a user