- L1: `general`, the scope of conversation not (yet) about a project: an
item per thread, no repository, build VM or execution; closes after a day
idle, a later reply starting a linked item.
- L5: every conversation is an item from its first message; chit-chat is
answered, only work gets a Gitea issue.
- D1: a conversation's record issue is opened once it's work on a project.
The new-project sequence: proposal, a provisioning checklist in Matrix,
verified when the operator says done, a tracking issue carrying the
distilled decisions (not the raw chat), then the architecture PRs.
- Q: a scratch build VM for general replaces the answered question about
records for pure conversations.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- Every milestone branch is cut from the latest default branch: the next
milestone builds on the merged previous one, and a redo after a closed PR
drops the rejected work. Uncommitted work comes along.
- While the item's PR is open, start_milestone is refused, with the PR's
state read from Gitea at that moment, not the last watch round.
- finish_milestone(final): earlier PRs say "Part of #N", the last "Closes
#N"; unsaid, it's final, as before.
- close_item with the item's PR still open waits for the merge (L2: a
closed item leaves nothing open).
- The driver's instructions cover ARCHITECTURE.md and build plans under
docs/plans/ (D2); the harness never parses a plan.
- Parking stashes uncommitted work and pushes it to <item>-parked, put back
on resume, instead of committing it onto the item's branch. The walk
found a half-done review fix parked onto a PR that had merged meanwhile,
where it was lost. Architecture D4's parking line says so.
Tests: unit tests for fresh milestone branches, the merge wait (including
a merge not yet watched), redo after a closed PR, Part of/Closes, close
waiting on an open PR, and the parked-fix regression; the walk's new
milestones_start_fresh property (fork-from-current fails 9 seeds); an e2e
two-milestone feature.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- Commands that name their item ("go #14", "abandon #14", "stop #14")
apply to it wherever they're said; go, approve and abandon alone at top
level are asked about (L5). stop is never taken as an answer.
- A plain-language reply to a restatement is read by GoAheadReader, one
model call that agrees only on a clear yes; questions never reach it. The
agent says "taking that as a go-ahead on rev-N" first, bound to the
revision on screen at arrival (L3, L5, D1).
- Labels steer (D1): `bug` goes ahead after tuning.loop.bug_grace_s
(default 600 s) unless the operator replies; `discuss` isn't offered
restate until the operator has spoken; `bug`, `discuss` and `design` add
to the driver's instructions. A restatement asks no questions.
Tests: label and command unit tests, the property walk with labelled
issues, plain yeses, named gos and time passing (grace_only_unanswered,
discuss_talks_first), and e2e tests for a bug proceeding after its grace
period and a plain-language go-ahead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Several work items on the new loop (L1, D4):
- Switching parks the item on the workspace: its work is committed and its
branch pushed; resuming checks the item's branch back out. Branches of
open items are never set aside by a fresh workspace.
- Next item: the operator's "do #N next" first, otherwise the oldest.
- Issues with open Gitea dependencies are blocked until they close, checked
at intake and on every round; status shows blocked items and the line.
Tests: queue unit tests, the property walk over several issues with
dependencies and a new parked_work_safe property (it catches parking
disabled), and an end-to-end test with two issues where B depends on A.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
From the new loop's first live run (sandbox #255, PR #256):
- A question asked on the PR is answered on the PR (an inline one quoted
with its file and line, since the API can't reply in its thread), with
a pointer in the item's Matrix thread; one asked in Matrix stays there.
- wait_for_review: when a review needs no change, or only the operator's
merge is left, the item waits on the merge instead of on an answer.
- finish_milestone takes the PR's title from the driver.
The e2e test's merge waits out Gitea's brief 'try again later' after a
push.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The first live run of the new loop's e2e test found that a fact recorded
while start_milestone was in flight landed between the tool call and its
result; both API formats reject that, so every later request in the run
failed. Views now keep each result right after its call. Scripted test
models check the APIs' ordering rules, so this can't pass unit tests
again.
The e2e tests no longer depend on the sandbox's history: each run asks for
a uniquely named function and works against a base branch of its own,
deleted afterwards. The review step asks for a change that can't already
be there. The scripted operator gives up after five fruitless answers, and
each run keeps its state store for inspection. The driver's prompt says
review fixes stay in the current milestone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
From an issue the scripted operator files and assigns in Gitea, through
the restatement, go-ahead, milestone and PR, a review comment answered
with a pushed fix, to an approved and merged PR and a closed item and
issue. The operator acts in Gitea through its own test account
(NISSE_OPERATOR_TOKEN) and in the item's thread through the in-memory
transport. Runs from scripts/e2e.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
test_loop_properties.py walks the new loop with a seeded mix of operator
and Gitea actions (go-aheads, stale go-aheads, answers, comments, merges,
the comment-then-merge race, closes, abandon, a crash and restart) and an
adaptive scripted model, checking every loop.qnt property after every
step. It found two bugs, both fixed with regression tests: abandoning an
item left its open PR open (the model's fix 1, now in code), and after a
review fix became a follow-up PR, the next fix went to the merged branch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A work item starts from a fresh workspace: fetch, set aside (never
discard) leftover changes, branches and unpushed commits, a clean default
branch, and a branch of its own (D3, #62). start_milestone cuts the
milestone's branch; finish_milestone commits only the work, pushes and
opens a PR that closes the issue, under an intent, adopting an open PR and
treating a closed one as the operator's answer (D4). Between runs the
loop watches each item's PR: operator comments and inline review notes
(#69) reach the item, a merge or close hands back to the driver. A review
fix is checked after it's pushed and goes out as a new PR if the PR merged
first (D5). At boot an interrupted publish is settled against Gitea (L8).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
nisse_dev.loop runs alongside control.py, chosen with `loop: new`.
Issues assigned to the agent (by an operator, or filed by one) become work
items, each with a Matrix thread. A driver run is one model with tools
until it calls a waiting tool: restate puts rev-N up for a go-ahead, bound
to the revision on screen when the reply arrived; ask_human waits for an
answer; close_item closes the item and its issue. Before the go-ahead the
driver can look but not change or close anything. One item is active at a
time; replies in an item's thread go to it. On boot, an item cut off
mid-run waits on the operator, and waiting points are put up again.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Transport.post(text, thread=...) returns the event ID, which can root a
thread; a threaded post carries m.thread with the reply fallback for
clients without threads. Inbound messages carry their event ID and thread
root: a message in a thread gives its root, a plain reply joins the
thread of the message it replies to when known, and the quoted lines of a
rich reply are no longer part of the text. The in-memory transport plays
threads for tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
tests/integration/test_end_to_end.py drives the whole assembled agent
(real models, the Gitea sandbox, a real build host) through a milestone
with a scripted operator, and checks what bring-up run 3 checked by hand:
a PR holding only the work, pushed with the push credential; no
credential in the clone; the planner's view carrying facts, not the
executor's turns. It cleans up after itself on the sandbox and the build
host.
scripts/e2e.sh runs it on the bring-up build VM (or --local). It has its
own marker, e2e, kept out of check.sh and CI since it spends tokens.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The priority is the smallest new loop running live, so M1.4 and M1.5 move
after M2.1. Each milestone is now checked by an automated end-to-end test
(new M1.6) against real resources, not a manual live run; manual runs are
for the new loop's first runs and phase ends.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A failing CI test printed a test key through SSHKey's default repr. Push
credentials, AgentConfig's tokens and MatrixConfig's token are now
excluded from their reprs, so no failure or log line can print them. In
CI's container killed ssh-agents linger as zombies, so the tests count
only live ones.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The clone gets the standard build artifacts in its own exclude file, so
they never show as changes, even to a model's `git add -A`. The
harness's commits leave the same patterns out even when the project
tracks them, as the sandbox's main did. What was left out is reported in
the publish note and the PR description. Found again in bring-up run 3
(sandbox PR #208).
Fixes#80.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Live run 3: the executor ran `git checkout -b <task id>` itself, and
publishing then failed, because git can't create '<task>/milestone-1'
while a branch named '<task>' exists. Milestone and delegation branches
are now '<task>-milestone-N' and 'delegation-milestone-N-F'. D4's
harness-owned branches (M2.1) remove the cause; this removes the
collision on the current loop.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The push credential is either a push-only account's token, handed to git
through a credential helper that names environment variables, never
values, or a deploy key in a short-lived ssh-agent. The remote URL
follows the credential. Configured as gitea.push_user and push_token, or
gitea.push_key.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
execution.SSHHost.execute is now the only way the agent runs anything
(L13). Commands travel as a script on SSH's stdin, so the secrets handed
to a call are never on a command line, and every value handed out is
redacted from the output. Timeouts kill the remote process group, and
an unreachable host is reported as such, not as a failed command.
Workspace and GitRepo run on the build VM. The clone uses the
repository's SSH URL, so no credential is stored in it (#89); fetches
and pushes get the push key over stdin, in a short-lived ssh-agent,
and PRs are still opened from the agent's host with its API token.
The bench runs on the build host too.
Tests reach "the build VM" through a throwaway local sshd (nisse_dev
.testing); CI installs openssh-server. The live push test now pushes
with the sandbox's deploy key (NISSE_PUSH_KEY).
Fixes#89.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
SSH execution and the agent as a container image come first (Phase 1),
then the new loop as a thin end-to-end slice, widened milestone by
milestone with a live run after each. Outline, Vikunja and per-agent VMs
are gone; the vault, the infrastructure repository and the agent host
replace them. The survey and issue map are refreshed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Name the action classes harmless, contained and dangerous instead of
numbering them; say what escalation doesn't depend on; say when a work
item's Matrix thread starts and how threads route commands; add open
questions for the state store's engine, ops issuing credentials, and
releases.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- abandon closes the item's open PRs
- a PR the operator closed is an answer, never reopened by a retry
- a review fix for a PR that merged meanwhile goes out as a new PR
- redoing a milestone after a closed PR gets a new branch and PR
- a record issue carries its work item's ID from creation
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A Quint model of the design in docs/Architecture.md (PR #90): intake,
restatements and go-aheads, park-and-switch, blocked dependencies,
harness-owned milestones, waiting for merges, review follow-ups, record
issues, crashes and build VM wipes. It found five gaps; the model carries
the proposed fixes, and loop_scenarios.qnt keeps each counterexample.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Ops work lives in one infrastructure repository for both sites, with
site labels; intake, labels and dependencies work as for nisse-dev, and
agent-to-agent requests are cross-repo issues.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Each rule is stated once in its home section and referred to by ID
elsewhere. No decision added or removed; implementation detail moved to
Development-Plan.md section 6, keyed by architecture section. P8 folded
into P1, P5 and X1; the Appendix about the old spec dropped. Section
IDs are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Issues assigned to dev-agent are the intake; the agent restates its
understanding and either proceeds (bug label) or waits for a go-ahead.
Labels steer and are never required. Build plans and ARCHITECTURE.md
are working documents reviewed in PRs, not gates; scope is the issue
and its conversation. Work items can be blocked on Gitea dependencies,
each gets a Matrix thread, milestones are started and finished by
control tools, and the next milestone waits for the previous merge.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Agents run as containers on one stable agent host that keeps their
state and secrets and never runs a command a model chose. Model-chosen
commands run on disposable tool VMs over SSH: a build VM per project
for nisse-dev, its own tool VM for nisse-ops. Secrets are a set per
agent, handed out per operation by the harness and redacted from tool
output. MCP servers stay on the agent host as the one exception.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A model no longer reads another role's turns as its own (#78). The log
stays one record per work item; what a role is shown is projected from
it by Transcript.view() (views.py), which the planner, executor and
helper now build their context from:
- own events unchanged;
- operator messages labelled with their channel, e.g.
[operator, Matrix] or [operator, comment on PR #165];
- orchestrator facts (a new `fact` event) as [orchestrator] statements;
- the executor sees a helper's handback as [helper report], not its turns;
- a helper sees the approved plan and the executor's milestone work as
one-line narration;
- the planner sees no other role's work at all.
The orchestrator now writes facts at the points the models must know
about: work item opened, plan approved or rejected, milestone started,
finished or stopped, published (with PR number and URL) or not, review
comment answered, next milestone, task completed or abandoned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M1.1, as discussed.
- Schema v4: a work_items table (id, scope, title, status, outcome,
position, data) and an agent row whose active_item points at one
(NULL: idle). Canonical events and intents carry their work item, and
events their actor. Older stores are refused with a clear message:
there was no production data, so no migration (the mechanism stays).
- The log is reached through a Transcript bound to one work item and
one actor: store.transcript(item, actor) reads only that item and
writes with both. The planner, executor, delegation and expert desk
get one per call from the orchestrator instead of holding the store,
so a component can neither read another task's events (the planner
read every earlier task's, #78) nor write without attribution.
store.all_events() is kept for readers that span items (evaluation
capture, dumps). Intents are begun through the transcript or with
work_item=, so each belongs to its item.
- The orchestrator opens a work item on 'use', closes it with its
outcome (completed or abandoned), and carries what the operator said
while idle into the new item as copies with their origin, so a task
sent before 'use' is still planned from (D1). control.py keeps its
state machine through get_state()/transition(), now over the active
work item, until M2.6 retires it.
- nisse_dev.dump shows work items and each event's item and actor.
Tests: test_work_items.py (scoping, attribution, intent ownership,
restart, outcomes, carried messages, a second task never seeing the
first); the suites updated for the per-call transcript.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Architecture L9's role names, for the two roles with a direct
counterpart: the delegation role is now 'helper' and the ask_expert role
'expert', in RoleMap, configuration (roles.helper, roles.expert),
recorded events and tests. 'planner' and 'executor' stay distinct for
now; they become 'driver' when the new loop merges them (Development
Plan M2.3).
Configurations must rename their keys: nisse-dev.example.yaml and
tests/integration/nisse-test.yaml are updated here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M0.2.
- scripts/bringup.sh: start [REF] sets the previous run aside under
/root/nisse-e2e/runs/, exports the ref under test and starts the agent
as the transient systemd unit nisse-bringup, so it outlives the shell
that started it; stop (SIGTERM, clean shutdown), status, logs, and
collect (the journal log and a readable transcript).
- nisse_dev.dump (python -m nisse_dev.dump STATE_DB [--full]): a readable
dump of a state store's state, pending intents and canonical log. Opens
the store read-only, so it is safe against a live agent.
- The agent runs as root on this development VM, as it will on its own
VM. Architecture D3/D6/D7 corrected to match the operator's intent:
installing packages and toolchains on its own VM is class 2, recorded
so the environment can be rebuilt (it was class 3 with provisioning an
open question); egress names OS package mirrors. The plan's M0.2, M4.1,
M4.2 and bring-up guards follow.
Acceptance: a smoke run started, answered status in the test room,
stopped cleanly, and collect produced the log and transcript.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M0.1. Every section reference in code, tests, specs/,
the example config, the tuning guide, and the CLAUDE.md and README files
now cites docs/Architecture.md (P, L, D, O, X, Q) instead of the old
spec's A-H, and docs/Architecture-Build-Spec.md is deleted (git keeps
it). The plan's old-to-new table goes with it.
Mostly mechanical, with a per-site choice where one old section split:
A4 became L2 where it meant stop/abandon/the loop and L3 where it meant
approvals; A10 became L6 in the context-budget code and L7 elsewhere; B2
became D4 for outbound Gitea and publishing and D5 for review intake;
'A3 step N' became the matching step of L8's boot sequence. Comments
describing behaviour the architecture drops (planning always escalating,
thresholds tuned at an old build step) now describe the current code and
point at the Development Plan.
docs/Architecture.md gains one line the rewrite missed: intake outages
delay but never fail work (old B2), now in L5, where the poller and
Matrix code cite it.
Comments and docs only; no behaviour change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
docs/Development-Plan.md: where the code stands against each section of
docs/Architecture.md, the environment today, how we work (plan-first per
milestone, tests, the Quint model, a bring-up run on this box ending
each phase), and seven phases of milestones from housekeeping to
nisse-ops, each marked pre-approved or needs discussion. Every open
issue is mapped to a milestone or marked superseded; an old-to-new
section table serves the reference update in M0.1.
docs/Architecture.md: status now agreed; D7 regains the browser tool
for the internal test network, left out of the rewrite by mistake.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A full rewrite of the architecture, following the outline agreed with the
operator: a threat model of mistakes, injection and crashes rather than a
malicious model; actions classed by their effects; hard walls, a
deterministic fast path and an action classifier; a conversational loop
driven by a local model with frontier helpers; approvals as scoped
permissions with pre-approved milestones and a trust dial; context and
memory, including agent memory; documents as nisse-dev's workflow.
Added beside docs/Architecture-Build-Spec.md, which stays until this is
agreed; the Development Plan and reference updates come after. New
section letters (P, L, D, O, X, Q) avoid clashing with the old A-H
references in the code meanwhile.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Fixes counterexamples 6 and 7 from specs/orchestrator.qnt, the last two.
6. A crash while the executor answered a review comment lost it: its
cursor is saved before the work (at most once), and a restart in
PR_OPEN resumed quietly. Now pr.reviewing marks the comment until
the executor returns (every way it returns, but not a crash), and
boot resumes the watch just before it, with a notice. The mark stays
until it is answered, so a second crash cannot lose it either.
7. 'project complete' left the plan issue open. It now closes it with a
note, after IDLE is saved, as abandon does (shared helper).
The model marks and redelivers, and closes the issue; scenarios 6 and 7
now expect their properties to hold. A mutant without redelivery fails
scenario 6. All eight properties held together over 300,000 traces.
No xfail reproductions remain.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Fixes counterexample 5 from specs/orchestrator.qnt. A crash after the
PR was opened but before PR_OPEN was recorded left the agent in
EXECUTING; the restart question's 'continue' and 'start' then published
again, Gitea refused a second PR, and there was no way back to PR_OPEN.
On the live run the executor model happened to notice the existing
work; recovery must not depend on that.
- Boot (A3 step 4): in EXECUTING, if Gitea shows the milestone's PR
open, adopt it, watch it from the start, and resume in PR_OPEN with a
notice. Gitea unreachable or the PR closed: the restart question, as
before.
- MilestonePublisher.publish is idempotent: after pushing, an open PR
for the branch is adopted (and so updated) rather than a second one
opened. This also covers a PR opened whose response was lost.
- The model adopts too, gains the lost-response outcome, and now counts
PRs opened per milestone (one_pr_per_milestone was a flag the fixed
model could never set). A mutant without adoption fails the new
scenario; the properties held over 200,000 traces each.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Fixes counterexamples 1-4 from specs/orchestrator.qnt: a repeated
'approve', 'approve rev-1' meant for the architecture, approving on both
channels, and 'approve' plus an afterthought all landed on the next
plan, one the operator had not seen.
- Revisions are numbered across the work item (A4 already required
monotonic tags; the code numbered per gate), so rev-N names one plan.
- Each reply is stamped on arrival with the revision that was up: in
Matrix the one whose approval request had been sent (a plain value
the transport thread reads, restored on boot), in Gitea the latest
plan comment older than the reply (board.posts).
- Gate.evaluate applies a reply that names no revision only if its stamp
is the current revision. Otherwise an approval is not applied and
feedback rejects nothing (it is still in the transcript), and the
agent says which revision is up. 'approve rev-N' binds by its number.
- Spec A4 states both rules. The model applies them; scenarios 1-4 now
expect the properties to hold, and both held over 200,000 random
traces and Apalache's exhaustive check to depth 8.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
specs/orchestrator.qnt models control.py: states, gate, pending
questions, comment cursors, publish intents and the worker queue, with
operator messages, Gitea comments, work outcomes (exceptions included)
and a crash at any point interleaving freely. Eight properties.
Seven counterexamples, each written out as a trace in
specs/orchestrator_scenarios.qnt and reproduced against the real code in
nisse-dev/tests/test_quint_counterexamples.py (xfail, strict, until
fixed):
1-4 an approval or feedback lands on a plan the operator had not seen:
a double 'approve', 'approve rev-1' meant for the architecture,
approving on both channels, and 'approve' plus an afterthought;
5 a crash after the PR opens but before PR_OPEN publishes again and
cannot get back to PR_OPEN;
6 a crash while answering a review comment loses it silently;
7 'project complete' leaves the plan issue open.
no_stranding, await_input_has_a_question and execution_needs_approval
held over 200,000 random traces of 40 steps, and under Apalache's
exhaustive check to depth 8, which does find counterexample 1 at that
depth. No behaviour changes here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A phone keyboard's "Approve" did not match the approval pattern, so it
counted as rejection feedback and threw the plan away; "Stop" did not
reach the cancel flag; other capitalised commands got a hint instead of
acting.
gates.command_form() is how a message reads as a command: case,
surrounding whitespace, inner runs of whitespace and a trailing '.' or
'!' do not matter. The approval gate and every command match in the
orchestrator (stop, status, abandon, use, start milestone, next
milestone, project complete, the restart choices) use it. Only matching
changes: the project name in 'use <name>' keeps its case, and feedback
and answers are recorded and passed on as written. A stale revision is
still refused, and approval still needs the whole message to be the
command.
Closes#64.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A4 already names abandon as one of the ways out of a halt (continue,
re-plan, or abandon) and 0.2 closes a work item as completed or
abandoned, but no command did it: from a gate or a question the only way
to drop a task was to edit the state database.
- 'abandon' ends the task from any waiting state and returns to IDLE.
The move is made durable first; then the plan issue and any open
milestone PR are closed with a note (branches kept), and any
unresolved intent is aborted so boot reconciliation does not gate on
work dropped on purpose. A Gitea failure is reported, not fatal.
- Like 'status' it is a command in every state, never read as an answer
or feedback. While work runs, 'stop' comes first, as before.
- The restart / stranded question and the PR_OPEN hints mention it.
- Spec: B1 gains the abandon edge and B2 the command.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- Record a comment cursor per thread (board.last_comment, the field that
already existed unused; pr.last_comment), advanced durably when a
comment is taken up. CommentPoller.resume_after() skips comments at or
below it, so re-attaching needs no request and cannot replay.
- Boot re-attaches the plan watch at an open gate and the review watch in
PR_OPEN from their cursors, catching up on comments posted while the
agent was down (#59). Data saved without a cursor skips what is there,
so an old approval cannot approve the current revision.
- A restart in PR_OPEN resumes with a notice instead of the restart
question (#58). A stored restart question for PR_OPEN returns to it on
either reply instead of re-planning the milestone.
- A stopped or failed review response gets its own pending kind; the
reply returns to PR_OPEN and resumes the watch from the cursor, so
comments that arrived meanwhile are delivered. It used to return to
milestone sign-off, where starting the milestone would open a second PR.
- Comments ignored because of the state leave the cursor alone, so they
are delivered once the agent is back where they apply.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The test checks that a request reaches the executor model through
LiteLLM and text comes back. It also asserted the model obeyed "Reply
with exactly one word: pong", which Haiku once answered with "ping" and
failed CI on #66. Assert non-empty text instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
It drove the orchestrator until EXECUTING and accepted EXECUTING as the
end state, which with no publisher configured was the stranding this
branch removes: the milestone now finishes in PR_OPEN. Return once
'start milestone 1' is sent, since the milestone runs inside it, and
expect PR_OPEN or AWAIT_INPUT.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The first live run found four dead ends one at a time: states where no
operator message could move the agent. Most had one shape: work ended,
by an exception or by a path that never transitioned, and left the agent
in PLANNING_* or EXECUTING, where every message is answered "busy".
The worker only prints an exception, so that would keep happening.
- Every job the orchestrator runs for an operator message or a Gitea
comment now goes through _settled(). If the job leaves a working state
behind, whatever the reason, the agent moves to AWAIT_INPUT with the
continue / re-plan choice a restart offers, and says why. An exception
in a waiting state is reported and the state kept, since its commands
still lead on.
- A sign-off state whose gate record is missing asks to plan again
instead of crashing on every reply.
- The stub configurations (no executor, no publisher) complete the
milestone and wait in PR_OPEN instead of resting in EXECUTING.
test_no_stranding.py drives a whole two-milestone project with a fault
injected at each planner, executor and publisher call, and asserts after
every message that the agent is not in a working state and that the
project still reaches 'project complete'.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
On the first live run the executor committed its milestone itself, as
its plan asked. The publisher only published a commit it made, found a
clean tree, reported "no changes to publish", and left the agent in
EXECUTING, where every message is answered "busy".
- publish(): with nothing left to commit, publish anyway when the branch
has commits origin/<base> lacks; report no changes only when it has
none.
- push_followup(): same for review fixes, comparing HEAD with what the
remote branch holds (ls-remote; pushes go by URL, so there is no
tracking ref).
- Nothing to publish now gates to AWAIT_INPUT, and the reply returns to
the still-approved milestone plan.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Clients put the addressee's name in front of a message that mentions
it ("Nisse Dev Testing: re-plan"), and the transport passed that text
on whole, so no command matched: the live run's re-plan was asked again,
and an addressed 'approve' would have counted as rejection feedback.
- A leading mention of the agent (an HTML pill, its user ID, or its
display name followed by ':') is removed before commands are matched.
The display name is learnt from the agent's own member event.
- A message whose leading mention is someone else is skipped: in a
shared room it is addressed to another agent or person.
- Only the start counts; a mention later in the text is left alone.
- The transcript records the message as sent (InboundMessage.original).
The sender is unaffected: it still comes from the event's homeserver-set
sender field, never from the text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A restart in AWAIT_INPUT replaced the saved pending item with a generic
"continue or re-plan?" whose answer nothing handled: the reply cleared
it and left the agent in AWAIT_INPUT with nothing to resume, where 'use'
was refused too. The first live run hit this after a planning failure
and could not be moved by any message.
- Waiting on the operator is a committed state (A3 step 8): on boot, ask
the saved question again instead of replacing it. _await_input keeps
the prompt with the pending item for this.
- The restart question (A3 step 7) records the state it interrupted, and
'continue' / 're-plan' now do what they offer: re-run the interrupted
planning turn, or return an interrupted milestone to its approved plan.
Any other reply asks again rather than being swallowed.
- With nothing pending, 'use <project>' starts over, as the reply
already advised.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It led the executor benchmark (PR #55): 37/40 over eight tasks and 17/20
on the harder four, ahead of claude-haiku-4-5's 15/20. local-fast, the
previous example, scored 0/8 on the harder four. Model selection is on
the dev executor alone for now; the shared-host question (spec Part H)
stays open.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Five runs per task at 320 W with max_turns 40, after the fixes in
#50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b,
devstral-small-2-24b and mimo-v2.6-9b; the harder four also for
claude-haiku-4-5 as the ceiling, and two calibration runs each for
local-fast as the floor. Runs failed by the proxy's rate limiter were
set aside, not stored.
Earlier results move to bench/results/archive/, which the report does
not read: they were made at other power caps, at max_turns 24, or on
the RTX 4070, and are kept as the record.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The host-sharing item now records the operator's preference: one local
model for every local role, with agents tagging in and out of the host
as the fallback if no single model fits. A new item records vision as a
requirement (OpenSCAD renders, later video files) and what the text-only
agent needs before any vision work.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
nisse-dev and nisse-ops both send local roles to the one 24GB host,
where llama-swap keeps one model resident. Different models per agent
would reload weights on every switch. Per-key rate limits don't help.
Lists the options, simplest first, to decide after the executor
benchmark and the ops evaluation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Calibration showed Haiku fixing the right constant but keeping only four
decimal places, which passes the visible test and still loses the JPY
rate's precision. An explicit plan should say so.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
All seven newer local models passed the original four tasks 20 of 20,
so the benchmark no longer told them apart. The new tasks keep plans
explicit, as the planner is told to write them, and make the work
itself harder:
- trace-bug: a wrong multi-currency total whose cause is a shared
rate helper; discounts and tax go through it too, so a fix at the
call site fails the check.
- refactor-rename: rename load_config across a ten-module package,
leaving load_config_defaults and reload_config alone.
- spec-parser: implement parse_duration from an eight-rule SPEC.md
given three examples, and write tests for every rule.
- long-milestone: eight dependent steps adding due dates to a task
tracker, from the model through storage versions to the CLI.
Self-tests prove that a call-site fix, a blind text replace and a
partial rename all fail their checks.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The report grouped runs by model, serving note and task version, so
runs made at different tuning were silently averaged together. Tuning
is now part of the key, and a tuning column shows how a row's settings
differ from today's defaults; it appears only when some row needs it.
tuning.executor.max_turns goes from 24 to 40. On the harder benchmark
tasks Haiku, the calibration ceiling, hit 24 on both refactor-rename
runs and both long-milestone runs, one of which had finished the work.
The cap guards against a model trying forever; it isn't a budget.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two setup failures found while calibrating the new benchmark tasks,
both of which would also fail real milestones.
Devstral's chat template (Mistral) rejected any turn in which the model
wrote some text before a tool call: the OpenAI renderer made one message
per event, so the turn became a text message and a tool-call message in
a row, and llama.cpp answered 500. The renderer now folds consecutive
user or assistant messages into one, which is the OpenAI format for a
turn anyway. Checked against the live server: the captured failing
request no longer occurs, over six real runs.
LiteLLMClient gave up on any request after a fixed 120 s. A dense 27B
model writing a whole module in one turn takes longer, so the executor
escalated work that was going well. llm.request_timeout_s now sets it
for both clients, default 600.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently.
A benchmark run that waited looked slower than the model is: on
2026-09-27, fast models under the unit-test key's 200k tokens/minute
limit had 20-45 s runs that were probably waits.
Both clients now report last_rate_limit_wait_s after every call, even
one that then fails. The benchmark sums it per run into
rate_limit_wait_s, the report's time column leaves it out, and a new
429 column counts the runs that waited. Results stored before this
load unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dongarandClaude Opus 5.5