tests/integration/test_end_to_end.py drives the whole assembled agent
(real models, the Gitea sandbox, a real build host) through a milestone
with a scripted operator, and checks what bring-up run 3 checked by hand:
a PR holding only the work, pushed with the push credential; no
credential in the clone; the planner's view carrying facts, not the
executor's turns. It cleans up after itself on the sandbox and the build
host.
scripts/e2e.sh runs it on the bring-up build VM (or --local). It has its
own marker, e2e, kept out of check.sh and CI since it spends tokens.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The priority is the smallest new loop running live, so M1.4 and M1.5 move
after M2.1. Each milestone is now checked by an automated end-to-end test
(new M1.6) against real resources, not a manual live run; manual runs are
for the new loop's first runs and phase ends.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A failing CI test printed a test key through SSHKey's default repr. Push
credentials, AgentConfig's tokens and MatrixConfig's token are now
excluded from their reprs, so no failure or log line can print them. In
CI's container killed ssh-agents linger as zombies, so the tests count
only live ones.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The clone gets the standard build artifacts in its own exclude file, so
they never show as changes, even to a model's `git add -A`. The
harness's commits leave the same patterns out even when the project
tracks them, as the sandbox's main did. What was left out is reported in
the publish note and the PR description. Found again in bring-up run 3
(sandbox PR #208).
Fixes#80.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Live run 3: the executor ran `git checkout -b <task id>` itself, and
publishing then failed, because git can't create '<task>/milestone-1'
while a branch named '<task>' exists. Milestone and delegation branches
are now '<task>-milestone-N' and 'delegation-milestone-N-F'. D4's
harness-owned branches (M2.1) remove the cause; this removes the
collision on the current loop.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The push credential is either a push-only account's token, handed to git
through a credential helper that names environment variables, never
values, or a deploy key in a short-lived ssh-agent. The remote URL
follows the credential. Configured as gitea.push_user and push_token, or
gitea.push_key.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
execution.SSHHost.execute is now the only way the agent runs anything
(L13). Commands travel as a script on SSH's stdin, so the secrets handed
to a call are never on a command line, and every value handed out is
redacted from the output. Timeouts kill the remote process group, and
an unreachable host is reported as such, not as a failed command.
Workspace and GitRepo run on the build VM. The clone uses the
repository's SSH URL, so no credential is stored in it (#89); fetches
and pushes get the push key over stdin, in a short-lived ssh-agent,
and PRs are still opened from the agent's host with its API token.
The bench runs on the build host too.
Tests reach "the build VM" through a throwaway local sshd (nisse_dev
.testing); CI installs openssh-server. The live push test now pushes
with the sandbox's deploy key (NISSE_PUSH_KEY).
Fixes#89.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
SSH execution and the agent as a container image come first (Phase 1),
then the new loop as a thin end-to-end slice, widened milestone by
milestone with a live run after each. Outline, Vikunja and per-agent VMs
are gone; the vault, the infrastructure repository and the agent host
replace them. The survey and issue map are refreshed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Name the action classes harmless, contained and dangerous instead of
numbering them; say what escalation doesn't depend on; say when a work
item's Matrix thread starts and how threads route commands; add open
questions for the state store's engine, ops issuing credentials, and
releases.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- abandon closes the item's open PRs
- a PR the operator closed is an answer, never reopened by a retry
- a review fix for a PR that merged meanwhile goes out as a new PR
- redoing a milestone after a closed PR gets a new branch and PR
- a record issue carries its work item's ID from creation
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A Quint model of the design in docs/Architecture.md (PR #90): intake,
restatements and go-aheads, park-and-switch, blocked dependencies,
harness-owned milestones, waiting for merges, review follow-ups, record
issues, crashes and build VM wipes. It found five gaps; the model carries
the proposed fixes, and loop_scenarios.qnt keeps each counterexample.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Ops work lives in one infrastructure repository for both sites, with
site labels; intake, labels and dependencies work as for nisse-dev, and
agent-to-agent requests are cross-repo issues.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Each rule is stated once in its home section and referred to by ID
elsewhere. No decision added or removed; implementation detail moved to
Development-Plan.md section 6, keyed by architecture section. P8 folded
into P1, P5 and X1; the Appendix about the old spec dropped. Section
IDs are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Issues assigned to dev-agent are the intake; the agent restates its
understanding and either proceeds (bug label) or waits for a go-ahead.
Labels steer and are never required. Build plans and ARCHITECTURE.md
are working documents reviewed in PRs, not gates; scope is the issue
and its conversation. Work items can be blocked on Gitea dependencies,
each gets a Matrix thread, milestones are started and finished by
control tools, and the next milestone waits for the previous merge.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Agents run as containers on one stable agent host that keeps their
state and secrets and never runs a command a model chose. Model-chosen
commands run on disposable tool VMs over SSH: a build VM per project
for nisse-dev, its own tool VM for nisse-ops. Secrets are a set per
agent, handed out per operation by the harness and redacted from tool
output. MCP servers stay on the agent host as the one exception.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A model no longer reads another role's turns as its own (#78). The log
stays one record per work item; what a role is shown is projected from
it by Transcript.view() (views.py), which the planner, executor and
helper now build their context from:
- own events unchanged;
- operator messages labelled with their channel, e.g.
[operator, Matrix] or [operator, comment on PR #165];
- orchestrator facts (a new `fact` event) as [orchestrator] statements;
- the executor sees a helper's handback as [helper report], not its turns;
- a helper sees the approved plan and the executor's milestone work as
one-line narration;
- the planner sees no other role's work at all.
The orchestrator now writes facts at the points the models must know
about: work item opened, plan approved or rejected, milestone started,
finished or stopped, published (with PR number and URL) or not, review
comment answered, next milestone, task completed or abandoned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M1.1, as discussed.
- Schema v4: a work_items table (id, scope, title, status, outcome,
position, data) and an agent row whose active_item points at one
(NULL: idle). Canonical events and intents carry their work item, and
events their actor. Older stores are refused with a clear message:
there was no production data, so no migration (the mechanism stays).
- The log is reached through a Transcript bound to one work item and
one actor: store.transcript(item, actor) reads only that item and
writes with both. The planner, executor, delegation and expert desk
get one per call from the orchestrator instead of holding the store,
so a component can neither read another task's events (the planner
read every earlier task's, #78) nor write without attribution.
store.all_events() is kept for readers that span items (evaluation
capture, dumps). Intents are begun through the transcript or with
work_item=, so each belongs to its item.
- The orchestrator opens a work item on 'use', closes it with its
outcome (completed or abandoned), and carries what the operator said
while idle into the new item as copies with their origin, so a task
sent before 'use' is still planned from (D1). control.py keeps its
state machine through get_state()/transition(), now over the active
work item, until M2.6 retires it.
- nisse_dev.dump shows work items and each event's item and actor.
Tests: test_work_items.py (scoping, attribution, intent ownership,
restart, outcomes, carried messages, a second task never seeing the
first); the suites updated for the per-call transcript.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Architecture L9's role names, for the two roles with a direct
counterpart: the delegation role is now 'helper' and the ask_expert role
'expert', in RoleMap, configuration (roles.helper, roles.expert),
recorded events and tests. 'planner' and 'executor' stay distinct for
now; they become 'driver' when the new loop merges them (Development
Plan M2.3).
Configurations must rename their keys: nisse-dev.example.yaml and
tests/integration/nisse-test.yaml are updated here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M0.2.
- scripts/bringup.sh: start [REF] sets the previous run aside under
/root/nisse-e2e/runs/, exports the ref under test and starts the agent
as the transient systemd unit nisse-bringup, so it outlives the shell
that started it; stop (SIGTERM, clean shutdown), status, logs, and
collect (the journal log and a readable transcript).
- nisse_dev.dump (python -m nisse_dev.dump STATE_DB [--full]): a readable
dump of a state store's state, pending intents and canonical log. Opens
the store read-only, so it is safe against a live agent.
- The agent runs as root on this development VM, as it will on its own
VM. Architecture D3/D6/D7 corrected to match the operator's intent:
installing packages and toolchains on its own VM is class 2, recorded
so the environment can be rebuilt (it was class 3 with provisioning an
open question); egress names OS package mirrors. The plan's M0.2, M4.1,
M4.2 and bring-up guards follow.
Acceptance: a smoke run started, answered status in the test room,
stopped cleanly, and collect produced the log and transcript.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M0.1. Every section reference in code, tests, specs/,
the example config, the tuning guide, and the CLAUDE.md and README files
now cites docs/Architecture.md (P, L, D, O, X, Q) instead of the old
spec's A-H, and docs/Architecture-Build-Spec.md is deleted (git keeps
it). The plan's old-to-new table goes with it.
Mostly mechanical, with a per-site choice where one old section split:
A4 became L2 where it meant stop/abandon/the loop and L3 where it meant
approvals; A10 became L6 in the context-budget code and L7 elsewhere; B2
became D4 for outbound Gitea and publishing and D5 for review intake;
'A3 step N' became the matching step of L8's boot sequence. Comments
describing behaviour the architecture drops (planning always escalating,
thresholds tuned at an old build step) now describe the current code and
point at the Development Plan.
docs/Architecture.md gains one line the rewrite missed: intake outages
delay but never fail work (old B2), now in L5, where the poller and
Matrix code cite it.
Comments and docs only; no behaviour change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
docs/Development-Plan.md: where the code stands against each section of
docs/Architecture.md, the environment today, how we work (plan-first per
milestone, tests, the Quint model, a bring-up run on this box ending
each phase), and seven phases of milestones from housekeeping to
nisse-ops, each marked pre-approved or needs discussion. Every open
issue is mapped to a milestone or marked superseded; an old-to-new
section table serves the reference update in M0.1.
docs/Architecture.md: status now agreed; D7 regains the browser tool
for the internal test network, left out of the rewrite by mistake.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A full rewrite of the architecture, following the outline agreed with the
operator: a threat model of mistakes, injection and crashes rather than a
malicious model; actions classed by their effects; hard walls, a
deterministic fast path and an action classifier; a conversational loop
driven by a local model with frontier helpers; approvals as scoped
permissions with pre-approved milestones and a trust dial; context and
memory, including agent memory; documents as nisse-dev's workflow.
Added beside docs/Architecture-Build-Spec.md, which stays until this is
agreed; the Development Plan and reference updates come after. New
section letters (P, L, D, O, X, Q) avoid clashing with the old A-H
references in the code meanwhile.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Fixes counterexamples 6 and 7 from specs/orchestrator.qnt, the last two.
6. A crash while the executor answered a review comment lost it: its
cursor is saved before the work (at most once), and a restart in
PR_OPEN resumed quietly. Now pr.reviewing marks the comment until
the executor returns (every way it returns, but not a crash), and
boot resumes the watch just before it, with a notice. The mark stays
until it is answered, so a second crash cannot lose it either.
7. 'project complete' left the plan issue open. It now closes it with a
note, after IDLE is saved, as abandon does (shared helper).
The model marks and redelivers, and closes the issue; scenarios 6 and 7
now expect their properties to hold. A mutant without redelivery fails
scenario 6. All eight properties held together over 300,000 traces.
No xfail reproductions remain.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Fixes counterexample 5 from specs/orchestrator.qnt. A crash after the
PR was opened but before PR_OPEN was recorded left the agent in
EXECUTING; the restart question's 'continue' and 'start' then published
again, Gitea refused a second PR, and there was no way back to PR_OPEN.
On the live run the executor model happened to notice the existing
work; recovery must not depend on that.
- Boot (A3 step 4): in EXECUTING, if Gitea shows the milestone's PR
open, adopt it, watch it from the start, and resume in PR_OPEN with a
notice. Gitea unreachable or the PR closed: the restart question, as
before.
- MilestonePublisher.publish is idempotent: after pushing, an open PR
for the branch is adopted (and so updated) rather than a second one
opened. This also covers a PR opened whose response was lost.
- The model adopts too, gains the lost-response outcome, and now counts
PRs opened per milestone (one_pr_per_milestone was a flag the fixed
model could never set). A mutant without adoption fails the new
scenario; the properties held over 200,000 traces each.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Fixes counterexamples 1-4 from specs/orchestrator.qnt: a repeated
'approve', 'approve rev-1' meant for the architecture, approving on both
channels, and 'approve' plus an afterthought all landed on the next
plan, one the operator had not seen.
- Revisions are numbered across the work item (A4 already required
monotonic tags; the code numbered per gate), so rev-N names one plan.
- Each reply is stamped on arrival with the revision that was up: in
Matrix the one whose approval request had been sent (a plain value
the transport thread reads, restored on boot), in Gitea the latest
plan comment older than the reply (board.posts).
- Gate.evaluate applies a reply that names no revision only if its stamp
is the current revision. Otherwise an approval is not applied and
feedback rejects nothing (it is still in the transcript), and the
agent says which revision is up. 'approve rev-N' binds by its number.
- Spec A4 states both rules. The model applies them; scenarios 1-4 now
expect the properties to hold, and both held over 200,000 random
traces and Apalache's exhaustive check to depth 8.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
specs/orchestrator.qnt models control.py: states, gate, pending
questions, comment cursors, publish intents and the worker queue, with
operator messages, Gitea comments, work outcomes (exceptions included)
and a crash at any point interleaving freely. Eight properties.
Seven counterexamples, each written out as a trace in
specs/orchestrator_scenarios.qnt and reproduced against the real code in
nisse-dev/tests/test_quint_counterexamples.py (xfail, strict, until
fixed):
1-4 an approval or feedback lands on a plan the operator had not seen:
a double 'approve', 'approve rev-1' meant for the architecture,
approving on both channels, and 'approve' plus an afterthought;
5 a crash after the PR opens but before PR_OPEN publishes again and
cannot get back to PR_OPEN;
6 a crash while answering a review comment loses it silently;
7 'project complete' leaves the plan issue open.
no_stranding, await_input_has_a_question and execution_needs_approval
held over 200,000 random traces of 40 steps, and under Apalache's
exhaustive check to depth 8, which does find counterexample 1 at that
depth. No behaviour changes here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
dongarandClaude Opus 5.5
dev-agent