A failing CI test printed a test key through SSHKey's default repr. Push
credentials, AgentConfig's tokens and MatrixConfig's token are now
excluded from their reprs, so no failure or log line can print them. In
CI's container killed ssh-agents linger as zombies, so the tests count
only live ones.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The clone gets the standard build artifacts in its own exclude file, so
they never show as changes, even to a model's `git add -A`. The
harness's commits leave the same patterns out even when the project
tracks them, as the sandbox's main did. What was left out is reported in
the publish note and the PR description. Found again in bring-up run 3
(sandbox PR #208).
Fixes#80.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Live run 3: the executor ran `git checkout -b <task id>` itself, and
publishing then failed, because git can't create '<task>/milestone-1'
while a branch named '<task>' exists. Milestone and delegation branches
are now '<task>-milestone-N' and 'delegation-milestone-N-F'. D4's
harness-owned branches (M2.1) remove the cause; this removes the
collision on the current loop.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The push credential is either a push-only account's token, handed to git
through a credential helper that names environment variables, never
values, or a deploy key in a short-lived ssh-agent. The remote URL
follows the credential. Configured as gitea.push_user and push_token, or
gitea.push_key.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
execution.SSHHost.execute is now the only way the agent runs anything
(L13). Commands travel as a script on SSH's stdin, so the secrets handed
to a call are never on a command line, and every value handed out is
redacted from the output. Timeouts kill the remote process group, and
an unreachable host is reported as such, not as a failed command.
Workspace and GitRepo run on the build VM. The clone uses the
repository's SSH URL, so no credential is stored in it (#89); fetches
and pushes get the push key over stdin, in a short-lived ssh-agent,
and PRs are still opened from the agent's host with its API token.
The bench runs on the build host too.
Tests reach "the build VM" through a throwaway local sshd (nisse_dev
.testing); CI installs openssh-server. The live push test now pushes
with the sandbox's deploy key (NISSE_PUSH_KEY).
Fixes#89.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A model no longer reads another role's turns as its own (#78). The log
stays one record per work item; what a role is shown is projected from
it by Transcript.view() (views.py), which the planner, executor and
helper now build their context from:
- own events unchanged;
- operator messages labelled with their channel, e.g.
[operator, Matrix] or [operator, comment on PR #165];
- orchestrator facts (a new `fact` event) as [orchestrator] statements;
- the executor sees a helper's handback as [helper report], not its turns;
- a helper sees the approved plan and the executor's milestone work as
one-line narration;
- the planner sees no other role's work at all.
The orchestrator now writes facts at the points the models must know
about: work item opened, plan approved or rejected, milestone started,
finished or stopped, published (with PR number and URL) or not, review
comment answered, next milestone, task completed or abandoned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M1.1, as discussed.
- Schema v4: a work_items table (id, scope, title, status, outcome,
position, data) and an agent row whose active_item points at one
(NULL: idle). Canonical events and intents carry their work item, and
events their actor. Older stores are refused with a clear message:
there was no production data, so no migration (the mechanism stays).
- The log is reached through a Transcript bound to one work item and
one actor: store.transcript(item, actor) reads only that item and
writes with both. The planner, executor, delegation and expert desk
get one per call from the orchestrator instead of holding the store,
so a component can neither read another task's events (the planner
read every earlier task's, #78) nor write without attribution.
store.all_events() is kept for readers that span items (evaluation
capture, dumps). Intents are begun through the transcript or with
work_item=, so each belongs to its item.
- The orchestrator opens a work item on 'use', closes it with its
outcome (completed or abandoned), and carries what the operator said
while idle into the new item as copies with their origin, so a task
sent before 'use' is still planned from (D1). control.py keeps its
state machine through get_state()/transition(), now over the active
work item, until M2.6 retires it.
- nisse_dev.dump shows work items and each event's item and actor.
Tests: test_work_items.py (scoping, attribution, intent ownership,
restart, outcomes, carried messages, a second task never seeing the
first); the suites updated for the per-call transcript.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Architecture L9's role names, for the two roles with a direct
counterpart: the delegation role is now 'helper' and the ask_expert role
'expert', in RoleMap, configuration (roles.helper, roles.expert),
recorded events and tests. 'planner' and 'executor' stay distinct for
now; they become 'driver' when the new loop merges them (Development
Plan M2.3).
Configurations must rename their keys: nisse-dev.example.yaml and
tests/integration/nisse-test.yaml are updated here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M0.2.
- scripts/bringup.sh: start [REF] sets the previous run aside under
/root/nisse-e2e/runs/, exports the ref under test and starts the agent
as the transient systemd unit nisse-bringup, so it outlives the shell
that started it; stop (SIGTERM, clean shutdown), status, logs, and
collect (the journal log and a readable transcript).
- nisse_dev.dump (python -m nisse_dev.dump STATE_DB [--full]): a readable
dump of a state store's state, pending intents and canonical log. Opens
the store read-only, so it is safe against a live agent.
- The agent runs as root on this development VM, as it will on its own
VM. Architecture D3/D6/D7 corrected to match the operator's intent:
installing packages and toolchains on its own VM is class 2, recorded
so the environment can be rebuilt (it was class 3 with provisioning an
open question); egress names OS package mirrors. The plan's M0.2, M4.1,
M4.2 and bring-up guards follow.
Acceptance: a smoke run started, answered status in the test room,
stopped cleanly, and collect produced the log and transcript.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M0.1. Every section reference in code, tests, specs/,
the example config, the tuning guide, and the CLAUDE.md and README files
now cites docs/Architecture.md (P, L, D, O, X, Q) instead of the old
spec's A-H, and docs/Architecture-Build-Spec.md is deleted (git keeps
it). The plan's old-to-new table goes with it.
Mostly mechanical, with a per-site choice where one old section split:
A4 became L2 where it meant stop/abandon/the loop and L3 where it meant
approvals; A10 became L6 in the context-budget code and L7 elsewhere; B2
became D4 for outbound Gitea and publishing and D5 for review intake;
'A3 step N' became the matching step of L8's boot sequence. Comments
describing behaviour the architecture drops (planning always escalating,
thresholds tuned at an old build step) now describe the current code and
point at the Development Plan.
docs/Architecture.md gains one line the rewrite missed: intake outages
delay but never fail work (old B2), now in L5, where the poller and
Matrix code cite it.
Comments and docs only; no behaviour change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Fixes counterexamples 6 and 7 from specs/orchestrator.qnt, the last two.
6. A crash while the executor answered a review comment lost it: its
cursor is saved before the work (at most once), and a restart in
PR_OPEN resumed quietly. Now pr.reviewing marks the comment until
the executor returns (every way it returns, but not a crash), and
boot resumes the watch just before it, with a notice. The mark stays
until it is answered, so a second crash cannot lose it either.
7. 'project complete' left the plan issue open. It now closes it with a
note, after IDLE is saved, as abandon does (shared helper).
The model marks and redelivers, and closes the issue; scenarios 6 and 7
now expect their properties to hold. A mutant without redelivery fails
scenario 6. All eight properties held together over 300,000 traces.
No xfail reproductions remain.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Fixes counterexample 5 from specs/orchestrator.qnt. A crash after the
PR was opened but before PR_OPEN was recorded left the agent in
EXECUTING; the restart question's 'continue' and 'start' then published
again, Gitea refused a second PR, and there was no way back to PR_OPEN.
On the live run the executor model happened to notice the existing
work; recovery must not depend on that.
- Boot (A3 step 4): in EXECUTING, if Gitea shows the milestone's PR
open, adopt it, watch it from the start, and resume in PR_OPEN with a
notice. Gitea unreachable or the PR closed: the restart question, as
before.
- MilestonePublisher.publish is idempotent: after pushing, an open PR
for the branch is adopted (and so updated) rather than a second one
opened. This also covers a PR opened whose response was lost.
- The model adopts too, gains the lost-response outcome, and now counts
PRs opened per milestone (one_pr_per_milestone was a flag the fixed
model could never set). A mutant without adoption fails the new
scenario; the properties held over 200,000 traces each.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Fixes counterexamples 1-4 from specs/orchestrator.qnt: a repeated
'approve', 'approve rev-1' meant for the architecture, approving on both
channels, and 'approve' plus an afterthought all landed on the next
plan, one the operator had not seen.
- Revisions are numbered across the work item (A4 already required
monotonic tags; the code numbered per gate), so rev-N names one plan.
- Each reply is stamped on arrival with the revision that was up: in
Matrix the one whose approval request had been sent (a plain value
the transport thread reads, restored on boot), in Gitea the latest
plan comment older than the reply (board.posts).
- Gate.evaluate applies a reply that names no revision only if its stamp
is the current revision. Otherwise an approval is not applied and
feedback rejects nothing (it is still in the transcript), and the
agent says which revision is up. 'approve rev-N' binds by its number.
- Spec A4 states both rules. The model applies them; scenarios 1-4 now
expect the properties to hold, and both held over 200,000 random
traces and Apalache's exhaustive check to depth 8.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
specs/orchestrator.qnt models control.py: states, gate, pending
questions, comment cursors, publish intents and the worker queue, with
operator messages, Gitea comments, work outcomes (exceptions included)
and a crash at any point interleaving freely. Eight properties.
Seven counterexamples, each written out as a trace in
specs/orchestrator_scenarios.qnt and reproduced against the real code in
nisse-dev/tests/test_quint_counterexamples.py (xfail, strict, until
fixed):
1-4 an approval or feedback lands on a plan the operator had not seen:
a double 'approve', 'approve rev-1' meant for the architecture,
approving on both channels, and 'approve' plus an afterthought;
5 a crash after the PR opens but before PR_OPEN publishes again and
cannot get back to PR_OPEN;
6 a crash while answering a review comment loses it silently;
7 'project complete' leaves the plan issue open.
no_stranding, await_input_has_a_question and execution_needs_approval
held over 200,000 random traces of 40 steps, and under Apalache's
exhaustive check to depth 8, which does find counterexample 1 at that
depth. No behaviour changes here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A phone keyboard's "Approve" did not match the approval pattern, so it
counted as rejection feedback and threw the plan away; "Stop" did not
reach the cancel flag; other capitalised commands got a hint instead of
acting.
gates.command_form() is how a message reads as a command: case,
surrounding whitespace, inner runs of whitespace and a trailing '.' or
'!' do not matter. The approval gate and every command match in the
orchestrator (stop, status, abandon, use, start milestone, next
milestone, project complete, the restart choices) use it. Only matching
changes: the project name in 'use <name>' keeps its case, and feedback
and answers are recorded and passed on as written. A stale revision is
still refused, and approval still needs the whole message to be the
command.
Closes#64.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A4 already names abandon as one of the ways out of a halt (continue,
re-plan, or abandon) and 0.2 closes a work item as completed or
abandoned, but no command did it: from a gate or a question the only way
to drop a task was to edit the state database.
- 'abandon' ends the task from any waiting state and returns to IDLE.
The move is made durable first; then the plan issue and any open
milestone PR are closed with a note (branches kept), and any
unresolved intent is aborted so boot reconciliation does not gate on
work dropped on purpose. A Gitea failure is reported, not fatal.
- Like 'status' it is a command in every state, never read as an answer
or feedback. While work runs, 'stop' comes first, as before.
- The restart / stranded question and the PR_OPEN hints mention it.
- Spec: B1 gains the abandon edge and B2 the command.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- Record a comment cursor per thread (board.last_comment, the field that
already existed unused; pr.last_comment), advanced durably when a
comment is taken up. CommentPoller.resume_after() skips comments at or
below it, so re-attaching needs no request and cannot replay.
- Boot re-attaches the plan watch at an open gate and the review watch in
PR_OPEN from their cursors, catching up on comments posted while the
agent was down (#59). Data saved without a cursor skips what is there,
so an old approval cannot approve the current revision.
- A restart in PR_OPEN resumes with a notice instead of the restart
question (#58). A stored restart question for PR_OPEN returns to it on
either reply instead of re-planning the milestone.
- A stopped or failed review response gets its own pending kind; the
reply returns to PR_OPEN and resumes the watch from the cursor, so
comments that arrived meanwhile are delivered. It used to return to
milestone sign-off, where starting the milestone would open a second PR.
- Comments ignored because of the state leave the cursor alone, so they
are delivered once the agent is back where they apply.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The first live run found four dead ends one at a time: states where no
operator message could move the agent. Most had one shape: work ended,
by an exception or by a path that never transitioned, and left the agent
in PLANNING_* or EXECUTING, where every message is answered "busy".
The worker only prints an exception, so that would keep happening.
- Every job the orchestrator runs for an operator message or a Gitea
comment now goes through _settled(). If the job leaves a working state
behind, whatever the reason, the agent moves to AWAIT_INPUT with the
continue / re-plan choice a restart offers, and says why. An exception
in a waiting state is reported and the state kept, since its commands
still lead on.
- A sign-off state whose gate record is missing asks to plan again
instead of crashing on every reply.
- The stub configurations (no executor, no publisher) complete the
milestone and wait in PR_OPEN instead of resting in EXECUTING.
test_no_stranding.py drives a whole two-milestone project with a fault
injected at each planner, executor and publisher call, and asserts after
every message that the agent is not in a working state and that the
project still reaches 'project complete'.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
On the first live run the executor committed its milestone itself, as
its plan asked. The publisher only published a commit it made, found a
clean tree, reported "no changes to publish", and left the agent in
EXECUTING, where every message is answered "busy".
- publish(): with nothing left to commit, publish anyway when the branch
has commits origin/<base> lacks; report no changes only when it has
none.
- push_followup(): same for review fixes, comparing HEAD with what the
remote branch holds (ls-remote; pushes go by URL, so there is no
tracking ref).
- Nothing to publish now gates to AWAIT_INPUT, and the reply returns to
the still-approved milestone plan.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Clients put the addressee's name in front of a message that mentions
it ("Nisse Dev Testing: re-plan"), and the transport passed that text
on whole, so no command matched: the live run's re-plan was asked again,
and an addressed 'approve' would have counted as rejection feedback.
- A leading mention of the agent (an HTML pill, its user ID, or its
display name followed by ':') is removed before commands are matched.
The display name is learnt from the agent's own member event.
- A message whose leading mention is someone else is skipped: in a
shared room it is addressed to another agent or person.
- Only the start counts; a mention later in the text is left alone.
- The transcript records the message as sent (InboundMessage.original).
The sender is unaffected: it still comes from the event's homeserver-set
sender field, never from the text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A restart in AWAIT_INPUT replaced the saved pending item with a generic
"continue or re-plan?" whose answer nothing handled: the reply cleared
it and left the agent in AWAIT_INPUT with nothing to resume, where 'use'
was refused too. The first live run hit this after a planning failure
and could not be moved by any message.
- Waiting on the operator is a committed state (A3 step 8): on boot, ask
the saved question again instead of replacing it. _await_input keeps
the prompt with the pending item for this.
- The restart question (A3 step 7) records the state it interrupted, and
'continue' / 're-plan' now do what they offer: re-run the interrupted
planning turn, or return an interrupted milestone to its approved plan.
Any other reply asks again rather than being swallowed.
- With nothing pending, 'use <project>' starts over, as the reply
already advised.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It led the executor benchmark (PR #55): 37/40 over eight tasks and 17/20
on the harder four, ahead of claude-haiku-4-5's 15/20. local-fast, the
previous example, scored 0/8 on the harder four. Model selection is on
the dev executor alone for now; the shared-host question (spec Part H)
stays open.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
All seven newer local models passed the original four tasks 20 of 20,
so the benchmark no longer told them apart. The new tasks keep plans
explicit, as the planner is told to write them, and make the work
itself harder:
- trace-bug: a wrong multi-currency total whose cause is a shared
rate helper; discounts and tax go through it too, so a fix at the
call site fails the check.
- refactor-rename: rename load_config across a ten-module package,
leaving load_config_defaults and reload_config alone.
- spec-parser: implement parse_duration from an eight-rule SPEC.md
given three examples, and write tests for every rule.
- long-milestone: eight dependent steps adding due dates to a task
tracker, from the model through storage versions to the CLI.
Self-tests prove that a call-site fix, a blind text replace and a
partial rename all fail their checks.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The report grouped runs by model, serving note and task version, so
runs made at different tuning were silently averaged together. Tuning
is now part of the key, and a tuning column shows how a row's settings
differ from today's defaults; it appears only when some row needs it.
tuning.executor.max_turns goes from 24 to 40. On the harder benchmark
tasks Haiku, the calibration ceiling, hit 24 on both refactor-rename
runs and both long-milestone runs, one of which had finished the work.
The cap guards against a model trying forever; it isn't a budget.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two setup failures found while calibrating the new benchmark tasks,
both of which would also fail real milestones.
Devstral's chat template (Mistral) rejected any turn in which the model
wrote some text before a tool call: the OpenAI renderer made one message
per event, so the turn became a text message and a tool-call message in
a row, and llama.cpp answered 500. The renderer now folds consecutive
user or assistant messages into one, which is the OpenAI format for a
turn anyway. Checked against the live server: the captured failing
request no longer occurs, over six real runs.
LiteLLMClient gave up on any request after a fixed 120 s. A dense 27B
model writing a whole module in one turn takes longer, so the executor
escalated work that was going well. llm.request_timeout_s now sets it
for both clients, default 600.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently.
A benchmark run that waited looked slower than the model is: on
2026-09-27, fast models under the unit-test key's 200k tokens/minute
limit had 20-45 s runs that were probably waits.
Both clients now report last_rate_limit_wait_s after every call, even
one that then fails. The benchmark sums it per run into
rate_limit_wait_s, the report's time column leaves it out, and a new
429 column counts the runs that waited. Results stored before this
load unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.
- Four canned milestones under nisse_dev/bench/tasks: write a function,
fix a bug a failing test exposes, thread an option through several
files, and find one failure in output far past max_output_chars. Each
is an approved plan over a seed project, judged by a hidden check the
model never sees; a reference solution proves each is solvable and
that its check tells the difference
- Runs use the agent's own client wiring, now shared as
make_llm_clients, with delegation and ask_expert off: an escalation is
a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
the model's backing name and declared facts, a note on how it was
served, the task's content hash, the agent's commit and tuning, the
outcome, turns, misbehaviour counts, context trims, tokens, time and
LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
a new model through the context probe, the capability matrix and the
benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
requests it had to trim
- The tuning guide gains "Benchmarking models"
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.
- The catalog reads supports_tool_choice from LiteLLM's model_info;
BudgetedClient sends tool_choice only to models declared true, as
"required" on the normalized route and {"type": "any"} on the
Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
milestone. Finishing with nothing changed is refused once; a prose
reply gets one nudge, then escalates; a call to a tool not offered
counts toward failure_limit, like a failed command; calls after
finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
recognises a proxy fallback, which once credited Claude's answers to
a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
that models call only offered tools, and that a declared
supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
declared and backed up
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every turn re-sent the whole transcript with nothing checking it against
the model's window. On a local model that fails silently: Ollama drops
the start of an over-long prompt, which is where the system prompt and
the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with
no error.
- The model catalog reads max_input_tokens and max_output_tokens from
LiteLLM's model_info, and startup refuses a role on a model with no
declared window
- BudgetedClient wraps both routes. It fits the canonical events before
rendering: older tool results cut to head and tail, then the oldest
turns dropped with a note, keeping calls with their results; a request
that still cannot fit fails with ContextOverflow and is never sent.
The output ceiling is the smaller of config and the model's limit
- Tokens are estimated from characters, pessimistically, and the
estimate is corrected from reported prompt sizes, only ever upward
- The executor takes its scope from the log: the latest plan, which is
the approved milestone plan, goes in its system prompt, and it sees
only events after it
- Tool output keeps its head and, mostly, its tail, so a test runner's
failure summary survives truncation
- tuning.context.keep_recent_turns and safety_margin are new knobs
- scripts/probe_context.py checks a declared window against what the
server does; it confirmed llama3.2 cuts at the declared 4096
- The tuning guide covers the budget, declaring a local model's limits,
and the probe
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With no production instance yet, there is no reason to carry two ways
of configuring the agent. Every environment variable that duplicated a
config key is gone; the environment now supplies only the secrets the
file references by ${…}, plus NISSE_CONFIG and NISSE_ENV to find the
files.
- AgentConfig loads from a file (load, from_file) or a parsed tree
(from_tree, for tests); a missing file is an error naming the example
- The from_env constructors on LiteLLMClient, AnthropicPassthroughClient,
GiteaClient, MatrixConfig and RoleMap are removed, with the env
reading for max output tokens; clients take their settings from config
- The live integration tests share a committed agent config,
tests/integration/nisse-test.yaml, aimed at test resources, and one
conftest that loads it and builds clients. NISSE_LIVE_PLANNER_MODEL
becomes its planner role; NISSE_CAPABILITY_MODELS stays as a per-run
switch
- A missing secret now fails the live tests instead of skipping them
- CI passes only the three tokens
- .env.example is secrets only; spec A5 says config comes from the file
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The thresholds Part H says to tune on the real inference hardware were
hardcoded, so every adjustment would have been a PR. They now live in a
`tuning:` section of a YAML config file, which also replaces flat
environment variables as the home for structure (#32).
- The file (NISSE_CONFIG, default ./nisse-dev.yaml) carries structure
and holds no secrets: credentials are ${VAR} references resolved from
the environment, and a reference to an unset variable is an error
- The environment still wins: every existing NISSE_* variable overrides
its key, and the file is optional, so .env and CI work unchanged
- Validation stays fail-fast and reports every problem at once; unknown
keys are errors, so a misspelt knob fails at startup instead of
silently doing nothing
- The config file joins the state store and .env in the check that
refuses to start when they sit inside the workspace clone
- Tuning covers the executor's turn and failure limits, the consult
budget, the delegation caps, the command timeout and the tool output
limit, which Workspace now takes as a setting; the defaults are the
values the agent already ran with
- llm.max_output_tokens reaches both LLM clients from the config
docs/Tuning-Guide.md explains the knobs for someone learning local
model tuning: how weights, quantization and the KV cache spend VRAM,
what each knob does and what goes wrong at either extreme, and an order
to tune in. nisse-dev.example.yaml documents every key and is checked
by a test so it cannot drift.
Closes#32
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`stop` could not have worked by adding the command alone. Planning
turns, milestones and review responses ran synchronously on the thread
that delivered the message, so the Matrix sync loop read nothing until
the work finished. The state store was also opened on the main thread
and used from the transport and poller threads, which SQLite refuses
with its default same-thread check; the in-memory test transport never
showed it.
A single worker thread now runs every handler and owns the store:
- The worker opens the store, so the same-thread check stays on and
fails loudly if anything touches the store from elsewhere
- Matrix messages are handed to the worker; pollers hand over and wait,
so their cursor never passes an unhandled comment
- A message that arrives during long work gets an immediate "busy; say
'stop' to halt" reply and is recorded, not applied later in a state
the operator was not answering
- Tests keep running inline through InlineWorker, which defers a job
submitted from inside another as the real worker would queue it
`stop` sets a shared Cancel flag straight from the transport's thread.
The executor and delegation loops check it before each model call and
between tool calls; the planner checks it before each turn. Calls left
unrun get a result saying so, so the transcript still renders. A
stopped delegation is discarded like a failed one. The orchestrator
moves to AWAIT_INPUT and asks why the work was stopped. The reply is
recorded for the model, and a stopped planning turn resumes with it as
the answer. `stop` with nothing running says so and changes nothing; at
a gate it is no longer taken as rejection feedback.
Closes#40
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Route selection sniffed model names: anything starting "claude-" or
"anthropic/" took the vendor-native passthrough, everything else took the
normalized route. That works only while an alias names its vendor. A
semantic role alias like "remote-architecture" is meant to point at
Anthropic today and somewhere else tomorrow, and the name says nothing
either way -- so it would have taken the normalized route regardless of
what was behind it, silently losing the fidelity A13 wants for delegations.
LiteLLM already knows the answer. /model/info reports each alias's backing
model, provider, and declared tool support, so the agent asks at startup and
routes on the provider it discovers. That keeps vendor knowledge where A5
already put it -- in the proxy -- rather than duplicating it as name patterns
here.
There is deliberately no name-based fallback: PASSTHROUGH_PREFIXES and
has_native_adapter are gone, and select_client now requires a catalog. A
second heuristic could only ever disagree with the proxy. An unknown model
raises instead of guessing a route.
Misconfiguration is therefore loud and early. build_agent validates every
role against the catalog before it clones anything, naming all the bad ones
at once. Discovery is per key -- /model/info returns only what the presented
token may use -- so the same check catches a role pointed at a real model
this key cannot reach. An unreachable proxy is reported separately, since an
outage and a typo call for different responses from the operator.
Two things now get recorded rather than inferred: the resolved model is
pinned alongside the role mapping at task start, for the reason A11 already
pins roles, and DelegationStarted carries backing_model so A16 eval history
is not averaged across whatever an alias happened to point at.
A model LiteLLM declares tool-less warns at startup rather than refusing;
A15 expects weak local models to be accommodated, and they are observed
driving tool loops anyway.
Verified live: all four aliases the production key can see route correctly
through discovery, and each chosen route works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An exhausted budget was a special case of something more general: the
work is not at fault. A network blip, a timeout, a 5xx from the proxy,
or a rate limit that did not clear all leave a delegation branch that
may be perfectly good, and discarding it would throw away work for a
reason that has nothing to do with the work.
UpstreamUnavailable now names that family, with BudgetExhausted a
member of it — distinguishable where the difference matters, since a
budget will not clear within the hour while a network blip will. A
delegation stopped by any of them is "interrupted": branch kept,
intent left pending, and A3's resume rule picks it up.
The benefit of the doubt stops at the server's own door. A 4xx that is
not a rate limit means the request itself was wrong, and preserving
branches on the strength of one would hide a real bug — so those still
fail and clean up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Exceeding a LiteLLM spend limit takes external models away for the
rest of the day, which makes three existing gaps consequential rather
than untidy.
- A13 calls the delegation caps hard on "turns, tokens, and wall
clock", and the module docstring repeated it, but the token cap was
decoration: nothing counted, and result.tokens was always zero. It
is now metered from the usage the provider reports, so a confused
delegate cannot spend a day's allowance. A5 keeps the *money* cap on
the LiteLLM key, which is a different thing; this is a scope cap
- The normalized route sent no ceiling at all, so truncation detection
there had nothing to detect against. Both routes now send one
- A budget refusal is recognised distinctly. It is not a rate limit:
retrying cannot help, so the planner no longer burns a second call
to fail identically
And a delegation cut off by an exhausted budget is now "interrupted"
rather than "failed", which preserves its branch and leaves its intent
pending. The work on that branch may be perfectly good; discarding it
because the budget ran out would throw away work for a reason that has
nothing to do with the work. A3's resume rule already knows how to
pick such a delegation back up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measuring Opus found a bug that had been silent all along: nothing
checked stop_reason or finish_reason, so a response cut off mid-answer
was parsed as a complete one.
The symptom was a submit_plan tool call whose required `content` field
was simply absent — structurally valid, substantively empty, and
indistinguishable from a model that chose to send nothing. The cause
was our max_tokens of 4096: Opus writing an architecture plan
routinely spends 3,000-4,000 output tokens, and was observed at 4,064
against that ceiling. Roughly one run in three crossed it.
Worse than the empty plan is what the same failure does elsewhere: a
truncated write_file writes half a file and reports success.
- Both routes now raise TruncatedResponse when the model was cut off.
It subclasses LLMError, so the planner's retry and the executor's
and delegation's gating already handle it without learning a new
type: a cut-off turn is a failed turn
- The default ceiling is 16384 and configurable via
NISSE_MAX_OUTPUT_TOKENS. Output is billed by what is produced, not
by the ceiling, so a low limit buys nothing and costs silent
truncation
Verified: three consecutive Opus planning turns now return plans of
5,700-6,200 characters where one in three previously returned nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The arrival of non-Anthropic models made a latent bug observable: the
delegate's client was hardcoded to the Anthropic passthrough, so
pointing NISSE_ROLE_DELEGATE at gpt-6-astra or a local model would
have sent Anthropic-shaped /v1/messages bodies to something that does
not speak them. A5 promises the frontier tier is swappable "as a
configuration change, not a refactor", and it was not.
select_client now picks the route from the model: the vendor-native
adapter where one exists, LiteLLM's normalization otherwise — which is
exactly what A13 prescribes for a vendor whose renderer adapter has
not been written, pending validation against the A16 eval set.
Also adds the capability matrix Part H asks for. It says the
thresholds that depend on local capability — A12's failure count,
A14's consult budget, A15's manifest size — must be tuned "on the real
hardware", and that needs measurement rather than reputation. The
harness probes each model for the things that actually break: whether
it calls a tool at all, whether the plan it submits has substance
(llama3.2 emitted "{"), whether it can consume its own tool result,
and whether prompt-level JSON holds. Opt in with
NISSE_CAPABILITY_MODELS; it spends real tokens across several models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Answering "is the workspace where the agent runs, or what it works
on?" — it is what it works on, and conflating the two is dangerous
rather than merely untidy.
The agent commits its workspace with git add -A and pushes it, so
anything inside the clone reaches the project repository. A .env
placed there would publish the LiteLLM, Matrix and Gitea tokens as a
side effect of the agent working correctly: silently, once, and
probably unnoticed. Startup now refuses both that and a state store
inside the clone, and .env.example documents the intended layout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing had ever assembled the agent. main.py was still the hello-world
placeholder from the scaffold, and no code outside the test suite
constructed an Orchestrator — which is how two fully built, fully
tested components came to be reachable from nothing: the delegation
harness, and the Gitea comment poller. Unit tests that construct a
component directly cannot catch that; only assembly can.
- config.py validates every variable before anything is built and
names all of them at once, rather than failing at first use twenty
minutes into a milestone on the call that mattered. Whitespace-only
counts as missing, since a token of spaces passes a truthiness check
and fails at the first request instead
- agent.py builds the whole graph and owns startup and shutdown order:
reconcile before accepting commands, and on stop, stop watching,
then stop talking, then close the store last
- The Gitea account is read via whoami rather than configured: it names
the identity every permission is evaluated against, and proves the
token works before anything depends on it
- Poller lifecycle is now a state-machine concern: watch the plan issue
while a gate stands, the PR while it is open, and stop on leaving.
The orchestrator owns when to watch; an injected factory owns how to
build one
- The executor reports each turn, because it runs synchronously inside
the transport callback: without it the agent is silent for a whole
milestone and a working run looks exactly like a hung one
- main.py loads .env, builds, and shuts down cleanly on SIGINT/SIGTERM
The entry-point tests changed shape: nisse-dev no longer greets, it
refuses to start unconfigured and says everything that is missing. Both
tests now run with a deliberately empty environment and a .env path
that does not exist, because the working directory's .env holds live
credentials and a test must never be one cwd away from starting a real
agent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two gaps found by auditing spec against code, the second created by
7b landing the harness without its boot-time half.
**The harness was not connected to anything.** Only a live test
constructed a Delegation; A12's failure-count trigger still gated to
the human, which A13 describes as the fallback rather than the path.
Consecutive command failures now hand the sub-task to the delegate,
with the failures in the brief so it need not rediscover what the
local model learned the hard way. A usable handback lets the local
loop carry on; anything else escalates, because a cap breach leaves a
half-edited tree.
**A3 step 5 was unimplemented.** A crash mid-delegation failed safe —
boot gated on "restarted mid-EXECUTING" — but always gated, even on
the clean cut the spec says should resume, and left the branch orphaned
because _abandon only runs while the process lives.
- A delegation now records an intent before the branch exists, and
settles it whichever way the delegation ends; an aborted delegation
is a finished one, and leaving it pending would make the next boot
believe work was still in flight
- delegation_verdict implements the clean-cut test as an observable
check, never model self-assessment: settled if a completion follows
the start, absent if the branch was never created, gate on a
tool_call with no recorded result, resume otherwise
- recover() sweeps at boot, announcing a resume with a notice rather
than a question, since a clean cut needs no decision from the
operator, and leaving an unclean one pending so the generic sweep
gates on it
- resume() continues the same delegation rather than starting a
second: no new branch, no second delegation_started, because A10
defines a session as the delegation, not the process lifetime
Honest limit, stated in the code: "the branch sits at the expected
commit" is approximated by the branch existing and carrying the work,
since per-turn shas are not recorded in the log, so a branch rewritten
underneath us would go undetected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes step 7.
- expert.py: one stateless call for a question needing no repo access.
Structured output is contracted at the prompt level, since vendors
differ on JSON enforcement, so the reply is parsed and validated
locally with one retry — tolerating a fenced code block, because
models wrap JSON habitually, but not missing fields
- The per-task budget is counted from the canonical log rather than
memory, so it survives a restart and cannot hand a circling executor
a fresh allowance. Exhaustion raises rather than answering: a caller
out of consults should get on with the work or escalate
- Consults are recorded as delegation_started/completed with role
consultant. A16 asks for a "consult/delegation eval set", and
reusing the composite events gets consult evaluation for free rather
than inventing a parallel record
- evals.py pairs starts with completions into eval records, with
verdicts kept outside the canonical log: the log records what
happened, a verdict is a later judgement about it, and judgements
get revised. Schema v3 adds the outcomes table, so a v1 store now
traverses two migrations — a better exercise of the hook than one
- ask_expert is offered to the executor as a tool only when a desk is
configured, in addition to the observable escalation triggers and
never instead of them (A12). Consulting is not mutating, so a
completion claim backed only by consults is still challenged
Found live: passing an empty message list and putting the question in
the system prompt fails outright — the Messages API requires at least
one non-system message. The question now travels as a synthesised user
message that is never appended to the log, which keeps the consult
stateless while remaining a valid request.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI failed on the delegation branch with a 429 from the proxy: token
limit 200000, 17399 remaining. Not a defect — the live delegation
tests are token-hungry, and every push to a PR branch fired two
identical suites concurrently, since the push and pull_request events
both matched.
- Push now only triggers on main. A PR branch is already covered by
the pull_request event, so this halves both the wall time and the
tokens spent per branch
- A 429 is retried once before being surfaced. Losing a ten-turn
delegation to a transient limit wastes far more than waiting out
the window, and retrying forever would hide a real ceiling. The
sleep is injectable, so the existing 429 test no longer waits 20
seconds for real
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A frontier model receives a scoped sub-task, works autonomously with
full tool access, and hands back; the local loop reads the handback
note and the file state, never the reasoning.
- Branch at delegation start, and abort discards it. That required
committing each turn's work onto the branch: without commits,
deleting a branch discards nothing, so "abort discards the work"
was a comforting phrase rather than a fact. The caller's own
in-flight changes are committed to their branch first, so an abort
cannot destroy work nobody delegated
- Hard caps on turns and wall clock; a breach is a defined failure
return, and its result is not usable, so the caller gates rather
than resuming on a half-edited tree
- Every event is checkpointed as it arrives, so a crash mid-delegation
leaves the branch at a known commit and a truthful transcript
- Handback is validated locally: blank fields or a non-boolean success
flag are refused and the delegation continues rather than accepting
a useless note
Two failure modes found by running it against a real model:
- A success claim with nothing changed was accepted. The handback plus
file state is the entire interface back, so a note describing work
that does not exist is worse than no note. Success now requires a
successful mutating tool call; reporting failure still does not,
since "this is impossible" is a legitimate finding
- The delegate invented the absolute path /workspace, wrote there, ran
its tests there, and handed back truthfully — with the repo empty
and git seeing nothing. Writes outside the workspace are now refused
with a message naming the real root, and both prompts state where
the workspace is instead of leaving the model to guess. This is not
sandboxing: work outside the repo is invisible to git, so the
milestone would publish an empty PR
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The second renderer adapter, and the concrete case for A13's decision
to normalise in the renderer rather than on the wire. The two vendors
disagree structurally, not cosmetically: the system prompt is a
top-level field rather than a message, a tool call is a tool_use
content block rather than an array, a tool result is a block inside a
user message rather than a "tool" role, and tool schemas use
input_schema rather than parameters. A middlebox flattening those
would be lossy exactly where a long-running delegation lives.
- render_anthropic.py builds vendor-native /v1/messages bodies from
the canonical log, coalescing adjacent same-role blocks so a
tool_result lands directly after its tool_use
- Tool manifests are declared once in OpenAI shape and converted per
vendor, so adding a tool does not mean editing two places
- AnthropicPassthroughClient implements the same LLMClient protocol,
so callers pick a route rather than a different kind of client, and
maps provider tool ids to canonical ones at the boundary
- Dangling tool calls get a placeholder result here too; this API
rejects them more strictly than the OpenAI shape
- Vendor-opaque reasoning blocks are dropped for now with the reason
recorded: A10 wants them stored and never replayed across sessions,
and the canonical schema has no event for that yet
Live tests prove the body is accepted as sent, that a tool_result
turn round-trips, and — as A5 asks — that prompt cache accounting
survives the passthrough rather than being flattened by the proxy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous fix challenged a completion claim made before any tool
call, but treated every tool as work — so a model that ran one
list_dir and then declared the milestone done was believed, and CI
caught the same empty milestone again.
Reading a file or listing a directory is the model orienting itself.
Only write_file and run_command now count, so a completion claim must
be backed by something that changed or verified the tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI caught the executor recording an empty milestone as a success: the
model answered with prose on its first turn, and the loop reads "no
tool call" as "the plan is done". It had therefore finished a
milestone having touched nothing, and the milestone would have gone
on to publish with no changes.
A completion claim is now challenged once when no tool has been
called yet — the same shape as the planner's nudge — and a model that
persists escalates instead of reporting success. After the first tool
call the claim is taken at face value, as before.
The live milestone test failed roughly one run in five on this; four
consecutive runs pass now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- approvals.py: PlanBoard opens one issue per project and posts each
plan revision as a comment carrying its rev tag. The state store
keeps only a pointer, so gate state can be re-derived from Gitea
after a state-volume loss — external truth wins
- Gitea comments are a second approval source: revision binding, the
typed word approve, and first-approval-wins work exactly as over
transport, with a per-channel allowlist because a Gitea login is
not a Matrix MXID
- Outcomes are echoed to the other channel, so the two records cannot
silently disagree about what was approved
- A Gitea outage degrades durability of the record, not the ability
to approve: planning proceeds and Matrix remains a complete channel
- Live test now asserts the server refuses the agent's own merge and
its push to the protected branch. That property was previously
true only because our code did not attempt it; branch protection
makes it structural, and this test fails if the protection is ever
relaxed
- Fixed a race in the process-group test: killing a group is
asynchronous and the grandchild is an orphan, so its exit is not
synchronised with our return; the test now waits for death
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- poller.py: CommentPoller polls PR comments rather than taking
webhooks, keeping the VM outbound-only (B3). On an outage it backs
off exponentially, caps, and retries indefinitely while logging:
polling is pure intake, so an outage delays a reaction but never
fails a task or forces a transition
- prime() marks the existing thread as seen without delivering it, so
starting a watch does not re-address feedback already handled
- Review loop: a comment in PR_OPEN enters the canonical transcript,
the executor responds, and the fix is pushed to the same branch with
a reply on the PR. No intent is recorded for a follow-up push — the
PR already exists, so a repeated push is harmless, unlike opening a
second PR
- Comment authors are checked against a separate Gitea operator
allowlist: a PR comment is an instruction channel into an agent that
writes code, so it is operator-scoped, and a Gitea login is not a
Matrix MXID (A4 per-channel allowlists)
- Orchestrator now holds a lock: transport and the poller deliver on
separate threads, and state transitions must not interleave
- Backoff timings are injectable, so the outage path is tested in
milliseconds instead of being untestable
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes the B1 loop past EXECUTING, and answers the question raised
reviewing #6: PR acceptance is a milestone's final gate, so there is
no separate FINISHED state — completion returns to milestone planning
or to IDLE.
- publish.py: MilestonePublisher commits the work on a task-namespaced
branch, pushes, and opens a PR carrying the approved plan. Intent is
recorded before the first externally visible act, so a crash
anywhere in the sequence leaves a durable record and boot
reconciliation asks Gitea whether the PR exists rather than opening
a second one
- Nothing to commit is reported, not raised: the executor may
legitimately conclude there was nothing to change
- control.py: PR_OPEN state with EXECUTING -> PR_OPEN and
PR_OPEN -> {MILESTONE_PLANNING, IDLE}; operator commands
'next milestone' and 'project complete'
- A configured publisher always contributes its own reconciler at
boot, so an interrupted publish is settled against external truth
without the caller having to remember to register it
- A publish failure gates to the operator rather than retrying into a
possible duplicate
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dongarandClaude Opus 5.5
dev-agent