- Every PR: lint, unit tests across the runner's cores (pytest-xdist), 16
property walks from a seed that moves with the run number, integration
tests, and the end-to-end smoke tests (the slice, a conversation filed
as work, a question; `smoke` marker).
- A PR labelled e2e:full, a push to main, or a manual run: all 48 walks and
the whole end-to-end suite. Adding e2e:full to a PR runs it; any other
label runs nothing.
- A published release runs the full set before its image is built.
- The walk's seeds come from NISSE_WALK_SEEDS / NISSE_WALK_FIRST (48 from 0
by default).
Measured here: the unit tier on 2 workers took 3m49s against 6-8 minutes
serially.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- Questions never change state (L5, #77): a message ending in "?" to a
waiting project item starts a question run (reply and look-only tools,
an instruction to change nothing), and the item goes back to the very
wait it was in: a restatement up stays up and a go binds to it; the
agent's own question stays open.
- Explanations: a restatement gives its reasons and the main alternative;
a revision says what changed (restate(changed), sent back without it);
a milestone's summary says what was done, verified and left open.
- Progress in prose (#42): the driver's and a helper's own words for a
turn are posted, trimmed, at most once a minute per item; `details`
shows the last tool calls; a message arriving mid-run is acknowledged at
once ("Noted; ...").
- CI: the e2e job's comments and the plan no longer claim a newer push
cancels the run it supersedes (Gitea here doesn't); the concurrency
block goes.
Tests: conversation unit tests (questions in a go-ahead and an answer wait,
an unanswerable question, revisions, progress rate and trimming, details,
mid-run notes), the helper's progress, the walk asking questions in every
state with a questions_change_nothing property (routing questions the old
way fails 32 of 48 seeds), and an e2e question while a restatement is up.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- propose_project(name, purpose, decisions) in a conversation goes up for
a go-ahead (checked: a slug, not an existing project, no trailing
question).
- On go, the provisioning checklist (D1): a repository with a README,
protection on its default branch, the agent as a collaborator, a build
VM, a deploy key, and the project entry as YAML with <...> to fill in.
- On "done", the agent reads its configuration again and adds the new
project live (build_loop_agent(load=...); only additions), then checks
what was provided (D4, verified, not assumed): permissions and
protection as it sees them, then on a throwaway branch with an empty
commit, a branch push must succeed and a push to the default branch and
merging its own PR must be refused. The check's PR is closed and its
branch deleted. Failures are said per item; nothing starts.
- Checked: the tracking issue (the distilled decisions, carrying the new
item's ID), and the project's item, agreed and labelled `design`, whose
first milestone is ARCHITECTURE.md and the build plan. A conversation
waiting on provisioning doesn't close as idle.
- The walk replays exactly from its seed (item IDs too), and event IDs
stay unique across simulated restarts, as a server's are.
- The test config holds output to 8192 tokens, under the test key's
per-request cost cap.
Tests: new-project unit tests (checklist and its YAML parse as config, done
before the entry, what isn't set up, a checked project's record and first
PR, an existing name), protection enforced by a hook on the fake forge,
the walk proposing projects and saying done, and an e2e new project in a
repository the test operator provisions from the agent's checklist.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- Every e2e role runs on qwen3.8-27b on the RTX 4090 (e2e_roles();
NISSE_E2E_MODEL overrides, e.g. claude-haiku-4-5). The first full run
passed 7 of 8 in 26 minutes for $0.0000; the eighth was a real finding,
fixed with the new-project PR.
- scripts/e2e.sh reports a run's spend on the test key and the day's total
against its budget: above $0 on the local model means a paid fallback.
- CI runs the suite as its own job on pull requests (not pushes to main),
on a throwaway sshd (--local), cancelling a run a newer push supersedes,
so the checks job still reports in minutes.
- The plan says so: e2e on every PR; locally only -k / --lf.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Every live test that touches a repository gets one of its own from
e2e_support.fresh_repo, made by the test operator as an operator makes a
project's (D4, D8): seeded from tests/integration/seed/ (the sandbox's
package, without past runs' leftovers), the agent's account a collaborator
with write access, main protected (pushes only by the owner, an approval to
merge), and a deploy key generated for the run. It is deleted afterwards
(NISSE_E2E_KEEP_REPOS keeps it); scripts/e2e-cleanup.sh deletes leftovers.
- Both e2e fixtures use it; per-run base branches are gone. The merge
helper approves first, as the protection now requires.
- test_gitea_live uses it, so CI needs NISSE_OPERATOR_TOKEN and no longer
writes the sandbox deploy key (NISSE_PUSH_KEY).
- A live test of the helper: seeded, shared with the agent, protected
against it, deleted.
- nisse-test.yaml's project entry is a placeholder; pytest and ruff skip
the seed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- A top-level message that isn't a command is a conversation: an item in
the `general` scope (L1), threaded under the operator's message, with
`reply`, `ask_human` and `propose_work` and no workspace or execution.
`general` is reserved as a project name.
- propose_work(project, title, summary) goes up as a revision for a
go-ahead (or a plain yes). On go, under a file_issue intent (L8), the
record issue is opened in that project, assigned to the agent and
carrying the new item's ID; its item starts agreed, with the summary as
its brief. Intake skips any issue carrying an ID (D1); boot adopts a
filed issue or puts the proposal up again.
- A conversation closes after tuning.loop.general_idle_s (default a day);
a reply in its thread then starts a new conversation that continues it.
Tests: conversation unit tests (threading, filing, the marker, an unknown
project, changes, idle close and continuation, a plain yes, both crash
cases, commands at top level); the walk with conversations, proposals and
idle time, and records_have_one_item / conversations_do_no_work; an e2e
conversation that becomes filed work.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- Config: `projects: {name: entry}` with repo, base_branch, workspace,
build_host, push credential, labels (defaults added to every issue),
secrets by ${VAR} reference with visibility (commands | harness), and
egress (recorded, not enforced). Every problem in every entry is
reported at once; an inline secret, a bad visibility, a repeated repo or
a non-slug name is an error.
- The old loop runs exactly one project (only_project) and refuses more.
- The new loop is per project: each its own clone, workspace (with its
commands-visible secrets, redacted on return), push credential and
milestones; intake polls every project's repository; items are scoped
to their project for every run, publish and PR watch. Issues are named
project#14 when there is more than one project, and a bare #14 that
matches two is asked about. Switching projects parks the one worked on
last (all others after a boot).
- A restatement that ends in a question is sent back to the driver, not
posted: the e2e driver kept asking "Is this the right interpretation?",
which a plain yes can't answer.
Tests: config entries and errors, two-project assembly, two-project loop
tests (intake, own workspaces and repos, #14 in both, do-next, label
defaults, parking across projects), the property walk over two projects
with a new own_repository_only property (an item using the first
project's repo fails 15 of 16 seeds), and the restatement check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- Every milestone branch is cut from the latest default branch: the next
milestone builds on the merged previous one, and a redo after a closed PR
drops the rejected work. Uncommitted work comes along.
- While the item's PR is open, start_milestone is refused, with the PR's
state read from Gitea at that moment, not the last watch round.
- finish_milestone(final): earlier PRs say "Part of #N", the last "Closes
#N"; unsaid, it's final, as before.
- close_item with the item's PR still open waits for the merge (L2: a
closed item leaves nothing open).
- The driver's instructions cover ARCHITECTURE.md and build plans under
docs/plans/ (D2); the harness never parses a plan.
- Parking stashes uncommitted work and pushes it to <item>-parked, put back
on resume, instead of committing it onto the item's branch. The walk
found a half-done review fix parked onto a PR that had merged meanwhile,
where it was lost. Architecture D4's parking line says so.
Tests: unit tests for fresh milestone branches, the merge wait (including
a merge not yet watched), redo after a closed PR, Part of/Closes, close
waiting on an open PR, and the parked-fix regression; the walk's new
milestones_start_fresh property (fork-from-current fails 9 seeds); an e2e
two-milestone feature.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- Commands that name their item ("go #14", "abandon #14", "stop #14")
apply to it wherever they're said; go, approve and abandon alone at top
level are asked about (L5). stop is never taken as an answer.
- A plain-language reply to a restatement is read by GoAheadReader, one
model call that agrees only on a clear yes; questions never reach it. The
agent says "taking that as a go-ahead on rev-N" first, bound to the
revision on screen at arrival (L3, L5, D1).
- Labels steer (D1): `bug` goes ahead after tuning.loop.bug_grace_s
(default 600 s) unless the operator replies; `discuss` isn't offered
restate until the operator has spoken; `bug`, `discuss` and `design` add
to the driver's instructions. A restatement asks no questions.
Tests: label and command unit tests, the property walk with labelled
issues, plain yeses, named gos and time passing (grace_only_unanswered,
discuss_talks_first), and e2e tests for a bug proceeding after its grace
period and a plain-language go-ahead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Several work items on the new loop (L1, D4):
- Switching parks the item on the workspace: its work is committed and its
branch pushed; resuming checks the item's branch back out. Branches of
open items are never set aside by a fresh workspace.
- Next item: the operator's "do #N next" first, otherwise the oldest.
- Issues with open Gitea dependencies are blocked until they close, checked
at intake and on every round; status shows blocked items and the line.
Tests: queue unit tests, the property walk over several issues with
dependencies and a new parked_work_safe property (it catches parking
disabled), and an end-to-end test with two issues where B depends on A.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
From the new loop's first live run (sandbox #255, PR #256):
- A question asked on the PR is answered on the PR (an inline one quoted
with its file and line, since the API can't reply in its thread), with
a pointer in the item's Matrix thread; one asked in Matrix stays there.
- wait_for_review: when a review needs no change, or only the operator's
merge is left, the item waits on the merge instead of on an answer.
- finish_milestone takes the PR's title from the driver.
The e2e test's merge waits out Gitea's brief 'try again later' after a
push.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The first live run of the new loop's e2e test found that a fact recorded
while start_milestone was in flight landed between the tool call and its
result; both API formats reject that, so every later request in the run
failed. Views now keep each result right after its call. Scripted test
models check the APIs' ordering rules, so this can't pass unit tests
again.
The e2e tests no longer depend on the sandbox's history: each run asks for
a uniquely named function and works against a base branch of its own,
deleted afterwards. The review step asks for a change that can't already
be there. The scripted operator gives up after five fruitless answers, and
each run keeps its state store for inspection. The driver's prompt says
review fixes stay in the current milestone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
From an issue the scripted operator files and assigns in Gitea, through
the restatement, go-ahead, milestone and PR, a review comment answered
with a pushed fix, to an approved and merged PR and a closed item and
issue. The operator acts in Gitea through its own test account
(NISSE_OPERATOR_TOKEN) and in the item's thread through the in-memory
transport. Runs from scripts/e2e.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Transport.post(text, thread=...) returns the event ID, which can root a
thread; a threaded post carries m.thread with the reply fallback for
clients without threads. Inbound messages carry their event ID and thread
root: a message in a thread gives its root, a plain reply joins the
thread of the message it replies to when known, and the quoted lines of a
rich reply are no longer part of the text. The in-memory transport plays
threads for tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
tests/integration/test_end_to_end.py drives the whole assembled agent
(real models, the Gitea sandbox, a real build host) through a milestone
with a scripted operator, and checks what bring-up run 3 checked by hand:
a PR holding only the work, pushed with the push credential; no
credential in the clone; the planner's view carrying facts, not the
executor's turns. It cleans up after itself on the sandbox and the build
host.
scripts/e2e.sh runs it on the bring-up build VM (or --local). It has its
own marker, e2e, kept out of check.sh and CI since it spends tokens.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The push credential is either a push-only account's token, handed to git
through a credential helper that names environment variables, never
values, or a deploy key in a short-lived ssh-agent. The remote URL
follows the credential. Configured as gitea.push_user and push_token, or
gitea.push_key.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
execution.SSHHost.execute is now the only way the agent runs anything
(L13). Commands travel as a script on SSH's stdin, so the secrets handed
to a call are never on a command line, and every value handed out is
redacted from the output. Timeouts kill the remote process group, and
an unreachable host is reported as such, not as a failed command.
Workspace and GitRepo run on the build VM. The clone uses the
repository's SSH URL, so no credential is stored in it (#89); fetches
and pushes get the push key over stdin, in a short-lived ssh-agent,
and PRs are still opened from the agent's host with its API token.
The bench runs on the build host too.
Tests reach "the build VM" through a throwaway local sshd (nisse_dev
.testing); CI installs openssh-server. The live push test now pushes
with the sandbox's deploy key (NISSE_PUSH_KEY).
Fixes#89.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M1.1, as discussed.
- Schema v4: a work_items table (id, scope, title, status, outcome,
position, data) and an agent row whose active_item points at one
(NULL: idle). Canonical events and intents carry their work item, and
events their actor. Older stores are refused with a clear message:
there was no production data, so no migration (the mechanism stays).
- The log is reached through a Transcript bound to one work item and
one actor: store.transcript(item, actor) reads only that item and
writes with both. The planner, executor, delegation and expert desk
get one per call from the orchestrator instead of holding the store,
so a component can neither read another task's events (the planner
read every earlier task's, #78) nor write without attribution.
store.all_events() is kept for readers that span items (evaluation
capture, dumps). Intents are begun through the transcript or with
work_item=, so each belongs to its item.
- The orchestrator opens a work item on 'use', closes it with its
outcome (completed or abandoned), and carries what the operator said
while idle into the new item as copies with their origin, so a task
sent before 'use' is still planned from (D1). control.py keeps its
state machine through get_state()/transition(), now over the active
work item, until M2.6 retires it.
- nisse_dev.dump shows work items and each event's item and actor.
Tests: test_work_items.py (scoping, attribution, intent ownership,
restart, outcomes, carried messages, a second task never seeing the
first); the suites updated for the per-call transcript.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Architecture L9's role names, for the two roles with a direct
counterpart: the delegation role is now 'helper' and the ask_expert role
'expert', in RoleMap, configuration (roles.helper, roles.expert),
recorded events and tests. 'planner' and 'executor' stay distinct for
now; they become 'driver' when the new loop merges them (Development
Plan M2.3).
Configurations must rename their keys: nisse-dev.example.yaml and
tests/integration/nisse-test.yaml are updated here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M0.1. Every section reference in code, tests, specs/,
the example config, the tuning guide, and the CLAUDE.md and README files
now cites docs/Architecture.md (P, L, D, O, X, Q) instead of the old
spec's A-H, and docs/Architecture-Build-Spec.md is deleted (git keeps
it). The plan's old-to-new table goes with it.
Mostly mechanical, with a per-site choice where one old section split:
A4 became L2 where it meant stop/abandon/the loop and L3 where it meant
approvals; A10 became L6 in the context-budget code and L7 elsewhere; B2
became D4 for outbound Gitea and publishing and D5 for review intake;
'A3 step N' became the matching step of L8's boot sequence. Comments
describing behaviour the architecture drops (planning always escalating,
thresholds tuned at an old build step) now describe the current code and
point at the Development Plan.
docs/Architecture.md gains one line the rewrite missed: intake outages
delay but never fail work (old B2), now in L5, where the poller and
Matrix code cite it.
Comments and docs only; no behaviour change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The test checks that a request reaches the executor model through
LiteLLM and text comes back. It also asserted the model obeyed "Reply
with exactly one word: pong", which Haiku once answered with "ping" and
failed CI on #66. Assert non-empty text instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
It drove the orchestrator until EXECUTING and accepted EXECUTING as the
end state, which with no publisher configured was the stranding this
branch removes: the milestone now finishes in PR_OPEN. Return once
'start milestone 1' is sent, since the milestone runs inside it, and
expect PR_OPEN or AWAIT_INPUT.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.
- Four canned milestones under nisse_dev/bench/tasks: write a function,
fix a bug a failing test exposes, thread an option through several
files, and find one failure in output far past max_output_chars. Each
is an approved plan over a seed project, judged by a hidden check the
model never sees; a reference solution proves each is solvable and
that its check tells the difference
- Runs use the agent's own client wiring, now shared as
make_llm_clients, with delegation and ask_expert off: an escalation is
a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
the model's backing name and declared facts, a note on how it was
served, the task's content hash, the agent's commit and tuning, the
outcome, turns, misbehaviour counts, context trims, tokens, time and
LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
a new model through the context probe, the capability matrix and the
benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
requests it had to trim
- The tuning guide gains "Benchmarking models"
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.
- The catalog reads supports_tool_choice from LiteLLM's model_info;
BudgetedClient sends tool_choice only to models declared true, as
"required" on the normalized route and {"type": "any"} on the
Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
milestone. Finishing with nothing changed is refused once; a prose
reply gets one nudge, then escalates; a call to a tool not offered
counts toward failure_limit, like a failed command; calls after
finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
recognises a proxy fallback, which once credited Claude's answers to
a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
that models call only offered tools, and that a declared
supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
declared and backed up
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every turn re-sent the whole transcript with nothing checking it against
the model's window. On a local model that fails silently: Ollama drops
the start of an over-long prompt, which is where the system prompt and
the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with
no error.
- The model catalog reads max_input_tokens and max_output_tokens from
LiteLLM's model_info, and startup refuses a role on a model with no
declared window
- BudgetedClient wraps both routes. It fits the canonical events before
rendering: older tool results cut to head and tail, then the oldest
turns dropped with a note, keeping calls with their results; a request
that still cannot fit fails with ContextOverflow and is never sent.
The output ceiling is the smaller of config and the model's limit
- Tokens are estimated from characters, pessimistically, and the
estimate is corrected from reported prompt sizes, only ever upward
- The executor takes its scope from the log: the latest plan, which is
the approved milestone plan, goes in its system prompt, and it sees
only events after it
- Tool output keeps its head and, mostly, its tail, so a test runner's
failure summary survives truncation
- tuning.context.keep_recent_turns and safety_margin are new knobs
- scripts/probe_context.py checks a declared window against what the
server does; it confirmed llama3.2 cuts at the declared 4096
- The tuning guide covers the budget, declaring a local model's limits,
and the probe
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
NISSE_CAPABILITY_MODELS was neither a secret nor configuration: which
models to measure is a question for one run, so it is now an argument,
--capability-models=a,b, declared in tests/conftest.py where pytest
reads options at startup. With no models the matrix skips as before, so
CI is unchanged. The last non-secret NISSE_* variable other than
NISSE_CONFIG and NISSE_ENV is gone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With no production instance yet, there is no reason to carry two ways
of configuring the agent. Every environment variable that duplicated a
config key is gone; the environment now supplies only the secrets the
file references by ${…}, plus NISSE_CONFIG and NISSE_ENV to find the
files.
- AgentConfig loads from a file (load, from_file) or a parsed tree
(from_tree, for tests); a missing file is an error naming the example
- The from_env constructors on LiteLLMClient, AnthropicPassthroughClient,
GiteaClient, MatrixConfig and RoleMap are removed, with the env
reading for max output tokens; clients take their settings from config
- The live integration tests share a committed agent config,
tests/integration/nisse-test.yaml, aimed at test resources, and one
conftest that loads it and builds clients. NISSE_LIVE_PLANNER_MODEL
becomes its planner role; NISSE_CAPABILITY_MODELS stays as a per-run
switch
- A missing secret now fails the live tests instead of skipping them
- CI passes only the three tokens
- .env.example is secrets only; spec A5 says config comes from the file
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Route selection sniffed model names: anything starting "claude-" or
"anthropic/" took the vendor-native passthrough, everything else took the
normalized route. That works only while an alias names its vendor. A
semantic role alias like "remote-architecture" is meant to point at
Anthropic today and somewhere else tomorrow, and the name says nothing
either way -- so it would have taken the normalized route regardless of
what was behind it, silently losing the fidelity A13 wants for delegations.
LiteLLM already knows the answer. /model/info reports each alias's backing
model, provider, and declared tool support, so the agent asks at startup and
routes on the provider it discovers. That keeps vendor knowledge where A5
already put it -- in the proxy -- rather than duplicating it as name patterns
here.
There is deliberately no name-based fallback: PASSTHROUGH_PREFIXES and
has_native_adapter are gone, and select_client now requires a catalog. A
second heuristic could only ever disagree with the proxy. An unknown model
raises instead of guessing a route.
Misconfiguration is therefore loud and early. build_agent validates every
role against the catalog before it clones anything, naming all the bad ones
at once. Discovery is per key -- /model/info returns only what the presented
token may use -- so the same check catches a role pointed at a real model
this key cannot reach. An unreachable proxy is reported separately, since an
outage and a typo call for different responses from the operator.
Two things now get recorded rather than inferred: the resolved model is
pinned alongside the role mapping at task start, for the reason A11 already
pins roles, and DelegationStarted carries backing_model so A16 eval history
is not averaged across whatever an alias happened to point at.
A model LiteLLM declares tool-less warns at startup rather than refusing;
A15 expects weak local models to be accommodated, and they are observed
driving tool loops anyway.
Verified live: all four aliases the production key can see route correctly
through discovery, and each chosen route works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The arrival of non-Anthropic models made a latent bug observable: the
delegate's client was hardcoded to the Anthropic passthrough, so
pointing NISSE_ROLE_DELEGATE at gpt-6-astra or a local model would
have sent Anthropic-shaped /v1/messages bodies to something that does
not speak them. A5 promises the frontier tier is swappable "as a
configuration change, not a refactor", and it was not.
select_client now picks the route from the model: the vendor-native
adapter where one exists, LiteLLM's normalization otherwise — which is
exactly what A13 prescribes for a vendor whose renderer adapter has
not been written, pending validation against the A16 eval set.
Also adds the capability matrix Part H asks for. It says the
thresholds that depend on local capability — A12's failure count,
A14's consult budget, A15's manifest size — must be tuned "on the real
hardware", and that needs measurement rather than reputation. The
harness probes each model for the things that actually break: whether
it calls a tool at all, whether the plan it submits has substance
(llama3.2 emitted "{"), whether it can consume its own tool result,
and whether prompt-level JSON holds. Opt in with
NISSE_CAPABILITY_MODELS; it spends real tokens across several models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing had ever assembled the agent. main.py was still the hello-world
placeholder from the scaffold, and no code outside the test suite
constructed an Orchestrator — which is how two fully built, fully
tested components came to be reachable from nothing: the delegation
harness, and the Gitea comment poller. Unit tests that construct a
component directly cannot catch that; only assembly can.
- config.py validates every variable before anything is built and
names all of them at once, rather than failing at first use twenty
minutes into a milestone on the call that mattered. Whitespace-only
counts as missing, since a token of spaces passes a truthiness check
and fails at the first request instead
- agent.py builds the whole graph and owns startup and shutdown order:
reconcile before accepting commands, and on stop, stop watching,
then stop talking, then close the store last
- The Gitea account is read via whoami rather than configured: it names
the identity every permission is evaluated against, and proves the
token works before anything depends on it
- Poller lifecycle is now a state-machine concern: watch the plan issue
while a gate stands, the PR while it is open, and stop on leaving.
The orchestrator owns when to watch; an injected factory owns how to
build one
- The executor reports each turn, because it runs synchronously inside
the transport callback: without it the agent is silent for a whole
milestone and a working run looks exactly like a hung one
- main.py loads .env, builds, and shuts down cleanly on SIGINT/SIGTERM
The entry-point tests changed shape: nisse-dev no longer greets, it
refuses to start unconfigured and says everything that is missing. Both
tests now run with a deliberately empty environment and a .env path
that does not exist, because the working directory's .env holds live
credentials and a test must never be one cwd away from starting a real
agent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes step 7.
- expert.py: one stateless call for a question needing no repo access.
Structured output is contracted at the prompt level, since vendors
differ on JSON enforcement, so the reply is parsed and validated
locally with one retry — tolerating a fenced code block, because
models wrap JSON habitually, but not missing fields
- The per-task budget is counted from the canonical log rather than
memory, so it survives a restart and cannot hand a circling executor
a fresh allowance. Exhaustion raises rather than answering: a caller
out of consults should get on with the work or escalate
- Consults are recorded as delegation_started/completed with role
consultant. A16 asks for a "consult/delegation eval set", and
reusing the composite events gets consult evaluation for free rather
than inventing a parallel record
- evals.py pairs starts with completions into eval records, with
verdicts kept outside the canonical log: the log records what
happened, a verdict is a later judgement about it, and judgements
get revised. Schema v3 adds the outcomes table, so a v1 store now
traverses two migrations — a better exercise of the hook than one
- ask_expert is offered to the executor as a tool only when a desk is
configured, in addition to the observable escalation triggers and
never instead of them (A12). Consulting is not mutating, so a
completion claim backed only by consults is still challenged
Found live: passing an empty message list and putting the question in
the system prompt fails outright — the Messages API requires at least
one non-system message. The question now travels as a synthesised user
message that is never appended to the log, which keeps the consult
stateless while remaining a valid request.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A frontier model receives a scoped sub-task, works autonomously with
full tool access, and hands back; the local loop reads the handback
note and the file state, never the reasoning.
- Branch at delegation start, and abort discards it. That required
committing each turn's work onto the branch: without commits,
deleting a branch discards nothing, so "abort discards the work"
was a comforting phrase rather than a fact. The caller's own
in-flight changes are committed to their branch first, so an abort
cannot destroy work nobody delegated
- Hard caps on turns and wall clock; a breach is a defined failure
return, and its result is not usable, so the caller gates rather
than resuming on a half-edited tree
- Every event is checkpointed as it arrives, so a crash mid-delegation
leaves the branch at a known commit and a truthful transcript
- Handback is validated locally: blank fields or a non-boolean success
flag are refused and the delegation continues rather than accepting
a useless note
Two failure modes found by running it against a real model:
- A success claim with nothing changed was accepted. The handback plus
file state is the entire interface back, so a note describing work
that does not exist is worse than no note. Success now requires a
successful mutating tool call; reporting failure still does not,
since "this is impossible" is a legitimate finding
- The delegate invented the absolute path /workspace, wrote there, ran
its tests there, and handed back truthfully — with the repo empty
and git seeing nothing. Writes outside the workspace are now refused
with a message naming the real root, and both prompts state where
the workspace is instead of leaving the model to guess. This is not
sandboxing: work outside the repo is invisible to git, so the
milestone would publish an empty PR
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The second renderer adapter, and the concrete case for A13's decision
to normalise in the renderer rather than on the wire. The two vendors
disagree structurally, not cosmetically: the system prompt is a
top-level field rather than a message, a tool call is a tool_use
content block rather than an array, a tool result is a block inside a
user message rather than a "tool" role, and tool schemas use
input_schema rather than parameters. A middlebox flattening those
would be lossy exactly where a long-running delegation lives.
- render_anthropic.py builds vendor-native /v1/messages bodies from
the canonical log, coalescing adjacent same-role blocks so a
tool_result lands directly after its tool_use
- Tool manifests are declared once in OpenAI shape and converted per
vendor, so adding a tool does not mean editing two places
- AnthropicPassthroughClient implements the same LLMClient protocol,
so callers pick a route rather than a different kind of client, and
maps provider tool ids to canonical ones at the boundary
- Dangling tool calls get a placeholder result here too; this API
rejects them more strictly than the OpenAI shape
- Vendor-opaque reasoning blocks are dropped for now with the reason
recorded: A10 wants them stored and never replayed across sessions,
and the canonical schema has no event for that yet
Live tests prove the body is accepted as sent, that a tool_result
turn round-trips, and — as A5 asks — that prompt cache accounting
survives the passthrough rather than being flattened by the proxy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- approvals.py: PlanBoard opens one issue per project and posts each
plan revision as a comment carrying its rev tag. The state store
keeps only a pointer, so gate state can be re-derived from Gitea
after a state-volume loss — external truth wins
- Gitea comments are a second approval source: revision binding, the
typed word approve, and first-approval-wins work exactly as over
transport, with a per-channel allowlist because a Gitea login is
not a Matrix MXID
- Outcomes are echoed to the other channel, so the two records cannot
silently disagree about what was approved
- A Gitea outage degrades durability of the record, not the ability
to approve: planning proceeds and Matrix remains a complete channel
- Live test now asserts the server refuses the agent's own merge and
its push to the protected branch. That property was previously
true only because our code did not attempt it; branch protection
makes it structural, and this test fails if the protection is ever
relaxed
- Fixed a race in the process-group test: killing a group is
asynchronous and the grandchild is an orphan, so its exit is not
synchronised with our return; the test now waits for death
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The standalone half of the Gitea layer; orchestrator wiring follows
once the executor loop lands, to avoid colliding with it in
control.py.
- gitea.py: GiteaClient over httpx — open PRs, list and post issue
comments, close PRs. find_pull_request is the A3 external-truth
query that makes opening a PR idempotent across a crash, and it
checks closed PRs too, since a crash after opening must not be
answered with "nothing is open, so open another"
- pull_request_reconciler: an unreachable Gitea returns None rather
than a guess, so the agent gates to the human instead of risking a
duplicate side effect
- git.py: branch, commit, push for the workspace. The token is passed
per-invocation rather than stored as a remote, so it never lands in
.git/config, and anything raised is scrubbed of it
- commit_all returns None when there is nothing to commit: an empty
milestone is a fact to report, not an error
- Live test does the real round trip against the sandbox repo and
cleans up after itself
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two invariants only a live homeserver can check, now enforced rather
than verified by hand:
- The configured NISSE_MATRIX_USER_ID must be the token's real owner.
MatrixTransport filters its own messages by comparing sender to that
value, so a mismatch would make the bot read its own posts as
operator input.
- The account must be a testing identity, so a stray live run can
never act as the production agent.
Both hold today; this keeps them holding when the production
nisse-dev account is created.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found while chasing a flaky live test, but the flakiness was a real
bug: only the ask_human path recorded a pending item, so AWAIT_INPUT
reached any other way — planning failure, stopped milestone, boot
reconciliation — refused every operator reply with "no pending
question to answer" and the agent could never be recovered. The
operator had no way back short of wiping state.
Every path into AWAIT_INPUT now records what a reply should resume,
and answering dispatches on it: planning failures and questions
resume planning, a stopped milestone returns to its still-approved
gate, and the fallback says what commands work instead of
dead-ending. Two regression tests cover the recovery paths.
The live test also scripted a fixed approve/approve/start sequence,
which a real model breaks by legitimately calling ask_human at a
gate. It now plays the operator adaptively, responding to whatever
the agent actually did. Five consecutive live runs pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- tools.py: four flat tools (read_file, write_file, list_dir,
run_command) per A15's reduced manifest. Deliberately unsandboxed:
the VM is the trust boundary (B3) and tool use is auto-approved
inside an approved unit of work (A4), so path confinement would add
failure modes without adding safety. What is enforced is
operational — timeout with process-group kill so a backgrounded
server cannot outlive its command, output truncation so a build log
cannot eat the context window, closed stdin so prompts fail fast,
and a scrubbed environment so project code never sees the agent's
credentials
- executor.py: renders the log, calls the executor role with the
reduced manifest, dispatches tool calls, and checkpoints every
result (A13). Escalation is observable, never self-assessed (A12):
consecutive command failures and a hard turn cap. Failed reads do
not count — looking around is not failing
- Until delegation exists (step 7), escalation gates to the human
with the failures attached
- control.py: start milestone now runs the executor; a stopped
milestone gates to AWAIT_INPUT
Two bugs found by running it live against real models:
- A turn-ending gate tool (submit_plan) left a tool_call with no
tool_result, which providers reject outright, so every planning
turn after the first failed with a 400. The planner now records the
gate as the call's outcome, which is also the truthful transcript
- The renderer now emits a placeholder for any tool_call lacking a
result, so a crash between call and result (A3) cannot poison the
transcript permanently
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- planner.py: LLMPlanningAgent offers exactly two tools, submit_plan
and ask_human, so gating stays structural at the tool layer; no
proceed tool exists. System prompt requires explicit, low-inference
plans because a weaker model executes them (A13)
- Model output is appended to the canonical log as it arrives, before
being interpreted, so a crash mid-planning leaves a truthful
transcript
- A bad turn (prose instead of a tool call, malformed arguments, or a
failed call) retries once, then raises PlanningError; the
orchestrator gates to AWAIT_INPUT rather than inventing a plan
- Orchestrator: appends every operator message, plan_submitted, and
gate_opened to the transcript, so re-planning sees rejection
feedback as conversation rather than out-of-band parameters; pins
the A11 role mapping at task start
- llm.py: non-JSON or non-object tool arguments now raise LLMError
instead of escaping as JSONDecodeError — observed from llama3.2,
which emits truncated argument strings
- Live planning test is opt-in on NISSE_LIVE_PLANNER_MODEL: planning
escalates unconditionally (A12) and the local model does not hold
the tool-call contract, so it awaits a frontier route
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Skipping now keys solely on NISSE_LLM_BASE_URL and
NISSE_MATRIX_HOMESERVER: if the sentinel is set, the live path is
expected to work and any other missing variable is an assertion
failure naming it, not a silent skip. A misnamed or misplaced CI
secret previously produced a green run with no live coverage.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- matrix.py: Transport implementation over the Matrix client-server
HTTP API directly (token auth, /sync long-poll, /send) using the
existing httpx dependency — no Matrix SDK; four endpoints total.
Deviation from the presented plan (matrix-nio) flagged in the PR
- First sync learns position and skips backlog, so room history is
never delivered as context (A1); own messages and non-message
events filtered; sender MXID passed through for the orchestrator's
operator allowlist
- Sync loop backs off (capped at 60s) and retries indefinitely on
homeserver errors, mirroring B2's polling posture
- Encrypted rooms unsupported: bot rooms are created unencrypted
until E2E lands (operator decision; room recreated accordingly)
- NISSE_MATRIX_TEST_ROOM_ID added: the live smoke test binds only
to the dedicated test room, never the production room
- 8 unit tests over a mocked HTTP layer; live integration test
verified against the real homeserver
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR 2 of the steps-3/4 plan.
- llm.py: LLMClient protocol — complete(events, tools, model) ->
canonical events — present from the first real LLM call.
LiteLLMClient renders requests via the A10 renderer, targets the
proxy's OpenAI-compatible route, and maps provider tool-call IDs
to locally-minted canonical IDs at the boundary
- RoleMap (A11): executor/planner/delegate/consultant resolved from
env; pinned() returns the mapping to record at task start
- Config strictly from env: NISSE_LLM_BASE_URL, NISSE_LLM_TOKEN;
envfile.py is a minimal .env loader where real env wins; .env now
gitignored with a committed .env.example documenting all variables
(Matrix ones included for the upcoming transport PR)
- Live smoke test against the proxy is integration-marked and skips
without credentials — CI stays credential-free per operator
decision; the test runs locally once .env exists
- httpx added as nisse-dev's HTTP client
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build-order step 4, sequenced before the LLM client (step 3) because
the client interface consumes the canonical log.
- canonical.py: typed events for the full A10 table (user_message
through compaction); canonical tool-call IDs minted locally,
provider IDs never a join key; strict payload validation on load
- StateStore: canonical_events table via schema v2 — the first real
migration, exercised in a test against a hand-built v1 database —
with append_event/read_events on the same fsync-per-commit path
- render.py: request bodies rendered from the log at call time;
first adapter targets the OpenAI-compatible shape of LiteLLM's
normalized route; compaction events replace their id range at
render time while staying in the durable log
- Crash integration test now also verifies transcript appends
survive process death
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step-2 PR A: the approval loop end to end with a stub agent, no LLM
and no network.
- transport.py: Transport interface per A1 (send_message,
send_approval_request, ask_question, on_message) with an
InMemoryTransport; no platform SDK outside an implementation
- gates.py: PlanRevision with monotonic rev-N tags; Gate.evaluate
enforces revision binding (stale tags refused with a notice),
operator allowlisting, and approve-word-only semantics; any other
reply is rejection feedback. Channel seam ready for Gitea/Vikunja
approval sources
- control.py: B1 state machine subset with structural gating — the
agent's only actions are SubmitPlan (ends its turn) and AskHuman;
no proceed tool exists. Gate state, revision counters, task_id,
and pending questions checkpoint through StateStore, and boot
resumes an open gate or gates to AWAIT_INPUT, never blind-resumes
- gate.open/approve/reject A9 events complete the log taxonomy
- CLAUDE.md: record the plan-first working convention
- Tests: 15 unit (gates + orchestrator flows), 2 integration
(mid-gate restart with revisions intact; unresolved intents gate
the whole boot)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Retrofit for the conventions merged in #4 (issue #3):
- OpLog: JSON-lines emitter with the fixed envelope (ts, agent,
event, task_id, cause); task_id set by the orchestrator at
unit-of-work start
- StateStore owns emission: transition, intent begin/commit/abort,
and schema migrations each emit their event right after the
durable write commits; callers pass the cause
- reconcile_on_boot emits boot.reconcile with action and verdicts,
and resolves intents with reconciliation-verdict causes
- Failed resolutions emit nothing (a line exists iff the write
committed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build-order step 1: the critical path everything else sits on.
- StateStore: SQLite in WAL mode with synchronous=FULL, so every
state transition and intent-log write is committed and fsynced
before returning; a power cut cannot tear state
- Durable intent log: begin_intent records a unit of work before it
starts; commit/abort resolve it; pending intents survive restart
- schema_version stamped from the first commit, with a sequential
migration hook; a newer state file than the code supports is
refused rather than misread
- reconcile_on_boot (A3 first cut): pending intents are checked
against external truth by registered reconcilers (observable
checks, never model self-assessment); verified effects are
committed, verified no-ops aborted, anything unverifiable gates
to AWAIT_INPUT — never blind-resume
- Lives in nisse_dev per spec 0.1: the library is extracted after
the dev agent works, not designed up front
- Integration test kills a subprocess mid-work with os._exit and
verifies state integrity and intent visibility on reopen
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dongarandClaude Opus 5.5
dev-agent