nisse-dev now runs only the conversational loop. The gated state machine
it was built alongside is gone:
- Removed control.py, planner.py, executor.py, approvals.py, gates.py
(command_form moves to loop/reader.py), publish.py and poller.py, the
old assembly in agent.py (its shared helpers move to loop/build.py),
the store's state-machine view (get_state, transition), their tests,
the old end-to-end and live planning/milestone/gate tests, and
specs/orchestrator.qnt with its scenarios.
- Roles are driver, helper, expert and classifier; driver is required.
loop:, roles.planner, roles.executor and tuning.executor are refused
with what to use instead; tuning.driver takes the turn cap and failure
limit.
- The benchmark and the capability matrix measure the driver: the
loop's own system prompt and tools, the task's plan as the agreed
restatement. Task plans end with finish_milestone, so earlier runs
show as (old); stored runs' executor tuning reads as the driver's.
- A delegation on the loop records what its helper's alias resolved to
(L9), as the old path's did.
- The dump shows a work item's revision and what it waits for.
- Behaviour the old tests covered and the loop shares stays covered:
driver runs that fail on a model call, stop mid-turn, reset the
failure count; go-aheads ignoring case and punctuation; a request too
big to fit is never sent; the assembly tests, now on build_loop_agent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
A 502, 503 or 504 from the LiteLLM proxy, or a dropped connection, is
retried after 5, 15 and 45 seconds before the call fails. Twice on
2026-10-07 a short burst of 502s stopped work outright: the integration
tier at 16:39 and an e2e conversation at 20:43, where the item waited on
the operator after one refused call.
Not retried: a 500 (it may be the request's fault), a timeout (it already
had its request_timeout_s), or a budget refusal. Rate limits keep their
single 20-second retry. The time spent waiting is reported with the rate
limit's (last_rate_limit_wait_s), so it isn't counted as the model's.
Tests: an outage waited out, one that outlasts the backoff, a dropped
connection, a 500 and a timeout not retried, a budget refusal not waited
on. The offline agent in test_agent_build skips the backoff (its proxy
doesn't exist).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
- roles.driver (unset: the planner's model) and roles.classifier (unset:
the expert's); the mapping is pinned when an item opens, and the driver
and the go-ahead reader run on the item's own (L9).
- profiles: in configuration (tools, strengths, weaknesses), used in
briefings: the driver's instructions say what its helper and expert are
good at, the helper's brief who reads its handback; never a model name.
- delegate(task) and ask_expert(question) for the driver. A helper works
on a branch off the milestone's; on a handback its work comes back as
uncommitted changes, otherwise it is discarded, so harness branches
never hold commits only the clone has. ask_expert keeps its budget.
- Triggers: failure_limit failed commands in a row (Driver's new "stuck"
outcome) or the turn cap, mid-milestone, call a helper in once per
milestone; if it fails too, the item waits on the operator. A
delegation cut off by a crash is discarded at boot.
- Evaluation capture: delegations and ask_expert as before, and the
reader's verdicts as classifier pairs.
Tests: help tests (merge-back, discard, needs a milestone, the stuck
trigger and its once-per-milestone limit, the expert and its budget,
pinning, briefings by profile, boot discarding a cut-off delegation),
config tests for roles and profiles, and the walk delegating with a
scripted helper that sometimes fails, with a delegations_settle property.
The walk found helper commits stranded on a merged PR's branch, fixed by
returning a helper's work uncommitted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Architecture L9's role names, for the two roles with a direct
counterpart: the delegation role is now 'helper' and the ask_expert role
'expert', in RoleMap, configuration (roles.helper, roles.expert),
recorded events and tests. 'planner' and 'executor' stay distinct for
now; they become 'driver' when the new loop merges them (Development
Plan M2.3).
Configurations must rename their keys: nisse-dev.example.yaml and
tests/integration/nisse-test.yaml are updated here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M0.1. Every section reference in code, tests, specs/,
the example config, the tuning guide, and the CLAUDE.md and README files
now cites docs/Architecture.md (P, L, D, O, X, Q) instead of the old
spec's A-H, and docs/Architecture-Build-Spec.md is deleted (git keeps
it). The plan's old-to-new table goes with it.
Mostly mechanical, with a per-site choice where one old section split:
A4 became L2 where it meant stop/abandon/the loop and L3 where it meant
approvals; A10 became L6 in the context-budget code and L7 elsewhere; B2
became D4 for outbound Gitea and publishing and D5 for review intake;
'A3 step N' became the matching step of L8's boot sequence. Comments
describing behaviour the architecture drops (planning always escalating,
thresholds tuned at an old build step) now describe the current code and
point at the Development Plan.
docs/Architecture.md gains one line the rewrite missed: intake outages
delay but never fail work (old B2), now in L5, where the poller and
Matrix code cite it.
Comments and docs only; no behaviour change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Two setup failures found while calibrating the new benchmark tasks,
both of which would also fail real milestones.
Devstral's chat template (Mistral) rejected any turn in which the model
wrote some text before a tool call: the OpenAI renderer made one message
per event, so the turn became a text message and a tool-call message in
a row, and llama.cpp answered 500. The renderer now folds consecutive
user or assistant messages into one, which is the OpenAI format for a
turn anyway. Checked against the live server: the captured failing
request no longer occurs, over six real runs.
LiteLLMClient gave up on any request after a fixed 120 s. A dense 27B
model writing a whole module in one turn takes longer, so the executor
escalated work that was going well. llm.request_timeout_s now sets it
for both clients, default 600.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently.
A benchmark run that waited looked slower than the model is: on
2026-09-27, fast models under the unit-test key's 200k tokens/minute
limit had 20-45 s runs that were probably waits.
Both clients now report last_rate_limit_wait_s after every call, even
one that then fails. The benchmark sums it per run into
rate_limit_wait_s, the report's time column leaves it out, and a new
429 column counts the runs that waited. Results stored before this
load unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.
- Four canned milestones under nisse_dev/bench/tasks: write a function,
fix a bug a failing test exposes, thread an option through several
files, and find one failure in output far past max_output_chars. Each
is an approved plan over a seed project, judged by a hidden check the
model never sees; a reference solution proves each is solvable and
that its check tells the difference
- Runs use the agent's own client wiring, now shared as
make_llm_clients, with delegation and ask_expert off: an escalation is
a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
the model's backing name and declared facts, a note on how it was
served, the task's content hash, the agent's commit and tuning, the
outcome, turns, misbehaviour counts, context trims, tokens, time and
LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
a new model through the context probe, the capability matrix and the
benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
requests it had to trim
- The tuning guide gains "Benchmarking models"
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.
- The catalog reads supports_tool_choice from LiteLLM's model_info;
BudgetedClient sends tool_choice only to models declared true, as
"required" on the normalized route and {"type": "any"} on the
Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
milestone. Finishing with nothing changed is refused once; a prose
reply gets one nudge, then escalates; a call to a tool not offered
counts toward failure_limit, like a failed command; calls after
finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
recognises a proxy fallback, which once credited Claude's answers to
a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
that models call only offered tools, and that a declared
supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
declared and backed up
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every turn re-sent the whole transcript with nothing checking it against
the model's window. On a local model that fails silently: Ollama drops
the start of an over-long prompt, which is where the system prompt and
the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with
no error.
- The model catalog reads max_input_tokens and max_output_tokens from
LiteLLM's model_info, and startup refuses a role on a model with no
declared window
- BudgetedClient wraps both routes. It fits the canonical events before
rendering: older tool results cut to head and tail, then the oldest
turns dropped with a note, keeping calls with their results; a request
that still cannot fit fails with ContextOverflow and is never sent.
The output ceiling is the smaller of config and the model's limit
- Tokens are estimated from characters, pessimistically, and the
estimate is corrected from reported prompt sizes, only ever upward
- The executor takes its scope from the log: the latest plan, which is
the approved milestone plan, goes in its system prompt, and it sees
only events after it
- Tool output keeps its head and, mostly, its tail, so a test runner's
failure summary survives truncation
- tuning.context.keep_recent_turns and safety_margin are new knobs
- scripts/probe_context.py checks a declared window against what the
server does; it confirmed llama3.2 cuts at the declared 4096
- The tuning guide covers the budget, declaring a local model's limits,
and the probe
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With no production instance yet, there is no reason to carry two ways
of configuring the agent. Every environment variable that duplicated a
config key is gone; the environment now supplies only the secrets the
file references by ${…}, plus NISSE_CONFIG and NISSE_ENV to find the
files.
- AgentConfig loads from a file (load, from_file) or a parsed tree
(from_tree, for tests); a missing file is an error naming the example
- The from_env constructors on LiteLLMClient, AnthropicPassthroughClient,
GiteaClient, MatrixConfig and RoleMap are removed, with the env
reading for max output tokens; clients take their settings from config
- The live integration tests share a committed agent config,
tests/integration/nisse-test.yaml, aimed at test resources, and one
conftest that loads it and builds clients. NISSE_LIVE_PLANNER_MODEL
becomes its planner role; NISSE_CAPABILITY_MODELS stays as a per-run
switch
- A missing secret now fails the live tests instead of skipping them
- CI passes only the three tokens
- .env.example is secrets only; spec A5 says config comes from the file
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Route selection sniffed model names: anything starting "claude-" or
"anthropic/" took the vendor-native passthrough, everything else took the
normalized route. That works only while an alias names its vendor. A
semantic role alias like "remote-architecture" is meant to point at
Anthropic today and somewhere else tomorrow, and the name says nothing
either way -- so it would have taken the normalized route regardless of
what was behind it, silently losing the fidelity A13 wants for delegations.
LiteLLM already knows the answer. /model/info reports each alias's backing
model, provider, and declared tool support, so the agent asks at startup and
routes on the provider it discovers. That keeps vendor knowledge where A5
already put it -- in the proxy -- rather than duplicating it as name patterns
here.
There is deliberately no name-based fallback: PASSTHROUGH_PREFIXES and
has_native_adapter are gone, and select_client now requires a catalog. A
second heuristic could only ever disagree with the proxy. An unknown model
raises instead of guessing a route.
Misconfiguration is therefore loud and early. build_agent validates every
role against the catalog before it clones anything, naming all the bad ones
at once. Discovery is per key -- /model/info returns only what the presented
token may use -- so the same check catches a role pointed at a real model
this key cannot reach. An unreachable proxy is reported separately, since an
outage and a typo call for different responses from the operator.
Two things now get recorded rather than inferred: the resolved model is
pinned alongside the role mapping at task start, for the reason A11 already
pins roles, and DelegationStarted carries backing_model so A16 eval history
is not averaged across whatever an alias happened to point at.
A model LiteLLM declares tool-less warns at startup rather than refusing;
A15 expects weak local models to be accommodated, and they are observed
driving tool loops anyway.
Verified live: all four aliases the production key can see route correctly
through discovery, and each chosen route works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An exhausted budget was a special case of something more general: the
work is not at fault. A network blip, a timeout, a 5xx from the proxy,
or a rate limit that did not clear all leave a delegation branch that
may be perfectly good, and discarding it would throw away work for a
reason that has nothing to do with the work.
UpstreamUnavailable now names that family, with BudgetExhausted a
member of it — distinguishable where the difference matters, since a
budget will not clear within the hour while a network blip will. A
delegation stopped by any of them is "interrupted": branch kept,
intent left pending, and A3's resume rule picks it up.
The benefit of the doubt stops at the server's own door. A 4xx that is
not a rate limit means the request itself was wrong, and preserving
branches on the strength of one would hide a real bug — so those still
fail and clean up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Exceeding a LiteLLM spend limit takes external models away for the
rest of the day, which makes three existing gaps consequential rather
than untidy.
- A13 calls the delegation caps hard on "turns, tokens, and wall
clock", and the module docstring repeated it, but the token cap was
decoration: nothing counted, and result.tokens was always zero. It
is now metered from the usage the provider reports, so a confused
delegate cannot spend a day's allowance. A5 keeps the *money* cap on
the LiteLLM key, which is a different thing; this is a scope cap
- The normalized route sent no ceiling at all, so truncation detection
there had nothing to detect against. Both routes now send one
- A budget refusal is recognised distinctly. It is not a rate limit:
retrying cannot help, so the planner no longer burns a second call
to fail identically
And a delegation cut off by an exhausted budget is now "interrupted"
rather than "failed", which preserves its branch and leaves its intent
pending. The work on that branch may be perfectly good; discarding it
because the budget ran out would throw away work for a reason that has
nothing to do with the work. A3's resume rule already knows how to
pick such a delegation back up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measuring Opus found a bug that had been silent all along: nothing
checked stop_reason or finish_reason, so a response cut off mid-answer
was parsed as a complete one.
The symptom was a submit_plan tool call whose required `content` field
was simply absent — structurally valid, substantively empty, and
indistinguishable from a model that chose to send nothing. The cause
was our max_tokens of 4096: Opus writing an architecture plan
routinely spends 3,000-4,000 output tokens, and was observed at 4,064
against that ceiling. Roughly one run in three crossed it.
Worse than the empty plan is what the same failure does elsewhere: a
truncated write_file writes half a file and reports success.
- Both routes now raise TruncatedResponse when the model was cut off.
It subclasses LLMError, so the planner's retry and the executor's
and delegation's gating already handle it without learning a new
type: a cut-off turn is a failed turn
- The default ceiling is 16384 and configurable via
NISSE_MAX_OUTPUT_TOKENS. Output is billed by what is produced, not
by the ceiling, so a low limit buys nothing and costs silent
truncation
Verified: three consecutive Opus planning turns now return plans of
5,700-6,200 characters where one in three previously returned nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The arrival of non-Anthropic models made a latent bug observable: the
delegate's client was hardcoded to the Anthropic passthrough, so
pointing NISSE_ROLE_DELEGATE at gpt-6-astra or a local model would
have sent Anthropic-shaped /v1/messages bodies to something that does
not speak them. A5 promises the frontier tier is swappable "as a
configuration change, not a refactor", and it was not.
select_client now picks the route from the model: the vendor-native
adapter where one exists, LiteLLM's normalization otherwise — which is
exactly what A13 prescribes for a vendor whose renderer adapter has
not been written, pending validation against the A16 eval set.
Also adds the capability matrix Part H asks for. It says the
thresholds that depend on local capability — A12's failure count,
A14's consult budget, A15's manifest size — must be tuned "on the real
hardware", and that needs measurement rather than reputation. The
harness probes each model for the things that actually break: whether
it calls a tool at all, whether the plan it submits has substance
(llama3.2 emitted "{"), whether it can consume its own tool result,
and whether prompt-level JSON holds. Opt in with
NISSE_CAPABILITY_MODELS; it spends real tokens across several models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI failed on the delegation branch with a 429 from the proxy: token
limit 200000, 17399 remaining. Not a defect — the live delegation
tests are token-hungry, and every push to a PR branch fired two
identical suites concurrently, since the push and pull_request events
both matched.
- Push now only triggers on main. A PR branch is already covered by
the pull_request event, so this halves both the wall time and the
tokens spent per branch
- A 429 is retried once before being surfaced. Losing a ten-turn
delegation to a transient limit wastes far more than waiting out
the window, and retrying forever would hide a real ceiling. The
sleep is injectable, so the existing 429 test no longer waits 20
seconds for real
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The second renderer adapter, and the concrete case for A13's decision
to normalise in the renderer rather than on the wire. The two vendors
disagree structurally, not cosmetically: the system prompt is a
top-level field rather than a message, a tool call is a tool_use
content block rather than an array, a tool result is a block inside a
user message rather than a "tool" role, and tool schemas use
input_schema rather than parameters. A middlebox flattening those
would be lossy exactly where a long-running delegation lives.
- render_anthropic.py builds vendor-native /v1/messages bodies from
the canonical log, coalescing adjacent same-role blocks so a
tool_result lands directly after its tool_use
- Tool manifests are declared once in OpenAI shape and converted per
vendor, so adding a tool does not mean editing two places
- AnthropicPassthroughClient implements the same LLMClient protocol,
so callers pick a route rather than a different kind of client, and
maps provider tool ids to canonical ones at the boundary
- Dangling tool calls get a placeholder result here too; this API
rejects them more strictly than the OpenAI shape
- Vendor-opaque reasoning blocks are dropped for now with the reason
recorded: A10 wants them stored and never replayed across sessions,
and the canonical schema has no event for that yet
Live tests prove the body is accepted as sent, that a tool_result
turn round-trips, and — as A5 asks — that prompt cache accounting
survives the passthrough rather than being flattened by the proxy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- planner.py: LLMPlanningAgent offers exactly two tools, submit_plan
and ask_human, so gating stays structural at the tool layer; no
proceed tool exists. System prompt requires explicit, low-inference
plans because a weaker model executes them (A13)
- Model output is appended to the canonical log as it arrives, before
being interpreted, so a crash mid-planning leaves a truthful
transcript
- A bad turn (prose instead of a tool call, malformed arguments, or a
failed call) retries once, then raises PlanningError; the
orchestrator gates to AWAIT_INPUT rather than inventing a plan
- Orchestrator: appends every operator message, plan_submitted, and
gate_opened to the transcript, so re-planning sees rejection
feedback as conversation rather than out-of-band parameters; pins
the A11 role mapping at task start
- llm.py: non-JSON or non-object tool arguments now raise LLMError
instead of escaping as JSONDecodeError — observed from llama3.2,
which emits truncated argument strings
- Live planning test is opt-in on NISSE_LIVE_PLANNER_MODEL: planning
escalates unconditionally (A12) and the local model does not hold
the tool-call contract, so it awaits a frontier route
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PR 2 of the steps-3/4 plan.
- llm.py: LLMClient protocol — complete(events, tools, model) ->
canonical events — present from the first real LLM call.
LiteLLMClient renders requests via the A10 renderer, targets the
proxy's OpenAI-compatible route, and maps provider tool-call IDs
to locally-minted canonical IDs at the boundary
- RoleMap (A11): executor/planner/delegate/consultant resolved from
env; pinned() returns the mapping to record at task start
- Config strictly from env: NISSE_LLM_BASE_URL, NISSE_LLM_TOKEN;
envfile.py is a minimal .env loader where real env wins; .env now
gitignored with a committed .env.example documenting all variables
(Matrix ones included for the upcoming transport PR)
- Live smoke test against the proxy is integration-marked and skips
without credentials — CI stays credential-free per operator
decision; the test runs locally once .env exists
- httpx added as nisse-dev's HTTP client
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dongarandClaude Opus 5.5