nisse-core records each model call of every agent into a file beside its
state store (<state>.calls.db), for the status page (L7, X6):
- the request body exactly as sent (never headers), the raw response with
its reasoning, the model asked and the one that answered, time, tokens
and llama-server's own prompt reuse (timings.cache_n);
- its work item, role, run and purpose: Driver.run and nisse-dev's
delegation loop open runs, nested; compaction, expert consults, the
classifier and the go-ahead reader label their calls;
- provenance, worked out in BudgetedClient by comparing the events at
each stage: each event sent with its log id and content as sent, which
a compaction replaced, which fitting shortened (by how much) and
dropped.
Content is stored once by hash, compressed. It is diagnostic only: the
agent never reads it, a failed write is logged once and recording stops
while the call goes on, and the file is deleted when the agent starts,
so its layout never migrates; CallReader refuses a layout it doesn't
know. While recording, the system prompt ends with one sentence saying
so. `call_record: false` in either agent's YAML turns it off.
The benchmark records each run beside that run's own state store, and
`--keep-runs DIR` keeps both for the page.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
Shared parts, not a shared scheduler (agreed): nisse-review will have its
own small scheduler; a common one waits until two exist to compare.
- nisse_core.loop.driver: the driver run, moved; the agent names its
waiting tools per run (nisse-dev's are WAITING_TOOLS), and execution
goes to any Executor (ToolOutcome moves with it).
- nisse_core.loop.tools: the tool helper and the tools any agent offers
(notes, memory, memory_done, ask_expert); nisse_core.loop.expert.
- nisse_core.knowledge.handlers: Knowledge, the vault and memory as
control tools, the index and prompt text, out of LoopAgent.
- nisse_core.config: file reading, references, people, tuning sections of
the agent's own, LLM numbers, the vault.
- New core tests: a driver run with a non-workspace executor and the
agent's own waiting set; Knowledge alone; the shared config pieces.
Docs: the Development Plan marks M7.2 done; Architecture Part L and the
nisse-core README name the new parts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
What isn't the dev agent's moves to nisse-core, unchanged in behaviour,
with its tests: canonical, render, render_anthropic, llm, catalog, probe,
context, views, state, reconcile, worker, cancel, dump, transport, matrix,
execution, testing, vault, memory, oplog, envfile and gitea (whole).
- nisse_core.state.StateStore holds the tables every agent has; an
agent's subclass adds its tables (version, migrations), roles and
views. nisse_dev.state.StateStore adds the build VM blocks (v5).
- views.project takes what each role is shown, rather than a fixed table.
- Vault takes the agent's name (agent=), for provenance.
- A test that nisse_core never imports an agent.
- The loop, config, git, delegation, expert, tools and evals stay in
nisse-dev until nisse-ops has a loop to share.
Docs: the Development Plan marks M7.1 done and notes that Phase 6 waits on
the agent host and Terraform; Architecture Part L names the package.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
nisse-dev now runs only the conversational loop. The gated state machine
it was built alongside is gone:
- Removed control.py, planner.py, executor.py, approvals.py, gates.py
(command_form moves to loop/reader.py), publish.py and poller.py, the
old assembly in agent.py (its shared helpers move to loop/build.py),
the store's state-machine view (get_state, transition), their tests,
the old end-to-end and live planning/milestone/gate tests, and
specs/orchestrator.qnt with its scenarios.
- Roles are driver, helper, expert and classifier; driver is required.
loop:, roles.planner, roles.executor and tuning.executor are refused
with what to use instead; tuning.driver takes the turn cap and failure
limit.
- The benchmark and the capability matrix measure the driver: the
loop's own system prompt and tools, the task's plan as the agreed
restatement. Task plans end with finish_milestone, so earlier runs
show as (old); stored runs' executor tuning reads as the driver's.
- A delegation on the loop records what its helper's alias resolved to
(L9), as the old path's did.
- The dump shows a work item's revision and what it waits for.
- Behaviour the old tests covered and the loop shares stays covered:
driver runs that fail on a model call, stop mid-turn, reset the
failure count; go-aheads ignoring case and punctuation; a request too
big to fit is never sent; the assembly tests, now on build_loop_agent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
execution.SSHHost.execute is now the only way the agent runs anything
(L13). Commands travel as a script on SSH's stdin, so the secrets handed
to a call are never on a command line, and every value handed out is
redacted from the output. Timeouts kill the remote process group, and
an unreachable host is reported as such, not as a failed command.
Workspace and GitRepo run on the build VM. The clone uses the
repository's SSH URL, so no credential is stored in it (#89); fetches
and pushes get the push key over stdin, in a short-lived ssh-agent,
and PRs are still opened from the agent's host with its API token.
The bench runs on the build host too.
Tests reach "the build VM" through a throwaway local sshd (nisse_dev
.testing); CI installs openssh-server. The live push test now pushes
with the sandbox's deploy key (NISSE_PUSH_KEY).
Fixes#89.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.
- Four canned milestones under nisse_dev/bench/tasks: write a function,
fix a bug a failing test exposes, thread an option through several
files, and find one failure in output far past max_output_chars. Each
is an approved plan over a seed project, judged by a hidden check the
model never sees; a reference solution proves each is solvable and
that its check tells the difference
- Runs use the agent's own client wiring, now shared as
make_llm_clients, with delegation and ask_expert off: an escalation is
a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
the model's backing name and declared facts, a note on how it was
served, the task's content hash, the agent's commit and tuning, the
outcome, turns, misbehaviour counts, context trims, tokens, time and
LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
a new model through the context probe, the capability matrix and the
benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
requests it had to trim
- The tuning guide gains "Benchmarking models"
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dongarandClaude Opus 5.5