- Every e2e role runs on qwen3.8-27b on the RTX 4090 (e2e_roles();
NISSE_E2E_MODEL overrides, e.g. claude-haiku-4-5). The first full run
passed 7 of 8 in 26 minutes for $0.0000; the eighth was a real finding,
fixed with the new-project PR.
- scripts/e2e.sh reports a run's spend on the test key and the day's total
against its budget: above $0 on the local model means a paid fallback.
- CI runs the suite as its own job on pull requests (not pushes to main),
on a throwaway sshd (--local), cancelling a run a newer push supersedes,
so the checks job still reports in minutes.
- The plan says so: e2e on every PR; locally only -k / --lf.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Every live test that touches a repository gets one of its own from
e2e_support.fresh_repo, made by the test operator as an operator makes a
project's (D4, D8): seeded from tests/integration/seed/ (the sandbox's
package, without past runs' leftovers), the agent's account a collaborator
with write access, main protected (pushes only by the owner, an approval to
merge), and a deploy key generated for the run. It is deleted afterwards
(NISSE_E2E_KEEP_REPOS keeps it); scripts/e2e-cleanup.sh deletes leftovers.
- Both e2e fixtures use it; per-run base branches are gone. The merge
helper approves first, as the protection now requires.
- test_gitea_live uses it, so CI needs NISSE_OPERATOR_TOKEN and no longer
writes the sandbox deploy key (NISSE_PUSH_KEY).
- A live test of the helper: seeded, shared with the agent, protected
against it, deleted.
- nisse-test.yaml's project entry is a placeholder; pytest and ruff skip
the seed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
The first live run of the new loop's e2e test found that a fact recorded
while start_milestone was in flight landed between the tool call and its
result; both API formats reject that, so every later request in the run
failed. Views now keep each result right after its call. Scripted test
models check the APIs' ordering rules, so this can't pass unit tests
again.
The e2e tests no longer depend on the sandbox's history: each run asks for
a uniquely named function and works against a base branch of its own,
deleted afterwards. The review step asks for a change that can't already
be there. The scripted operator gives up after five fruitless answers, and
each run keeps its state store for inspection. The driver's prompt says
review fixes stay in the current milestone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
From an issue the scripted operator files and assigns in Gitea, through
the restatement, go-ahead, milestone and PR, a review comment answered
with a pushed fix, to an approved and merged PR and a closed item and
issue. The operator acts in Gitea through its own test account
(NISSE_OPERATOR_TOKEN) and in the item's thread through the in-memory
transport. Runs from scripts/e2e.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
tests/integration/test_end_to_end.py drives the whole assembled agent
(real models, the Gitea sandbox, a real build host) through a milestone
with a scripted operator, and checks what bring-up run 3 checked by hand:
a PR holding only the work, pushed with the push credential; no
credential in the clone; the planner's view carrying facts, not the
executor's turns. It cleans up after itself on the sandbox and the build
host.
scripts/e2e.sh runs it on the bring-up build VM (or --local). It has its
own marker, e2e, kept out of check.sh and CI since it spends tokens.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
execution.SSHHost.execute is now the only way the agent runs anything
(L13). Commands travel as a script on SSH's stdin, so the secrets handed
to a call are never on a command line, and every value handed out is
redacted from the output. Timeouts kill the remote process group, and
an unreachable host is reported as such, not as a failed command.
Workspace and GitRepo run on the build VM. The clone uses the
repository's SSH URL, so no credential is stored in it (#89); fetches
and pushes get the push key over stdin, in a short-lived ssh-agent,
and PRs are still opened from the agent's host with its API token.
The bench runs on the build host too.
Tests reach "the build VM" through a throwaway local sshd (nisse_dev
.testing); CI installs openssh-server. The live push test now pushes
with the sandbox's deploy key (NISSE_PUSH_KEY).
Fixes#89.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Development Plan M0.2.
- scripts/bringup.sh: start [REF] sets the previous run aside under
/root/nisse-e2e/runs/, exports the ref under test and starts the agent
as the transient systemd unit nisse-bringup, so it outlives the shell
that started it; stop (SIGTERM, clean shutdown), status, logs, and
collect (the journal log and a readable transcript).
- nisse_dev.dump (python -m nisse_dev.dump STATE_DB [--full]): a readable
dump of a state store's state, pending intents and canonical log. Opens
the store read-only, so it is safe against a live agent.
- The agent runs as root on this development VM, as it will on its own
VM. Architecture D3/D6/D7 corrected to match the operator's intent:
installing packages and toolchains on its own VM is class 2, recorded
so the environment can be rebuilt (it was class 3 with provisioning an
open question); egress names OS package mirrors. The plan's M0.2, M4.1,
M4.2 and bring-up guards follow.
Acceptance: a smoke run started, answered status in the test room,
stopped cleanly, and collect produced the log and transcript.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.
- Four canned milestones under nisse_dev/bench/tasks: write a function,
fix a bug a failing test exposes, thread an option through several
files, and find one failure in output far past max_output_chars. Each
is an approved plan over a seed project, judged by a hidden check the
model never sees; a reference solution proves each is solvable and
that its check tells the difference
- Runs use the agent's own client wiring, now shared as
make_llm_clients, with delegation and ask_expert off: an escalation is
a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
the model's backing name and declared facts, a note on how it was
served, the task's content hash, the agent's commit and tuning, the
outcome, turns, misbehaviour counts, context trims, tokens, time and
LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
a new model through the context probe, the capability matrix and the
benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
requests it had to trim
- The tuning guide gains "Benchmarking models"
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.
- The catalog reads supports_tool_choice from LiteLLM's model_info;
BudgetedClient sends tool_choice only to models declared true, as
"required" on the normalized route and {"type": "any"} on the
Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
milestone. Finishing with nothing changed is refused once; a prose
reply gets one nudge, then escalates; a call to a tool not offered
counts toward failure_limit, like a failed command; calls after
finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
recognises a proxy fallback, which once credited Claude's answers to
a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
that models call only offered tools, and that a declared
supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
declared and backed up
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every turn re-sent the whole transcript with nothing checking it against
the model's window. On a local model that fails silently: Ollama drops
the start of an over-long prompt, which is where the system prompt and
the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with
no error.
- The model catalog reads max_input_tokens and max_output_tokens from
LiteLLM's model_info, and startup refuses a role on a model with no
declared window
- BudgetedClient wraps both routes. It fits the canonical events before
rendering: older tool results cut to head and tail, then the oldest
turns dropped with a note, keeping calls with their results; a request
that still cannot fit fails with ContextOverflow and is never sent.
The output ceiling is the smaller of config and the model's limit
- Tokens are estimated from characters, pessimistically, and the
estimate is corrected from reported prompt sizes, only ever upward
- The executor takes its scope from the log: the latest plan, which is
the approved milestone plan, goes in its system prompt, and it sees
only events after it
- Tool output keeps its head and, mostly, its tail, so a test runner's
failure summary survives truncation
- tuning.context.keep_recent_turns and safety_margin are new knobs
- scripts/probe_context.py checks a declared window against what the
server does; it confirmed llama3.2 cuts at the declared 4096
- The tuning guide covers the budget, declaring a local model's limits,
and the probe
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- uv workspace tying nisse-core, nisse-dev, nisse-ops together
(src layouts, workspace-sourced nisse-core dependency)
- Placeholder hello-world entry points for both agents, calling
through nisse-core to exercise the package boundary
- pytest unit tests per package; integration tests (marker-gated)
running the installed console scripts as subprocesses
- ruff lint + format, nisse_* declared first-party
- Gitea Actions workflow running lint, format check, unit and
integration tests; scripts/check.sh mirrors it locally
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dongarandClaude Opus 5.5