Compare commits

...
Author SHA1 Message Date
dongarandClaude Opus 5 2352a84004 Map A17 onto Outline's actual primitives, and record the MCP decision
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 58s
Observed rather than assumed, after checking the running instance:
collections carry per-user permissions in three tiers and permissions
are collection-scoped, not per-document. So one collection per writing
principal, with read membership for everyone else.

- States precisely what that enforces: no other agent can write a
  given agent's world model. It does not enforce operator-versus-agent
  exclusion, since the operator holds manager on their own wiki; that
  half is procedural, via the break-glass routine that stops the
  agents first. Two different strengths, so the spec no longer implies
  one guarantee covers both
- Per-user grants, not groups: Outline groups are not federated with
  LLDAP, so using them would put membership somewhere nothing syncs.
  Three principals make the manual mapping trivial
- Outline API keys are per-user, so each agent's token must belong to
  that agent's own Outline user. A token minted from an admin account
  voids every boundary above while appearing to work — the same shape
  as the Gitea permission surprise
- Collections identified by ID from config, documents by ID recorded
  in the state store, which also makes writes idempotent across a
  retry
- The conditional-write requirement softens to a revision *read*:
  break-glass stops the agents, so compare-and-swap is unnecessary and
  only divergence detection is needed for A3

Also records Outline's MCP server as considered and deferred. A17's
enforcement depends on this library sitting between the model and the
store; MCP's purpose is direct model access, and a procedure the model
may decline to fetch does not constrain it. Still a candidate for the
discretionary search tool, with two things to verify first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-08 19:36:39 +00:00
dongarandClaude Opus 5 9362a134c9 Give A17 its canonical event, and point Part G at the contract
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 58s
Two loose ends in the A17 amendment, both found by asking whether the
build order needed changing:

- A17 requires retrieved content to be snapshotted into the canonical
  log, but A10's event table — the authoritative list — had no row for
  it, so an implementer reading A10 would never know to add one.
  knowledge_retrieved now carries the text as read, the document ID,
  its revision, and when it was read
- Part G step 10 already scheduled Outline, so the ordering does not
  change; it now names A17 rather than only Part D, and says which
  half arrives first — the world model, with operator procedures
  waiting on the C2 taxonomy they bind to

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-08 19:23:17 +00:00
dongarandClaude Opus 5 b8ba683595 Add A17, the knowledge store contract (Outline)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m3s
Part D said what Outline is for but never how to build against it —
the only external integration with no A-series component, while
Transport, LLM, Browser, Secrets, SSO and Logging all have one. A17
supplies the contract.

Settled here:

- Sharing is read access, never write access. Every document has
  exactly one writer, so concurrent writers never exist and Part D's
  last-write-wins objection disposes of itself. Default shared
  membership is operator + nisse-ops; nisse-dev gets private space
  only, since B3 seals it from the LAN and a shared network map is a
  map of what it is sealed from
- Agent-owned knowledge is always a cache, never authority: rely on
  it to know what exists and where to look, verify a specific fact
  against the live system before acting on it
- Enforcement is structural, because a prompt is not enforcement. A
  model cannot be stopped from believing what it reads, so the
  enforceable goal is narrower — an unverified belief must not become
  an irreversible action — and four mechanisms carry it, leaning
  hardest on surfacing provenance at the human gate that C2 already
  requires
- Never an instruction channel and never an approval channel, stated
  both here and in A4 where approvals are specified
- Corrections flow through transport rather than direct edits, which
  is better provenance: a hand edit is invisible to the agent's
  record, an instruction is authenticated, logged, and reconcilable.
  Break-glass reuses Part F's idle flag
- Retrieved content is snapshotted into the canonical log, since A10
  forbids a mutable provider handle being the only reference — with
  the A3 consequence that boot must re-fetch snapshots for resumed
  tasks
- Outage posture is asymmetric: write failures never block, but a
  failed read of a procedure bound to a confirm-first action gates,
  because proceeding is acting unconstrained on exactly the operation
  someone wrote a procedure for

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-08 19:20:43 +00:00
dongar d8d37b5441 Merge pull request 'Add the delegation harness (A13) — step 7b' (#21) from delegation into main
CI / checks (push) Successful in 1m11s
Reviewed-on: dongar/nisse#21
Reviewed-by: Don Garrett <dgarrett@acm.org>
2026-09-08 00:47:08 +00:00
dongarandClaude Opus 5 af9bdaac8d Stop CI running two suites at once, and retry rate limits
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m5s
CI failed on the delegation branch with a 429 from the proxy: token
limit 200000, 17399 remaining. Not a defect — the live delegation
tests are token-hungry, and every push to a PR branch fired two
identical suites concurrently, since the push and pull_request events
both matched.

- Push now only triggers on main. A PR branch is already covered by
  the pull_request event, so this halves both the wall time and the
  tokens spent per branch
- A 429 is retried once before being surfaced. Losing a ten-turn
  delegation to a transient limit wastes far more than waiting out
  the window, and retrying forever would hide a real ceiling. The
  sleep is injectable, so the existing 429 test no longer waits 20
  seconds for real

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-08 00:18:32 +00:00
dongarandClaude Opus 5 53ed11f9c4 Add the delegation harness (A13) — step 7b
CI / checks (push) Failing after 53s
CI / checks (pull_request) Failing after 43s
A frontier model receives a scoped sub-task, works autonomously with
full tool access, and hands back; the local loop reads the handback
note and the file state, never the reasoning.

- Branch at delegation start, and abort discards it. That required
  committing each turn's work onto the branch: without commits,
  deleting a branch discards nothing, so "abort discards the work"
  was a comforting phrase rather than a fact. The caller's own
  in-flight changes are committed to their branch first, so an abort
  cannot destroy work nobody delegated
- Hard caps on turns and wall clock; a breach is a defined failure
  return, and its result is not usable, so the caller gates rather
  than resuming on a half-edited tree
- Every event is checkpointed as it arrives, so a crash mid-delegation
  leaves the branch at a known commit and a truthful transcript
- Handback is validated locally: blank fields or a non-boolean success
  flag are refused and the delegation continues rather than accepting
  a useless note

Two failure modes found by running it against a real model:

- A success claim with nothing changed was accepted. The handback plus
  file state is the entire interface back, so a note describing work
  that does not exist is worse than no note. Success now requires a
  successful mutating tool call; reporting failure still does not,
  since "this is impossible" is a legitimate finding
- The delegate invented the absolute path /workspace, wrote there, ran
  its tests there, and handed back truthfully — with the repo empty
  and git seeing nothing. Writes outside the workspace are now refused
  with a message naming the real root, and both prompts state where
  the workspace is instead of leaving the model to guess. This is not
  sandboxing: work outside the repo is invisible to git, so the
  milestone would publish an empty PR

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-08 00:14:24 +00:00
dongar 1c8e7e35fc Merge pull request 'Add the Anthropic passthrough renderer and client (A10/A13) — step 7a' (#20) from anthropic-renderer into main
CI / checks (push) Successful in 1m11s
Reviewed-on: dongar/nisse#20
Reviewed-by: Don Garrett <dgarrett@acm.org>
2026-09-08 00:01:30 +00:00
dongarandClaude Opus 5 f4a94281ef Add the Anthropic passthrough renderer and client (A10/A13) — 7a
CI / checks (pull_request) Successful in 43s
CI / checks (push) Successful in 44s
The second renderer adapter, and the concrete case for A13's decision
to normalise in the renderer rather than on the wire. The two vendors
disagree structurally, not cosmetically: the system prompt is a
top-level field rather than a message, a tool call is a tool_use
content block rather than an array, a tool result is a block inside a
user message rather than a "tool" role, and tool schemas use
input_schema rather than parameters. A middlebox flattening those
would be lossy exactly where a long-running delegation lives.

- render_anthropic.py builds vendor-native /v1/messages bodies from
  the canonical log, coalescing adjacent same-role blocks so a
  tool_result lands directly after its tool_use
- Tool manifests are declared once in OpenAI shape and converted per
  vendor, so adding a tool does not mean editing two places
- AnthropicPassthroughClient implements the same LLMClient protocol,
  so callers pick a route rather than a different kind of client, and
  maps provider tool ids to canonical ones at the boundary
- Dangling tool calls get a placeholder result here too; this API
  rejects them more strictly than the OpenAI shape
- Vendor-opaque reasoning blocks are dropped for now with the reason
  recorded: A10 wants them stored and never replayed across sessions,
  and the canonical schema has no event for that yet

Live tests prove the body is accepted as sent, that a tool_result
turn round-trips, and — as A5 asks — that prompt cache accounting
survives the passthrough rather than being flattened by the proxy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 23:13:22 +00:00
dev-agent 1f292793c1 Merge pull request 'Record default-branch protection as a B2 requirement' (#19) from b2-branch-protection into main
CI / checks (push) Successful in 39s
2026-09-07 22:55:22 +00:00
dev-agent f03c201729 Merge pull request 'Add PR comment intake and the review loop (B2) — step 6b' (#17) from review-loop into main
CI / checks (push) Successful in 43s
2026-09-07 22:55:11 +00:00
dongarandClaude Opus 5 886cb80583 Count only mutating tools as evidence of work
CI / checks (push) Successful in 38s
CI / checks (pull_request) Successful in 41s
The previous fix challenged a completion claim made before any tool
call, but treated every tool as work — so a model that ran one
list_dir and then declared the milestone done was believed, and CI
caught the same empty milestone again.

Reading a file or listing a directory is the model orienting itself.
Only write_file and run_command now count, so a completion claim must
be backed by something that changed or verified the tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 22:53:20 +00:00
dongarandClaude Opus 5 846e5494ca Record default-branch protection as a B2 requirement
CI / checks (pull_request) Successful in 39s
CI / checks (push) Successful in 32s
B2 specified the service account and its scoped token but never said
the project repo's default branch must be protected — which turns out
to be the load-bearing part of the PR gate.

In Gitea, write access includes merging pull requests, and repository
ownership does not restrict merging either. So an agent with the
access it needs to push branches can also merge its own PR, and the
review gate holds only because the agent's code does not call the
merge endpoint. That is the advisory guarantee A4 exists to rule out.

Verified rather than assumed: before protection was configured, the
agent's own token merged its own PR (HTTP 200). With protection it is
refused (405), as is a direct push to the default branch. The
amendment says to assert both, so relaxing the protection later fails
a test instead of silently removing the gate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 22:48:42 +00:00
dongar 8f365263c4 Merge pull request 'Externalise plans to Gitea and accept approvals there (A4, Part H)' (#18) from gitea-approvals into review-loop
CI / checks (push) Failing after 56s
CI / checks (pull_request) Successful in 39s
Reviewed-on: dongar/nisse#18
Reviewed-by: Don Garrett <dgarrett@acm.org>
2026-09-07 22:47:40 +00:00
dongarandClaude Opus 5 acee8b96f0 Do not believe a completion claim before any work was done
CI / checks (push) Successful in 44s
CI / checks (pull_request) Successful in 41s
CI caught the executor recording an empty milestone as a success: the
model answered with prose on its first turn, and the loop reads "no
tool call" as "the plan is done". It had therefore finished a
milestone having touched nothing, and the milestone would have gone
on to publish with no changes.

A completion claim is now challenged once when no tool has been
called yet — the same shape as the planner's nudge — and a model that
persists escalates instead of reporting success. After the first tool
call the claim is taken at face value, as before.

The live milestone test failed roughly one run in five on this; four
consecutive runs pass now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 22:19:15 +00:00
dongarandClaude Opus 5 d508e9a9b7 Externalise plans to Gitea and accept approvals there (A4, Part H)
CI / checks (push) Successful in 43s
CI / checks (pull_request) Successful in 41s
- approvals.py: PlanBoard opens one issue per project and posts each
  plan revision as a comment carrying its rev tag. The state store
  keeps only a pointer, so gate state can be re-derived from Gitea
  after a state-volume loss — external truth wins
- Gitea comments are a second approval source: revision binding, the
  typed word approve, and first-approval-wins work exactly as over
  transport, with a per-channel allowlist because a Gitea login is
  not a Matrix MXID
- Outcomes are echoed to the other channel, so the two records cannot
  silently disagree about what was approved
- A Gitea outage degrades durability of the record, not the ability
  to approve: planning proceeds and Matrix remains a complete channel
- Live test now asserts the server refuses the agent's own merge and
  its push to the protected branch. That property was previously
  true only because our code did not attempt it; branch protection
  makes it structural, and this test fails if the protection is ever
  relaxed
- Fixed a race in the process-group test: killing a group is
  asynchronous and the grandchild is an orphan, so its exit is not
  synchronised with our return; the test now waits for death

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 22:15:08 +00:00
dongarandClaude Opus 5 affdb9e8aa Add PR comment intake and the review loop (B2) — step 6b
CI / checks (push) Successful in 42s
CI / checks (pull_request) Successful in 43s
- poller.py: CommentPoller polls PR comments rather than taking
  webhooks, keeping the VM outbound-only (B3). On an outage it backs
  off exponentially, caps, and retries indefinitely while logging:
  polling is pure intake, so an outage delays a reaction but never
  fails a task or forces a transition
- prime() marks the existing thread as seen without delivering it, so
  starting a watch does not re-address feedback already handled
- Review loop: a comment in PR_OPEN enters the canonical transcript,
  the executor responds, and the fix is pushed to the same branch with
  a reply on the PR. No intent is recorded for a follow-up push — the
  PR already exists, so a repeated push is harmless, unlike opening a
  second PR
- Comment authors are checked against a separate Gitea operator
  allowlist: a PR comment is an instruction channel into an agent that
  writes code, so it is operator-scoped, and a Gitea login is not a
  Matrix MXID (A4 per-channel allowlists)
- Orchestrator now holds a lock: transport and the poller deliver on
  separate threads, and state transitions must not interleave
- Backoff timings are injectable, so the outage path is tested in
  milliseconds instead of being untestable

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 21:59:50 +00:00
dongar 914bc26221 Merge pull request 'Publish finished milestones as PRs (B2/A3) — step 6a, part 2' (#16) from pr-open into main
CI / checks (push) Successful in 34s
Reviewed-on: dongar/nisse#16
Reviewed-by: Don Garrett <dgarrett@acm.org>
2026-09-07 20:45:57 +00:00
dongar 3b1b84162a Merge pull request 'Add a manual CI trigger (workflow_dispatch)' (#15) from ci-scheduled into main
CI / checks (push) Successful in 35s
Reviewed-on: dongar/nisse#15
Reviewed-by: Don Garrett <dgarrett@acm.org>
2026-09-07 20:44:48 +00:00
dongarandClaude Opus 5 3117565928 Publish finished milestones as PRs (B2/A3) — step 6a, part 2
CI / checks (push) Successful in 41s
CI / checks (pull_request) Successful in 38s
Completes the B1 loop past EXECUTING, and answers the question raised
reviewing #6: PR acceptance is a milestone's final gate, so there is
no separate FINISHED state — completion returns to milestone planning
or to IDLE.

- publish.py: MilestonePublisher commits the work on a task-namespaced
  branch, pushes, and opens a PR carrying the approved plan. Intent is
  recorded before the first externally visible act, so a crash
  anywhere in the sequence leaves a durable record and boot
  reconciliation asks Gitea whether the PR exists rather than opening
  a second one
- Nothing to commit is reported, not raised: the executor may
  legitimately conclude there was nothing to change
- control.py: PR_OPEN state with EXECUTING -> PR_OPEN and
  PR_OPEN -> {MILESTONE_PLANNING, IDLE}; operator commands
  'next milestone' and 'project complete'
- A configured publisher always contributes its own reconciler at
  boot, so an interrupted publish is settled against external truth
  without the caller having to remember to register it
- A publish failure gates to the operator rather than retrying into a
  possible duplicate

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 20:43:20 +00:00
dongarandClaude Opus 5 4d0b24794b Drop the scheduled run, keep the manual trigger
CI / checks (push) Successful in 35s
CI / checks (pull_request) Successful in 37s
Operator does not want nightly or weekly runs. Pushes to main already
cover code changes, so the remaining addition is workflow_dispatch: a
manual re-run for after a credential rotation or a budget top-up,
which also retires the empty-commit trick used to pick up new secrets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 20:40:12 +00:00
dongarandClaude Opus 5 3e664984ac Run CI nightly and on manual dispatch
CI / checks (pull_request) Successful in 35s
CI / checks (push) Successful in 33s
Pushes to main already trigger a full run, so code changes were
covered. What was not: the live tests depend on external state that
drifts with no commit at all — the LiteLLM budget filling up, a model
withdrawn from the proxy, a token revoked or expired. A nightly run
against the default branch surfaces that the morning it happens
instead of mid-task.

workflow_dispatch adds a manual trigger, which also removes the need
for the empty-commit trick used earlier to pick up new secrets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 20:38:06 +00:00
dev-agent 4025ac7875 Merge pull request 'Add executor tool loop (A12/A15) — step 5b' (#12) from executor-loop into main
CI / checks (push) Successful in 34s
2026-09-07 20:35:33 +00:00
dev-agent 1ebd2a9988 Merge pull request 'Add Gitea client and git operations (B2) — step 6a, part 1' (#14) from gitea-client into main
CI / checks (push) Successful in 15s
2026-09-07 20:35:17 +00:00
dev-agent ad45e33517 Merge pull request 'Assert the live Matrix tests run as a testing identity' (#13) from matrix-identity-guard into main
CI / checks (push) Successful in 17s
2026-09-07 20:35:09 +00:00
dongarandClaude Opus 5 8dd5e3eb15 Page to the end when searching for an existing PR
CI / checks (push) Successful in 19s
CI / checks (pull_request) Successful in 15s
find_pull_request read only the newest 50 per state, so once a repo
passed that many pull requests a "not found" would mean "not in the
newest page" — and the A3 reconciler would read that as "the PR was
never opened" and open a duplicate. Exactly the side effect the
intent log exists to prevent.

It now pages until a short page proves the end, and raises rather
than answering "absent" from an incomplete search; the reconciler
turns that into unverifiable and gates to the human.

Found while wiring CI: every CI run opens a PR in the sandbox, so the
threshold would have been crossed in weeks rather than never.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 20:31:31 +00:00
dongarandClaude Opus 5 2071e6ab6d Add Gitea client and git operations (B2) — step 6a, part 1
CI / checks (push) Successful in 16s
CI / checks (pull_request) Successful in 14s
The standalone half of the Gitea layer; orchestrator wiring follows
once the executor loop lands, to avoid colliding with it in
control.py.

- gitea.py: GiteaClient over httpx — open PRs, list and post issue
  comments, close PRs. find_pull_request is the A3 external-truth
  query that makes opening a PR idempotent across a crash, and it
  checks closed PRs too, since a crash after opening must not be
  answered with "nothing is open, so open another"
- pull_request_reconciler: an unreachable Gitea returns None rather
  than a guess, so the agent gates to the human instead of risking a
  duplicate side effect
- git.py: branch, commit, push for the workspace. The token is passed
  per-invocation rather than stored as a remote, so it never lands in
  .git/config, and anything raised is scrubbed of it
- commit_all returns None when there is nothing to commit: an empty
  milestone is a fact to report, not an error
- Live test does the real round trip against the sandbox repo and
  cleans up after itself

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 20:27:34 +00:00
dongarandClaude Opus 5 e04d786840 Assert the live Matrix tests run as a testing identity
CI / checks (pull_request) Successful in 12s
CI / checks (push) Successful in 14s
Two invariants only a live homeserver can check, now enforced rather
than verified by hand:

- The configured NISSE_MATRIX_USER_ID must be the token's real owner.
  MatrixTransport filters its own messages by comparing sender to that
  value, so a mismatch would make the bot read its own posts as
  operator input.
- The account must be a testing identity, so a stray live run can
  never act as the production agent.

Both hold today; this keeps them holding when the production
nisse-dev account is created.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 20:16:18 +00:00
dongarandClaude Opus 5 bacf65dbe2 Fix AWAIT_INPUT deadlock and make the live milestone test robust
CI / checks (push) Successful in 36s
CI / checks (pull_request) Successful in 29s
Found while chasing a flaky live test, but the flakiness was a real
bug: only the ask_human path recorded a pending item, so AWAIT_INPUT
reached any other way — planning failure, stopped milestone, boot
reconciliation — refused every operator reply with "no pending
question to answer" and the agent could never be recovered. The
operator had no way back short of wiping state.

Every path into AWAIT_INPUT now records what a reply should resume,
and answering dispatches on it: planning failures and questions
resume planning, a stopped milestone returns to its still-approved
gate, and the fallback says what commands work instead of
dead-ending. Two regression tests cover the recovery paths.

The live test also scripted a fixed approve/approve/start sequence,
which a real model breaks by legitimately calling ask_human at a
gate. It now plays the operator adaptively, responding to whatever
the agent actually did. Five consecutive live runs pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 16:46:07 +00:00
dongarandClaude Opus 5 999a38a705 Fix process-group kill test to work without an init process
CI / checks (push) Failing after 23s
CI / checks (pull_request) Successful in 30s
os.kill(pid, 0) succeeds on a zombie, and a killed orphan stays a
zombie until reaped — which never happens inside a CI container with
no init. The test read that as "still alive" and failed there while
passing locally. It now reads the process state from /proc, with a
control assertion so a broken liveness check cannot pass silently.
The group kill itself was correct.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 16:37:38 +00:00
dongarandClaude Opus 5 0fd0b2e4a5 Add executor tool loop (A12/A15) — step 5b
CI / checks (push) Failing after 13s
CI / checks (pull_request) Failing after 13s
- tools.py: four flat tools (read_file, write_file, list_dir,
  run_command) per A15's reduced manifest. Deliberately unsandboxed:
  the VM is the trust boundary (B3) and tool use is auto-approved
  inside an approved unit of work (A4), so path confinement would add
  failure modes without adding safety. What is enforced is
  operational — timeout with process-group kill so a backgrounded
  server cannot outlive its command, output truncation so a build log
  cannot eat the context window, closed stdin so prompts fail fast,
  and a scrubbed environment so project code never sees the agent's
  credentials
- executor.py: renders the log, calls the executor role with the
  reduced manifest, dispatches tool calls, and checkpoints every
  result (A13). Escalation is observable, never self-assessed (A12):
  consecutive command failures and a hard turn cap. Failed reads do
  not count — looking around is not failing
- Until delegation exists (step 7), escalation gates to the human
  with the failures attached
- control.py: start milestone now runs the executor; a stopped
  milestone gates to AWAIT_INPUT

Two bugs found by running it live against real models:

- A turn-ending gate tool (submit_plan) left a tool_call with no
  tool_result, which providers reject outright, so every planning
  turn after the first failed with a 400. The planner now records the
  gate as the call's outcome, which is also the truthful transcript
- The renderer now emits a placeholder for any tool_call lacking a
  result, so a crash between call and result (A3) cannot poison the
  transcript permanently

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 16:35:39 +00:00
dongar 47b2bdfdbd Merge pull request 'Add LLM-backed planning agent (A4/A11/A13) — step 5a' (#11) from llm-planning into main
CI / checks (push) Successful in 22s
Reviewed-on: dongar/nisse#11
Reviewed-by: Don Garrett <dgarrett@acm.org>
2026-09-07 16:15:47 +00:00
dongarandClaude Opus 5 fb35ca774d Trigger CI to pick up the NISSE_LIVE_PLANNER_MODEL secret
CI / checks (push) Successful in 20s
CI / checks (pull_request) Successful in 18s
Empty commit: Gitea 1.27 exposes no rerun API endpoint, and Actions
secrets are read at run start, so a new run is needed to exercise the
live planning test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 16:02:56 +00:00
dongarandClaude Opus 5 6c17f2bf57 Enable the live planning test against claude-haiku-4-5
CI / checks (push) Successful in 13s
CI / checks (pull_request) Successful in 12s
The virtual key now reaches a Haiku-class route, which holds the
tool-call contract llama3.2 could not: a real planning turn returns a
well-formed submit_plan call with an explicit, low-inference plan.

Wires NISSE_LIVE_PLANNER_MODEL through the CI workflow (add the
matching Actions secret to run it there) and records the finding in
.env.example.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 16:01:18 +00:00
dongarandClaude Opus 5 cf85648719 Add LLM-backed planning agent (A4/A11/A13) — step 5a
CI / checks (push) Successful in 12s
CI / checks (pull_request) Successful in 11s
- planner.py: LLMPlanningAgent offers exactly two tools, submit_plan
  and ask_human, so gating stays structural at the tool layer; no
  proceed tool exists. System prompt requires explicit, low-inference
  plans because a weaker model executes them (A13)
- Model output is appended to the canonical log as it arrives, before
  being interpreted, so a crash mid-planning leaves a truthful
  transcript
- A bad turn (prose instead of a tool call, malformed arguments, or a
  failed call) retries once, then raises PlanningError; the
  orchestrator gates to AWAIT_INPUT rather than inventing a plan
- Orchestrator: appends every operator message, plan_submitted, and
  gate_opened to the transcript, so re-planning sees rejection
  feedback as conversation rather than out-of-band parameters; pins
  the A11 role mapping at task start
- llm.py: non-JSON or non-object tool arguments now raise LLMError
  instead of escaping as JSONDecodeError — observed from llama3.2,
  which emits truncated argument strings
- Live planning test is opt-in on NISSE_LIVE_PLANNER_MODEL: planning
  escalates unconditionally (A12) and the local model does not hold
  the tool-call contract, so it awaits a frontier route

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 14:47:12 +00:00
36 changed files with 5365 additions and 43 deletions

No files matched your search

+15
View File
@@ -11,6 +11,12 @@ NISSE_ROLE_PLANNER=anthropic/claude-opus-5
NISSE_ROLE_DELEGATE=anthropic/claude-opus-5 NISSE_ROLE_DELEGATE=anthropic/claude-opus-5
NISSE_ROLE_CONSULTANT=anthropic/claude-opus-5 NISSE_ROLE_CONSULTANT=anthropic/claude-opus-5
# Opt-in for the live planning test: a model the virtual key can reach that
# holds the tool-call contract. Planning escalates unconditionally (A12), and
# small local models do not hold the contract reliably — llama3.2 returns
# truncated tool arguments, Haiku-class and up are solid.
NISSE_LIVE_PLANNER_MODEL=claude-haiku-4-5
# --- Matrix transport (A1), used by nisse-dev's MatrixTransport --- # --- Matrix transport (A1), used by nisse-dev's MatrixTransport ---
NISSE_MATRIX_HOMESERVER=https://matrix.example NISSE_MATRIX_HOMESERVER=https://matrix.example
NISSE_MATRIX_USER_ID=@nisse-dev:example NISSE_MATRIX_USER_ID=@nisse-dev:example
@@ -19,3 +25,12 @@ NISSE_MATRIX_ROOM_ID=!roomid:example
# Dedicated room for the live integration test; never the production room. # Dedicated room for the live integration test; never the production room.
NISSE_MATRIX_TEST_ROOM_ID=!testroomid:example NISSE_MATRIX_TEST_ROOM_ID=!testroomid:example
NISSE_MATRIX_OPERATORS=@you:example NISSE_MATRIX_OPERATORS=@you:example
# --- Gitea (B2): service account for branches, PRs, and comment intake ---
NISSE_GITEA_URL=https://gitea.example
NISSE_GITEA_TOKEN=your-scoped-service-account-token
# Sandbox repo for live tests; never the real project repo.
NISSE_GITEA_TEST_REPO=owner/sandbox-repo
# Gitea logins allowed to drive the agent via PR comments (A4 per-channel
# allowlist; a Matrix MXID and a Gitea login are different names).
NISSE_GITEA_OPERATORS=your-gitea-login
+11
View File
@@ -1,7 +1,14 @@
name: CI name: CI
on: on:
# Only main on push. A PR branch is already covered by the pull_request
# event, and running both fired two identical suites concurrently — which
# doubled the LLM tokens the live tests spend and tripped the proxy's
# per-minute limit.
push: push:
branches: [main]
pull_request: pull_request:
# Manual re-run, e.g. after rotating a credential or raising a budget cap.
workflow_dispatch:
jobs: jobs:
checks: checks:
@@ -34,8 +41,12 @@ jobs:
NISSE_LLM_BASE_URL: ${{ secrets.NISSE_LLM_BASE_URL }} NISSE_LLM_BASE_URL: ${{ secrets.NISSE_LLM_BASE_URL }}
NISSE_LLM_TOKEN: ${{ secrets.NISSE_LLM_TOKEN }} NISSE_LLM_TOKEN: ${{ secrets.NISSE_LLM_TOKEN }}
NISSE_ROLE_EXECUTOR: ${{ secrets.NISSE_ROLE_EXECUTOR }} NISSE_ROLE_EXECUTOR: ${{ secrets.NISSE_ROLE_EXECUTOR }}
NISSE_LIVE_PLANNER_MODEL: ${{ secrets.NISSE_LIVE_PLANNER_MODEL }}
NISSE_MATRIX_HOMESERVER: ${{ secrets.NISSE_MATRIX_HOMESERVER }} NISSE_MATRIX_HOMESERVER: ${{ secrets.NISSE_MATRIX_HOMESERVER }}
NISSE_MATRIX_USER_ID: ${{ secrets.NISSE_MATRIX_USER_ID }} NISSE_MATRIX_USER_ID: ${{ secrets.NISSE_MATRIX_USER_ID }}
NISSE_MATRIX_TOKEN: ${{ secrets.NISSE_MATRIX_TOKEN }} NISSE_MATRIX_TOKEN: ${{ secrets.NISSE_MATRIX_TOKEN }}
NISSE_MATRIX_TEST_ROOM_ID: ${{ secrets.NISSE_MATRIX_TEST_ROOM_ID }} NISSE_MATRIX_TEST_ROOM_ID: ${{ secrets.NISSE_MATRIX_TEST_ROOM_ID }}
NISSE_GITEA_URL: ${{ secrets.NISSE_GITEA_URL }}
NISSE_GITEA_TOKEN: ${{ secrets.NISSE_GITEA_TOKEN }}
NISSE_GITEA_TEST_REPO: ${{ secrets.NISSE_GITEA_TEST_REPO }}
run: uv run pytest -m integration -rs run: uv run pytest -m integration -rs
+191 -2
View File
@@ -175,7 +175,9 @@ Requirements:
task. The agent has no path to approve itself. task. The agent has no path to approve itself.
- **Approval sources.** A gate waits on an approval event from any registered - **Approval sources.** A gate waits on an approval event from any registered
**`ApprovalSource`** — the Matrix transport (A1), the Gitea comment poller (B2), or **`ApprovalSource`** — the Matrix transport (A1), the Gitea comment poller (B2), or
the Vikunja comment stream (C4). First valid approval wins and is consumed; the the Vikunja comment stream (C4). **The knowledge store (A17) is never an approval
source**, and never an instruction channel: a wiki page has neither the identity
guarantees nor the revision binding an approval requires. First valid approval wins and is consumed; the
gate records which channel satisfied it, and the orchestrator echoes the outcome to gate records which channel satisfied it, and the orchestrator echoes the outcome to
the other channels so the records converge. Each source authenticates the approver the other channels so the records converge. Each source authenticates the approver
against a per-channel operator allowlist (Matrix MXID, Gitea username, Vikunja against a per-channel operator allowlist (Matrix MXID, Gitea username, Vikunja
@@ -333,6 +335,7 @@ The durable record is a **canonical event log** in SQLite, not a provider-shaped
| `delegation_started` | brief, target role, resolved model, caps | | `delegation_started` | brief, target role, resolved model, caps |
| `delegation_completed` | handback note, artifacts, tokens, duration, exit status | | `delegation_completed` | handback note, artifacts, tokens, duration, exit status |
| `compaction` | summary, producing model, range of events replaced | | `compaction` | summary, producing model, range of events replaced |
| `knowledge_retrieved` | document text as read, document ID, revision, retrieved-at (A17) |
Rules: Rules:
- **Canonical tool-call IDs are minted locally.** Provider IDs are mapped at the - **Canonical tool-call IDs are minted locally.** Provider IDs are mapped at the
@@ -507,6 +510,167 @@ additional cost is marking outcomes as good or bad.
Without this, "best model this week" is decided by benchmark chatter, and swapping the Without this, "best model this week" is decided by benchmark chatter, and swapping the
frontier tier becomes something to avoid because the effect is unmeasurable. With it, a frontier tier becomes something to avoid because the effect is unmeasurable. With it, a
new model is a config change and a re-run. new model is a config change and a re-run.
### A17. Knowledge Store (Outline)
**Purpose:** durable knowledge that outlives a single task — an agent-maintained
**world model** (what infrastructure exists, how it is arranged) and, later,
operator-authored **procedures**. Distinct from A2, which holds *task* state: A2
is the agent's memory of what it is doing, A17 is its memory of what the world
looks like. Neither is a fallback for the other.
Part D describes the service; this defines the contract.
**Layout — sharing is read access, never write access.**
- Every document has **exactly one writer**: one agent, or the operator. Sharing
governs who may *read*. Concurrent writers therefore never exist, which
disposes of the last-write-wins objection Part D raises, without needing
distributed-write machinery.
- **Agent-private** space per agent: its own world model.
- **Shared** space: readable by the operator and designated agents. Default
membership is **operator + `nisse-ops`**. `nisse-dev` gets private space only —
B3 seals it from RFC1918 precisely so it cannot reach the NAS, Proxmox, or the
UDM, and a shared network-architecture document is a map of exactly those
things. Cross-agent read access is justified per collection, never assumed.
- The dev agent's project conventions live in the **repo it is working on**, not
here: versioned with the code, reviewed in the PR, and impossible to
desynchronise from the project.
**Agent-owned knowledge is always a cache, never authority.** An agent may rely on
it to know *what exists and where to look*; it must confirm a specific fact
against the live system *before acting on it*. Hard requirement for anything in
C2's confirm-first tier; advisory for read-only and reversible actions.
**Enforcement is structural, because a prompt is not enforcement.** "Treat this as
a cache" is exactly the advisory guarantee A4 and A12 reject, and the model cannot
be prevented from *believing* a fact it has read. The enforceable goal is
narrower: **an unverified belief must not become an irreversible action.**
1. **The store decides what it serves.** Facts carry provenance and an observed-at
timestamp; retrieval marks or withholds anything past a staleness bound for
high-consequence classes.
2. **Verification as a precondition, checked against the log.** For confirm-first
actions, refuse unless a verifying observation exists *in this task's canonical
log*, recorded after the task began, naming the action's target. Inspectable
without asking the model anything — the same move as A12's observable triggers.
Approximate by nature, so build it for the confirm-first tier only.
3. **Provenance at the human gate** — the primary reliance, because C2 already
stops these actions for confirmation. The prompt states evidence quality:
"target recalled from the world model, observed 2026-08-02, not verified this
session" versus "confirmed by `zfs list` two minutes ago". The operator is
already the authority there; this makes the risk legible rather than invisible.
4. **Audit, not prevention, for the cheap tiers.** Log which facts were retrieved
and which actions followed, so a wrong fact is traceable afterwards.
Residual risk, stated plainly: a model can retype a fact from context with no
retrieval event, and no mechanism here catches that. That is why the world model
stays out of the authority path entirely.
**Never an instruction channel, and never an approval channel.**
- **Task instructions never come from the knowledge store.** Work is driven by
Vikunja tickets (C4) and operator commands (B2). An agent must never read the
store looking for work to do.
- **Approvals never come from the knowledge store.** The registered
`ApprovalSource`s are Matrix (A1), Gitea (B2), and Vikunja (C4). A wiki page is
not one and must not become one.
- An agent must never write here intending to cause another agent to act. This
extends Part D's rejection of the store as a message bus: the erosion path is
one well-meaning "note for the other agent" at a time.
**Corrections flow through transport, not through direct edits.** The operator
corrects a fact by telling the agent, which is strictly better than editing the
page by hand: a hand edit is invisible to the agent's record, whereas an
instruction arrives on an authenticated channel from an allowlisted identity, is
recorded in the canonical log, and can be reconciled against what the agent
believed. Corrections are stored **verbatim with attribution**, never paraphrased —
a model rewriting a precise fact loses precision invisibly.
- **Break-glass:** the operator keeps write permission. The procedure reuses Part
F's idle flag — wait for idle, stop the agent, edit freely, restart. Single
writer holds trivially while the agent is down, and A2/A3 make stopping safe.
- Because that permission exists, an agent re-reads and compares the revision
immediately before writing, and re-merges rather than clobbering if it moved.
A safety net for the rare edit, not a designed workflow.
**Canonical log representation (A10).** Retrieved content is **snapshotted into the
log** — the text, plus document id and revision for provenance — never referenced
by handle alone: A10 forbids a provider handle being the only reference to content,
and these documents are mutable. Snapshot once per task at first use, not per turn.
**A3 consequence:** a task resumed across a restart carries the old snapshot, so
boot reconciliation re-fetches snapshotted knowledge and flags divergence rather
than silently continuing on a stale copy.
**Retrieval is structural first, discretionary second** — the A12/A14 pattern.
- Procedures are **bound to C2 action classes**: the policy taxonomy and the
procedure index are the same list. Binding is deterministic and auditable
("which procedure governed this action?" has an exact answer), and costs nothing
extra because the taxonomy must exist anyway.
- A model-invoked **search** exists *in addition* for discovery, never as a
replacement for the bound retrieval — a procedure the model may skip does not
constrain it.
**Outage posture is asymmetric.** Unlike B2's poller, "never fails a task" does not
generalise here.
- **Write failure never blocks:** retry, or record locally and reconcile.
- **Read failure for a procedure bound to a confirm-first action gates.**
Proceeding is not a delayed reaction; it is acting unconstrained on precisely the
operation someone wrote a procedure for.
- Read failure for read-only or reversible actions proceeds, logging the gap.
**Accepted single point of failure.** Once the agent is the sole author and Outline
the only home, losing Outline loses the world model. No local mirror: infrastructure
facts are re-derivable by observation, which is the same premise the cache rule
rests on.
**Deferred:** the operator-writable procedures collection. The interface is defined
here; the content and the C2 binding list arrive with the policy taxonomy (Part H).
**Concrete mapping onto Outline.** Observed in the running instance: a **collection**
carries per-user permissions with three tiers — visible-to (read), read-write, and
manager. Permissions are collection-scoped, not per-document.
- Therefore **one collection per writing principal**, with read membership granted to
everyone else who needs it: `nisse-ops-world` (write: `nisse-ops`),
`nisse-dev-world` (write: `nisse-dev`), `procedures` (write: operator).
- **What this enforces, precisely:** no *other agent* can write a given agent's world
model — the collision that would otherwise be silent and unattributable. It does not
enforce operator-versus-agent exclusion, because the operator holds manager on their
own wiki and can always grant themselves write. That half is procedural: the
break-glass routine stops the agents first, so concurrent writers do not exist while
editing.
- **Assign per user, not per group.** Outline groups appear to be internal to Outline
rather than federated with LLDAP (A8), so using them would put membership in a second
place that nothing syncs. With three principals the manual mapping is trivial and
obvious; revisit only if the cast grows.
- **Each agent's API token must belong to that agent's own Outline user.** Outline API
keys are per-user, so the token *is* the identity every permission above evaluates
against. A token minted from the operator's account — or an admin's — silently voids
the entire boundary while appearing to work.
- **Identify collections and documents by ID, from config and state.** Collection IDs
come from env (`NISSE_OUTLINE_*_COLLECTION`), never discovered from "whatever the
token can write": a permission change must not silently redefine what an agent
considers its own. Document IDs are recorded in the state store (A2) when created and
used for updates, which also makes writing idempotent — a retry after a crash updates
the document instead of creating a second one.
- **Revision reads, not conditional writes.** Because break-glass stops the agents,
compare-and-swap is unnecessary; what is needed is the ability to *observe* that a
document changed since the agent last read it, feeding the A3 divergence check.
Whether Outline exposes a revision identifier or only a modified timestamp is
unverified — either satisfies this.
#### Considered and deferred: Outline's MCP server
Outline can expose search, read, create and edit over MCP (remote, Streamable HTTP,
enabled workspace-wide). **Not adopted for the retrieval path**, because A17's
enforcement depends on this library sitting *between* the model and the store: staleness
marking, snapshotting into the canonical log, binding procedures to action classes, and
gating a failed confirm-first read all happen to a retrieval before the model sees it.
MCP's purpose is the opposite — direct model access — and a procedure the model may
decline to fetch does not constrain it (A12).
It remains a reasonable candidate for the *discretionary* search tool later, and unlike
the vendor agent SDKs A13 rejects, MCP is a neutral standard rather than lock-in: one
client would serve many servers if others are ever added. Before adopting it, verify
that it honours the key owner's **collection** permissions rather than mere workspace
membership, and that the workspace-level toggle does not force broader exposure than
intended.
--- ---
@@ -550,6 +714,25 @@ PR_OPEN
- *Outbound:* `git` + Gitea API to push branches and open PRs, via a dedicated - *Outbound:* `git` + Gitea API to push branches and open PRs, via a dedicated
Gitea **service account** with a scoped token. Record any runnable test service's Gitea **service account** with a scoped token. Record any runnable test service's
URL/port in the PR description. URL/port in the PR description.
- **The project repo's default branch must be protected, and this is what makes
the PR gate real.** The service account needs write access to push branches —
and in Gitea, *write access includes merging pull requests*. Repository
ownership does not help: an account that merely collaborates can still merge,
and one that owns the repo can merge and delete it. So without branch
protection the agent can merge its own PR, and the review gate holds only
because the agent's code does not call the merge endpoint. That is precisely
the advisory guarantee A4 exists to rule out.
- Required on the default branch: **block direct pushes** by the agent
account, and **restrict merging** to the operator (a merge whitelist, or
required approvals that the agent's own account cannot satisfy).
- Feature-branch pushes are unaffected, so the agent's normal workflow is
unchanged.
- **Verify it rather than assume it.** The property is observable from
outside: the agent's own token must be refused when it attempts to merge
its own PR and when it pushes to the default branch. Assert both, so that
relaxing the protection later fails a test instead of silently removing
the gate. (Discovered the hard way — before protection was configured, the
agent's token merged its own PR successfully.)
- *Inbound:* react to PR comments, commits, pushes. **Poll** the Gitea API - *Inbound:* react to PR comments, commits, pushes. **Poll** the Gitea API
(15–30s) rather than take webhooks, to keep the VM outbound-only. Intake is (15–30s) rather than take webhooks, to keep the VM outbound-only. Intake is
structured JSON — never via a chat bridge. structured JSON — never via a chat bridge.
@@ -670,6 +853,10 @@ Requirements:
back to Outline: searchable, off-box, and an extra externalized record that back to Outline: searchable, off-box, and an extra externalized record that
survives volume loss. survives volume loss.
- Access via Outline **REST API**, dedicated service account, token from Bitwarden. - Access via Outline **REST API**, dedicated service account, token from Bitwarden.
- **The library contract is A17**, which settles what this section left implicit:
per-agent private space plus a shared space, one writer per document, agent-owned
knowledge treated as a cache rather than authority, and the structural means of
keeping it that way.
**Deliberately rejected:** using a shared notes store (Outline *or* an Obsidian vault) **Deliberately rejected:** using a shared notes store (Outline *or* an Obsidian vault)
as an **agent-to-agent message bus**. It's unstructured, race-prone (last-write-wins, as an **agent-to-agent message bus**. It's unstructured, race-prone (last-write-wins,
@@ -748,7 +935,8 @@ each agent is built — not standing up the service itself.
here). Crash-atomic writes and reconcile-on-boot must be in place first. here). Crash-atomic writes and reconcile-on-boot must be in place first.
10. **Extract `nisse-core`**; build the **Sysadmin Agent** (`nisse-ops`): policy 10. **Extract `nisse-core`**; build the **Sysadmin Agent** (`nisse-ops`): policy
engine (C2), SSO + scoped browser (A8/A6), secrets (A7), Vikunja control (C4), engine (C2), SSO + scoped browser (A8/A6), secrets (A7), Vikunja control (C4),
Outline runbooks (D), scheduled checks, alert intake. knowledge store (A17/D) — world model first, operator procedures when the
C2 taxonomy they bind to exists — scheduled checks, alert intake.
- With the monorepo layout (Part H, decided), extraction is a **package split** - With the monorepo layout (Part H, decided), extraction is a **package split**
within one tree: move code behind a package boundary, no release machinery within one tree: move code behind a package boundary, no release machinery
required. required.
@@ -816,6 +1004,7 @@ nisse-core/ # shared library (extracted after nisse-dev works)
secrets/ # Bitwarden machine-account client (A7) secrets/ # Bitwarden machine-account client (A7)
sso/ # LLDAP identity + auth path (A8) sso/ # LLDAP identity + auth path (A8)
logging/ # structured operational logs (A9) logging/ # structured operational logs (A9)
knowledge/ # Outline client, world model, procedures (A17)
nisse-dev/ # app: sealed VM (Part B) nisse-dev/ # app: sealed VM (Part B)
workflow (arch/milestone gating), Gitea PR layer, allow-all sandbox workflow (arch/milestone gating), Gitea PR layer, allow-all sandbox
+80
View File
@@ -0,0 +1,80 @@
"""Plan externalisation and Gitea-backed approvals (spec A4, Part H).
Plans are posted as comments on a per-project Gitea issue. The approval
comment — its author, timestamp, and revision tag — is the durable off-box
record, authenticated by Gitea's own auth. The state store keeps only a
pointer to it, so boot reconciliation can re-derive gate state from Gitea
even after the state volume is lost: external truth wins.
Approvals arriving here bind to a revision exactly as they do over transport
(A4). The first valid approval on any channel wins; the orchestrator echoes
the outcome to the other channel so the records converge.
"""
from dataclasses import dataclass
from nisse_dev.gitea import Comment, GiteaClient, GiteaError
APPROVAL_HINT = (
"Reply `approve {tag}` to approve this revision, or comment with feedback "
"to reject it."
)
@dataclass(frozen=True)
class PlanPost:
issue: int
comment_id: int
class PlanBoard:
"""The Gitea side of plan gating: one issue per project, one comment per
plan revision."""
def __init__(self, client: GiteaClient, repo: str) -> None:
self._client = client
self._repo = repo
@property
def repo(self) -> str:
return self._repo
def open_project_issue(self, project: str, task_id: str) -> int:
return self._client.create_issue(
self._repo,
title=f"nisse-dev: {project}",
body=(
f"Plan gate for project `{project}` (task `{task_id}`).\n\n"
"Each plan revision is posted here as a comment. Approving a "
"revision by comment is equivalent to approving it in Matrix; "
"the first valid approval on either channel wins."
),
)
def post_plan(self, issue: int, gate: str, rev: int, plan: str) -> PlanPost:
tag = f"rev-{rev}"
body = "\n".join(
[
f"## Plan for `{gate}` ({tag})",
"",
plan,
"",
"---",
APPROVAL_HINT.format(tag=tag),
]
)
comment = self._client.post_comment(self._repo, issue, body)
return PlanPost(issue=issue, comment_id=comment.id)
def echo(self, issue: int, text: str) -> None:
"""Mirror an outcome decided on another channel, so the Gitea record
does not silently disagree with Matrix."""
try:
self._client.post_comment(self._repo, issue, text)
except GiteaError:
pass
def comments_since(self, issue: int, after_id: int) -> list[Comment]:
return [
c for c in self._client.list_comments(self._repo, issue) if c.id > after_id
]
+384 -32
View File
@@ -8,12 +8,17 @@ arrives over transport. Every position change is checkpointed through the state
store before the loop continues, so a restart lands mid-gate, not mid-air. store before the loop continues, so a restart lands mid-gate, not mid-air.
""" """
import threading
import uuid import uuid
from collections.abc import Iterable, Mapping from collections.abc import Iterable, Mapping
from dataclasses import dataclass from dataclasses import dataclass
from typing import Any, Protocol from typing import Any, Protocol
from nisse_dev.canonical import GateOpened, PlanSubmitted, UserMessage
from nisse_dev.gates import Approved, Gate, Ignored, PlanRevision, Rejected from nisse_dev.gates import Approved, Gate, Ignored, PlanRevision, Rejected
from nisse_dev.git import GitError
from nisse_dev.gitea import GiteaError
from nisse_dev.llm import RoleMap
from nisse_dev.reconcile import Reconciler, reconcile_on_boot from nisse_dev.reconcile import Reconciler, reconcile_on_boot
from nisse_dev.state import StateStore from nisse_dev.state import StateStore
from nisse_dev.transport import InboundMessage, Transport from nisse_dev.transport import InboundMessage, Transport
@@ -24,6 +29,7 @@ AWAIT_ARCH_SIGNOFF = "AWAIT_ARCH_SIGNOFF"
MILESTONE_PLANNING = "MILESTONE_PLANNING" MILESTONE_PLANNING = "MILESTONE_PLANNING"
AWAIT_MILESTONE_SIGNOFF = "AWAIT_MILESTONE_SIGNOFF" AWAIT_MILESTONE_SIGNOFF = "AWAIT_MILESTONE_SIGNOFF"
EXECUTING = "EXECUTING" EXECUTING = "EXECUTING"
PR_OPEN = "PR_OPEN"
AWAIT_INPUT = "AWAIT_INPUT" AWAIT_INPUT = "AWAIT_INPUT"
TRANSITIONS = { TRANSITIONS = {
@@ -34,6 +40,12 @@ TRANSITIONS = {
(MILESTONE_PLANNING, AWAIT_MILESTONE_SIGNOFF), (MILESTONE_PLANNING, AWAIT_MILESTONE_SIGNOFF),
(AWAIT_MILESTONE_SIGNOFF, MILESTONE_PLANNING), (AWAIT_MILESTONE_SIGNOFF, MILESTONE_PLANNING),
(AWAIT_MILESTONE_SIGNOFF, EXECUTING), (AWAIT_MILESTONE_SIGNOFF, EXECUTING),
(EXECUTING, PR_OPEN),
# PR acceptance is the milestone's final gate (B1): a merged PR is the
# durable record, so completion returns to planning or to idle rather
# than passing through a separate FINISHED state.
(PR_OPEN, MILESTONE_PLANNING),
(PR_OPEN, IDLE),
} }
@@ -41,6 +53,12 @@ class TransitionError(RuntimeError):
pass pass
class PlanningError(RuntimeError):
"""A planning turn produced nothing usable. Part of the PlanningAgent
contract: the orchestrator gates to AWAIT_INPUT rather than inventing a
plan or retrying forever."""
@dataclass(frozen=True) @dataclass(frozen=True)
class SubmitPlan: class SubmitPlan:
content: str content: str
@@ -51,6 +69,42 @@ class AskHuman:
question: str question: str
class MilestoneExecutor(Protocol):
"""What the orchestrator needs from an executor loop; the concrete one
lives in nisse_dev.executor (imported there, not here, to keep the
dependency pointing one way)."""
def run(self, milestone: int) -> Any: ...
class MilestonePublisher(Protocol):
"""What the orchestrator needs to publish a milestone; the concrete one
lives in nisse_dev.publish."""
def reconcilers(self) -> Mapping[str, Reconciler]: ...
def publish(
self, *, task_id: str, milestone: int, project: str, plan: str
) -> Any: ...
def push_followup(
self, *, task_id: str, milestone: int, message: str
) -> str | None: ...
def comment(self, index: int, body: str) -> None: ...
class PlanBoard(Protocol):
"""The Gitea side of plan gating; the concrete one is in
nisse_dev.approvals."""
def open_project_issue(self, project: str, task_id: str) -> int: ...
def post_plan(self, issue: int, gate: str, rev: int, plan: str) -> Any: ...
def echo(self, issue: int, text: str) -> None: ...
class PlanningAgent(Protocol): class PlanningAgent(Protocol):
def plan( def plan(
self, self,
@@ -83,9 +137,17 @@ def _fresh_data() -> dict[str, Any]:
return { return {
"revs": {}, "revs": {},
"gate": None, "gate": None,
"question": None, # What an operator reply in AWAIT_INPUT should resume. Every path into
# AWAIT_INPUT must set this, or the reply has nowhere to go.
"pending": None,
"milestone": 1, "milestone": 1,
"task_id": None, "task_id": None,
"roles": None, # A11 mapping, pinned at task start
"pr": None, # the open PR for the current milestone, once published
# Pointer to the Gitea plan record (Part H): the state store holds
# only this, so gate state can be re-derived from Gitea after a
# state-volume loss.
"board": None,
} }
@@ -96,17 +158,38 @@ class Orchestrator:
transport: Transport, transport: Transport,
agent: PlanningAgent, agent: PlanningAgent,
operators: Iterable[str], operators: Iterable[str],
roles: RoleMap | None = None,
executor: "MilestoneExecutor | None" = None,
publisher: "MilestonePublisher | None" = None,
gitea_operators: Iterable[str] = (),
board: "PlanBoard | None" = None,
) -> None: ) -> None:
self._store = store self._store = store
self._transport = transport self._transport = transport
self._agent = agent self._agent = agent
self._roles = roles
self._executor = executor
self._publisher = publisher
self._operators = set(operators) self._operators = set(operators)
# Per-channel allowlists (A4): a Matrix MXID and a Gitea login
# are different names for the same person, and neither implies
# the other.
self._gitea_operators = set(gitea_operators)
self._board = board
# Transport and the comment poller deliver on separate threads;
# state transitions must not interleave.
self._lock = threading.RLock()
self._data = _fresh_data() self._data = _fresh_data()
self._project: str | None = None self._project: str | None = None
transport.on_message(self._handle) transport.on_message(self._handle)
def start(self, reconcilers: Mapping[str, Reconciler] | None = None) -> None: def start(self, reconcilers: Mapping[str, Reconciler] | None = None) -> None:
report = reconcile_on_boot(self._store, reconcilers or {}) # The publisher owns the external-truth query for its own intents, so
# a configured publisher always contributes its reconciler (A3).
registered: dict[str, Reconciler] = dict(reconcilers or {})
if self._publisher is not None:
registered = {**self._publisher.reconcilers(), **registered}
report = reconcile_on_boot(self._store, registered)
state = self._store.get_state() state = self._store.get_state()
if state is None: if state is None:
self._transition(IDLE, cause="boot: fresh state") self._transition(IDLE, cause="boot: fresh state")
@@ -117,11 +200,14 @@ class Orchestrator:
self._store.oplog.task_id = self._data.get("task_id") self._store.oplog.task_id = self._data.get("task_id")
gate = self._data.get("gate") gate = self._data.get("gate")
if report.action == "await_input": if report.action == "await_input":
self._transition(AWAIT_INPUT, cause="boot: unresolved intents") self._await_input(
self._transport.ask_question( cause="boot: unresolved intents",
"restart found unresolved work: " prompt=(
+ "; ".join(report.notes) "restart found unresolved work: "
+ " — how should I proceed?" + "; ".join(report.notes)
+ " — how should I proceed?"
),
pending={"kind": "boot"},
) )
elif state.state == IDLE or ( elif state.state == IDLE or (
state.state in (AWAIT_ARCH_SIGNOFF, AWAIT_MILESTONE_SIGNOFF) and gate state.state in (AWAIT_ARCH_SIGNOFF, AWAIT_MILESTONE_SIGNOFF) and gate
@@ -136,14 +222,25 @@ class Orchestrator:
else: else:
# Interrupted mid-turn (planning, executing, unanswered question): # Interrupted mid-turn (planning, executing, unanswered question):
# never blind-resume (A3.6). # never blind-resume (A3.6).
self._transition(AWAIT_INPUT, cause=f"boot: restarted in {state.state}") self._await_input(
self._transport.ask_question( cause=f"boot: restarted in {state.state}",
f"restarted mid-{state.state}; continue or re-plan?" prompt=f"restarted mid-{state.state}; continue or re-plan?",
pending={"kind": "boot"},
) )
def _handle(self, msg: InboundMessage) -> None: def _handle(self, msg: InboundMessage) -> None:
with self._lock:
self._handle_locked(msg)
def _handle_locked(self, msg: InboundMessage) -> None:
if msg.sender not in self._operators: if msg.sender not in self._operators:
return return
# Every operator message enters the canonical transcript, so planning
# turns see feedback and answers as conversation rather than as
# out-of-band parameters (A10).
self._store.append_event(
UserMessage(text=msg.text.strip(), source=f"transport:{msg.sender}")
)
state = self._store.get_state() state = self._store.get_state()
current = state.state if state else IDLE current = state.state if state else IDLE
text = msg.text.strip() text = msg.text.strip()
@@ -156,6 +253,8 @@ class Orchestrator:
self._transport.send_message("no active project; say 'use <project>'") self._transport.send_message("no active project; say 'use <project>'")
elif current in (AWAIT_ARCH_SIGNOFF, AWAIT_MILESTONE_SIGNOFF): elif current in (AWAIT_ARCH_SIGNOFF, AWAIT_MILESTONE_SIGNOFF):
self._handle_gate_reply(msg, text) self._handle_gate_reply(msg, text)
elif current == PR_OPEN:
self._handle_pr_open(text)
elif current == AWAIT_INPUT: elif current == AWAIT_INPUT:
self._handle_answer(text) self._handle_answer(text)
else: else:
@@ -172,9 +271,25 @@ class Orchestrator:
def _begin_project(self, project: str) -> None: def _begin_project(self, project: str) -> None:
task_id = f"{project}-{uuid.uuid4().hex[:8]}" task_id = f"{project}-{uuid.uuid4().hex[:8]}"
self._data = {**_fresh_data(), "task_id": task_id} # Pin the role mapping now (A11): a mid-task config change must not
# split one task across two planners.
pinned = self._roles.pinned() if self._roles is not None else None
self._data = {**_fresh_data(), "task_id": task_id, "roles": pinned}
self._project = project self._project = project
self._store.oplog.task_id = task_id self._store.oplog.task_id = task_id
if self._board is not None:
try:
self._data["board"] = {
"issue": self._board.open_project_issue(project, task_id),
"last_comment": 0,
}
except GiteaError as exc:
# Gitea being down must not block planning; Matrix remains a
# complete approval channel on its own.
self._transport.send_message(
f"note: could not open the Gitea plan issue ({exc}); "
"approvals will be Matrix-only for this task"
)
self._transition(PLANNING_ARCH, cause=f"operator command: use {project}") self._transition(PLANNING_ARCH, cause=f"operator command: use {project}")
self._run_planning_turn("arch") self._run_planning_turn("arch")
@@ -182,15 +297,23 @@ class Orchestrator:
self, gate_name: str, feedback: str | None = None, answer: str | None = None self, gate_name: str, feedback: str | None = None, answer: str | None = None
) -> None: ) -> None:
assert self._project is not None assert self._project is not None
action = self._agent.plan( try:
gate_name, self._project, feedback=feedback, answer=answer action = self._agent.plan(
) gate_name, self._project, feedback=feedback, answer=answer
)
except PlanningError as exc:
self._await_input(
cause=f"planning failed: {exc}",
prompt=f"planning for {gate_name} failed ({exc}); how should I proceed?",
pending={"kind": "planning_failed", "gate": gate_name},
)
return
if isinstance(action, AskHuman): if isinstance(action, AskHuman):
self._data["question"] = {"text": action.question, "gate": gate_name} self._await_input(
self._transition( cause=f"agent question during {gate_name} planning",
AWAIT_INPUT, cause=f"agent question during {gate_name} planning" prompt=action.question,
pending={"kind": "question", "gate": gate_name},
) )
self._transport.ask_question(action.question)
return return
rev = self._data["revs"].get(gate_name, 0) + 1 rev = self._data["revs"].get(gate_name, 0) + 1
self._data["revs"][gate_name] = rev self._data["revs"][gate_name] = rev
@@ -203,11 +326,19 @@ class Orchestrator:
await_state = ( await_state = (
AWAIT_ARCH_SIGNOFF if gate_name == "arch" else AWAIT_MILESTONE_SIGNOFF AWAIT_ARCH_SIGNOFF if gate_name == "arch" else AWAIT_MILESTONE_SIGNOFF
) )
roles = self._data.get("roles") or {}
self._store.append_event(
PlanSubmitted(plan=action.content, model=roles.get("planner", "stub"))
)
self._store.append_event(
GateOpened(gate=gate_name, reason=f"plan submitted (rev-{rev})")
)
self._transition(await_state, cause=f"plan submitted ({gate_name} rev-{rev})") self._transition(await_state, cause=f"plan submitted ({gate_name} rev-{rev})")
self._transport.send_approval_request( self._transport.send_approval_request(
f"{action.content}\n\nreply 'approve' to approve rev-{rev}; " f"{action.content}\n\nreply 'approve' to approve rev-{rev}; "
"any other reply is rejection feedback" "any other reply is rejection feedback"
) )
self._post_plan_to_board(gate_name, rev, action.content)
self._store.oplog.emit( self._store.oplog.emit(
"gate.open", "gate.open",
cause=f"plan submitted (rev-{rev})", cause=f"plan submitted (rev-{rev})",
@@ -217,6 +348,12 @@ class Orchestrator:
) )
def _handle_gate_reply(self, msg: InboundMessage, text: str) -> None: def _handle_gate_reply(self, msg: InboundMessage, text: str) -> None:
self._apply_gate_reply(sender=msg.sender, text=text, channel="transport")
def _apply_gate_reply(self, *, sender: str, text: str, channel: str) -> None:
"""Shared by every approval source (A4). The allowlist is per channel:
a Matrix MXID and a Gitea login are different names, and holding one
says nothing about holding the other."""
gate_data = self._data["gate"] gate_data = self._data["gate"]
name, rev = gate_data["name"], gate_data["rev"] name, rev = gate_data["name"], gate_data["rev"]
milestone = self._data["milestone"] milestone = self._data["milestone"]
@@ -225,16 +362,15 @@ class Orchestrator:
self._transition( self._transition(
EXECUTING, cause=f"operator command: start milestone {milestone}" EXECUTING, cause=f"operator command: start milestone {milestone}"
) )
self._transport.send_message(f"[stub] executing milestone {milestone}") self._execute_milestone(milestone)
else: else:
self._transport.send_message( self._transport.send_message(
f"plan approved; say 'start milestone {milestone}' to begin" f"plan approved; say 'start milestone {milestone}' to begin"
) )
return return
revision = PlanRevision(gate=name, rev=rev, content=gate_data["plan"]) revision = PlanRevision(gate=name, rev=rev, content=gate_data["plan"])
result = Gate(revision, self._operators).evaluate( allowlist = self._operators if channel == "transport" else self._gitea_operators
msg.sender, text, channel="transport" result = Gate(revision, allowlist).evaluate(sender, text, channel=channel)
)
if isinstance(result, Ignored): if isinstance(result, Ignored):
if result.notice: if result.notice:
self._transport.send_message(result.notice) self._transport.send_message(result.notice)
@@ -247,12 +383,19 @@ class Orchestrator:
rev=f"rev-{rev}", rev=f"rev-{rev}",
channel=result.channel, channel=result.channel,
) )
self._echo_outcome(
channel,
f"{name} rev-{rev} rejected by {result.approver}: {result.feedback}",
)
self._data["gate"] = None self._data["gate"] = None
back = PLANNING_ARCH if name == "arch" else MILESTONE_PLANNING back = PLANNING_ARCH if name == "arch" else MILESTONE_PLANNING
self._transition(back, cause=f"plan rejected ({name} rev-{rev})") self._transition(back, cause=f"plan rejected ({name} rev-{rev})")
self._run_planning_turn(name, feedback=result.feedback) self._run_planning_turn(name, feedback=result.feedback)
return return
assert isinstance(result, Approved) assert isinstance(result, Approved)
self._echo_outcome(
channel, f"{name} rev-{rev} approved by {result.approver} via {channel}"
)
self._store.oplog.emit( self._store.oplog.emit(
"gate.approve", "gate.approve",
cause=f"approved by {result.approver}", cause=f"approved by {result.approver}",
@@ -279,18 +422,227 @@ class Orchestrator:
f"milestone plan approved; say 'start milestone {milestone}' to begin" f"milestone plan approved; say 'start milestone {milestone}' to begin"
) )
def _handle_answer(self, text: str) -> None: def _execute_milestone(self, milestone: int) -> None:
question = self._data.get("question") if self._executor is None:
if question is None: self._transport.send_message(f"[stub] executing milestone {milestone}")
self._transport.send_message( return
"awaiting operator resolution; no pending question to answer" self._transport.send_message(f"executing milestone {milestone}")
outcome = self._executor.run(milestone)
if outcome.status == "completed":
self._transport.send_message(f"milestone {milestone} done: {outcome.note}")
self._publish_milestone(milestone)
return
self._await_input(
cause=f"milestone {milestone} {outcome.status}",
prompt=f"milestone {milestone} stopped ({outcome.status}): {outcome.note}",
pending={"kind": "milestone_stopped", "milestone": milestone},
)
def _echo_outcome(self, deciding_channel: str, text: str) -> None:
"""A gate satisfied on one channel is echoed to the other, so neither
record silently disagrees about what was approved (A4)."""
board_state = self._data.get("board")
if deciding_channel != "gitea" and self._board is not None and board_state:
self._board.echo(board_state["issue"], text)
elif deciding_channel == "gitea":
self._transport.send_message(text)
def _post_plan_to_board(self, gate_name: str, rev: int, plan: str) -> None:
board_state = self._data.get("board")
if self._board is None or not board_state:
return
try:
post = self._board.post_plan(board_state["issue"], gate_name, rev, plan)
except GiteaError:
# Matrix already carries the plan; a Gitea outage degrades the
# durability of the record, not the ability to approve.
return
board_state["comment_id"] = post.comment_id
def handle_plan_comment(self, comment: Any) -> None:
"""A comment on the project issue may satisfy an open plan gate (A4).
Approvals bind to a revision here exactly as over transport, and the
first valid approval on either channel wins.
"""
with self._lock:
self._handle_plan_comment_locked(comment)
def _handle_plan_comment_locked(self, comment: Any) -> None:
if comment.author not in self._gitea_operators:
return
state = self._store.get_state()
if state is None or state.state not in (
AWAIT_ARCH_SIGNOFF,
AWAIT_MILESTONE_SIGNOFF,
):
return
self._store.append_event(
UserMessage(text=comment.body, source=f"gitea:comment:{comment.id}")
)
self._apply_gate_reply(
sender=comment.author, text=comment.body.strip(), channel="gitea"
)
def handle_review_comment(self, comment: Any) -> None:
"""React to a PR review comment (B2). Called from the poller thread.
Comment authors are checked against the Gitea operator allowlist:
a PR comment is an instruction channel into an agent that writes
code, so it is operator-scoped like any other (A4, and the same
reasoning C4 applies to tickets).
"""
with self._lock:
self._handle_review_comment_locked(comment)
def _handle_review_comment_locked(self, comment: Any) -> None:
if comment.author not in self._gitea_operators:
self._store.oplog.emit(
"gitea.comment_ignored",
cause=f"author {comment.author!r} not an operator",
comment_id=comment.id,
) )
return return
self._data["question"] = None state = self._store.get_state()
gate_name = question["gate"] if state is None or state.state != PR_OPEN:
back = PLANNING_ARCH if gate_name == "arch" else MILESTONE_PLANNING self._store.oplog.emit(
self._transition(back, cause="operator answered question") "gitea.comment_ignored",
self._run_planning_turn(gate_name, answer=text) cause=f"not in PR_OPEN (state={state.state if state else None})",
comment_id=comment.id,
)
return
if self._executor is None or self._publisher is None:
return
pr = self._data.get("pr") or {}
milestone = int(pr.get("milestone", self._data.get("milestone", 1)))
# The comment enters the transcript, so the executor sees review
# feedback as conversation rather than as an out-of-band instruction.
self._store.append_event(
UserMessage(text=comment.body, source=f"gitea:comment:{comment.id}")
)
outcome = self._executor.run(milestone)
if outcome.status != "completed":
self._await_input(
cause=f"review response {outcome.status}",
prompt=(
f"could not address the review comment ({outcome.status}): "
f"{outcome.note}"
),
pending={"kind": "milestone_stopped", "milestone": milestone},
)
return
try:
sha = self._publisher.push_followup(
task_id=str(self._data.get("task_id")),
milestone=milestone,
message=f"Address review comment {comment.id}",
)
except (GiteaError, GitError) as exc:
self._transport.send_message(f"could not push the fix: {exc}")
return
note = (
f"pushed {sha[:8]} addressing the review comment"
if sha
else "no code change was needed for that comment"
)
self._transport.send_message(note)
if pr.get("number"):
self._publisher.comment(int(pr["number"]), note)
def _publish_milestone(self, milestone: int) -> None:
if self._publisher is None:
return
gate = self._data.get("gate") or {}
try:
result = self._publisher.publish(
task_id=str(self._data.get("task_id")),
milestone=milestone,
project=str(self._project),
plan=str(gate.get("plan", "")),
)
except (GiteaError, GitError) as exc:
# The intent log already records what was in flight, so boot
# reconciliation can settle it against Gitea. Gate rather than
# retry blindly into a possible duplicate.
self._await_input(
cause=f"publishing milestone {milestone} failed: {exc}",
prompt=f"could not publish milestone {milestone}: {exc}",
pending={"kind": "milestone_stopped", "milestone": milestone},
)
return
if result.pull_request is None:
self._transport.send_message(result.note)
return
self._data["pr"] = {
"number": result.pull_request.number,
"url": result.pull_request.url,
"milestone": milestone,
}
self._transition(PR_OPEN, cause=f"milestone {milestone} published")
self._transport.send_message(
f"{result.note} — review it, then say 'next milestone' or "
"'project complete'"
)
def _handle_pr_open(self, text: str) -> None:
if text == "next milestone":
self._data["milestone"] = int(self._data.get("milestone", 1)) + 1
self._data["gate"] = None
self._data["pr"] = None
self._transition(MILESTONE_PLANNING, cause="operator: next milestone")
self._run_planning_turn(f"milestone-{self._data['milestone']}")
elif text == "project complete":
self._data = {**_fresh_data()}
self._project = None
self._store.oplog.task_id = None
self._transition(IDLE, cause="operator: project complete")
self._transport.send_message(
"project closed; say 'use <project>' to start another"
)
else:
pr = self._data.get("pr") or {}
self._transport.send_message(
f"PR #{pr.get('number')} is open; say 'next milestone' or "
"'project complete'"
)
def _await_input(self, *, cause: str, prompt: str, pending: dict[str, Any]) -> None:
self._data["pending"] = pending
self._transition(AWAIT_INPUT, cause=cause)
self._transport.ask_question(prompt)
def _handle_answer(self, text: str) -> None:
pending = self._data.get("pending")
if pending is None:
# Reached only if state was forced externally; say what works
# rather than dead-ending the operator.
self._transport.send_message(
"waiting for direction, but I have no pending item to resume; "
"try 'status', or 'use <project>' to start over"
)
return
self._data["pending"] = None
match pending.get("kind"):
case "question" | "planning_failed":
gate_name = pending["gate"]
back = PLANNING_ARCH if gate_name == "arch" else MILESTONE_PLANNING
self._transition(back, cause="operator answered; resuming planning")
self._run_planning_turn(gate_name, answer=text)
case "milestone_stopped":
milestone = pending["milestone"]
self._transition(
AWAIT_MILESTONE_SIGNOFF,
cause="operator answered; plan still approved",
)
self._transport.send_message(
f"noted. the milestone {milestone} plan is still approved — "
f"say 'start milestone {milestone}' to run it again"
)
case _:
self._transport.send_message(
"noted. say 'status' for where things stand"
)
def _transition(self, new_state: str, *, cause: str) -> None: def _transition(self, new_state: str, *, cause: str) -> None:
state = self._store.get_state() state = self._store.get_state()
+363
View File
@@ -0,0 +1,363 @@
"""Delegation harness (spec A13).
Escalation is delegation, not consultation: a frontier model receives a scoped
sub-task, works autonomously with full tool access, and hands back. The local
model does not interpret or re-derive its reasoning — the handback note plus
the resulting file state is the entire interface back.
Three properties do the work here:
**Isolation.** The delegation runs on its own branch, taken at start. Abort
discards the branch, which gives a delegation the same atomicity that boot
reconciliation gives an interrupted state transition.
**Caps that are hard.** Turns, tokens, and wall clock. A breach is a defined
failure return, not a nudge: the local loop gates to the human rather than
resuming on top of a half-edited tree.
**Checkpointing as it happens.** Every tool call, result, and assistant turn
is written to the canonical log when it occurs, never batched at handback.
That is what makes a delegation resumable after a crash (A3).
Scope is bounded by construction: a delegation is handed a sub-task the
approved plan already contains. It is smart, not authorised to widen the work.
"""
import time
from collections.abc import Callable
from dataclasses import dataclass, field
from typing import Any, Literal
from nisse_dev.canonical import (
AssistantText,
DelegationCompleted,
DelegationStarted,
ToolCall,
ToolResult,
)
from nisse_dev.git import GitError, GitRepo
from nisse_dev.llm import LLMClient, LLMError
from nisse_dev.state import StateStore
from nisse_dev.tools import EXECUTOR_TOOLS, MUTATING_TOOLS, Workspace
DELEGATION_INTENT = "delegation"
SYSTEM_PROMPT = """You are a delegate working on one scoped sub-task for
nisse-dev. You have full tool access and are expected to finish the sub-task
yourself, including running the tests.
Stay inside the sub-task you were given. It comes from a plan a human already
approved; widening it is not yours to do. If the sub-task turns out to be
wrong or impossible, say so in the handback rather than substituting your own.
When you are done, call hand_back exactly once. Write it for a materially
weaker model that cannot see your reasoning and will act on your note alone:
- what changed
- what is now true that was not before
- what to do next
Do not call hand_back until the work is actually done."""
HAND_BACK_TOOL: dict[str, Any] = {
"type": "function",
"function": {
"name": "hand_back",
"description": (
"End the delegation and report. Call exactly once, when the "
"sub-task is complete or you have determined it cannot be."
),
"parameters": {
"type": "object",
"properties": {
"changed": {
"type": "string",
"description": "What you changed, concretely.",
},
"now_true": {
"type": "string",
"description": "What is now true that was not before.",
},
"next": {
"type": "string",
"description": "What the caller should do next.",
},
"succeeded": {
"type": "boolean",
"description": "Whether the sub-task was completed.",
},
},
"required": ["changed", "now_true", "next", "succeeded"],
},
},
}
DELEGATE_TOOLS = [*EXECUTOR_TOOLS, HAND_BACK_TOOL]
@dataclass(frozen=True)
class Caps:
turns: int = 30
tokens: int = 200_000
wall_clock_s: float = 900.0
def as_payload(self) -> dict[str, Any]:
return {
"turns": self.turns,
"tokens": self.tokens,
"wall_clock_s": self.wall_clock_s,
}
@dataclass(frozen=True)
class Handback:
"""The entire interface back to the local loop, alongside file state."""
changed: str
now_true: str
next: str
succeeded: bool
def render(self) -> str:
return (
f"What changed: {self.changed}\n"
f"What is now true: {self.now_true}\n"
f"What to do next: {self.next}"
)
@dataclass
class DelegationResult:
status: Literal["handback", "capped", "failed"]
note: str
handback: Handback | None = None
branch: str | None = None
artifacts: list[str] = field(default_factory=list)
tokens: int = 0
duration_s: float = 0.0
@property
def usable(self) -> bool:
"""Only a successful handback may be built upon. A cap breach or a
failure leaves a half-edited tree, which A13 says must gate rather
than be resumed."""
return self.status == "handback" and self.handback is not None
class Delegation:
def __init__(
self,
client: LLMClient,
store: StateStore,
workspace: Workspace,
git_repo: GitRepo,
*,
model: str,
caps: Caps | None = None,
now: Callable[[], float] = time.monotonic,
) -> None:
self._client = client
self._store = store
self._workspace = workspace
self._git = git_repo
self._model = model
self._caps = caps or Caps()
self._now = now
def run(
self, *, brief: str, branch: str, role: str = "delegate"
) -> DelegationResult:
started_at = self._now()
base_branch = self._git.current_branch()
try:
# Commit whatever the caller had in flight onto their own branch
# first. Otherwise it would ride along onto the delegation branch
# and be destroyed by an abort — discarding work nobody delegated.
self._git.commit_all("Checkpoint before delegating")
self._git.checkout_new_branch(branch)
except GitError as exc:
return DelegationResult("failed", f"could not branch for delegation: {exc}")
self._store.append_event(
DelegationStarted(
brief=brief,
role=role,
model=self._model,
caps=self._caps.as_payload(),
)
)
result = self._loop(brief, branch, started_at)
result.duration_s = self._now() - started_at
self._store.append_event(
DelegationCompleted(
handback=result.handback.render() if result.handback else result.note,
artifacts=result.artifacts,
tokens=result.tokens,
duration_s=result.duration_s,
exit_status=result.status,
)
)
if not result.usable:
# Abort discards the branch: the caller must never resume on top
# of a partially-edited tree (A13).
self._abandon(branch, base_branch)
return result
def _loop(self, brief: str, branch: str, started_at: float) -> DelegationResult:
system = (
f"{SYSTEM_PROMPT}\n\n{self._workspace.describe()}"
f"\n\nYour sub-task:\n{brief}"
)
touched: set[str] = set()
did_work = False
for turn in range(self._caps.turns):
elapsed = self._now() - started_at
if elapsed >= self._caps.wall_clock_s:
return DelegationResult(
"capped",
f"wall-clock cap of {self._caps.wall_clock_s}s reached "
f"after {turn} turns",
branch=branch,
artifacts=sorted(touched),
)
try:
produced = self._client.complete(
self._store.read_events(),
tools=DELEGATE_TOOLS,
model=self._model,
system=system,
)
except LLMError as exc:
return DelegationResult(
"failed", f"delegate call failed: {exc}", branch=branch
)
calls: list[ToolCall] = []
for event in produced:
# Checkpoint as it arrives, never batched at handback (A13):
# this is what makes the delegation resumable after a crash.
self._store.append_event(event)
if isinstance(event, ToolCall):
calls.append(event)
if not calls:
# A delegate that stops without handing back has not reported,
# and its tree state is unknown. Treat it as a failure rather
# than inventing a handback on its behalf.
if any(isinstance(e, AssistantText) for e in produced):
continue
return DelegationResult(
"failed",
"delegate produced neither a tool call nor text",
branch=branch,
artifacts=sorted(touched),
)
for call in calls:
if call.tool == "hand_back":
handback = _parse_handback(call.arguments)
if handback is not None and handback.succeeded and not did_work:
# Success claimed with nothing changed. The handback
# and the file state are the whole interface back, so
# an unbacked claim would hand the caller a note about
# work that does not exist.
self._store.append_event(
ToolResult(
call_id=call.call_id,
result=(
"you have not changed anything yet, so the "
"sub-task cannot be complete; do the work, "
"or hand back with succeeded=false and say "
"why"
),
success=False,
)
)
continue
if handback is None:
self._store.append_event(
ToolResult(
call_id=call.call_id,
result=(
"hand_back needs changed, now_true, next and "
"succeeded; the delegation is still open"
),
success=False,
)
)
continue
self._store.append_event(
ToolResult(
call_id=call.call_id,
result="handback accepted; delegation closed",
success=True,
)
)
self._commit_turn(turn)
return DelegationResult(
"handback" if handback.succeeded else "failed",
handback.next if handback.succeeded else handback.changed,
handback=handback if handback.succeeded else None,
branch=branch,
artifacts=sorted(touched),
)
outcome = self._workspace.dispatch(call.tool, call.arguments)
if call.tool in MUTATING_TOOLS and outcome.success:
did_work = True
path = call.arguments.get("path")
if isinstance(path, str):
touched.add(path)
self._store.append_event(
ToolResult(
call_id=call.call_id,
result=outcome.result,
success=outcome.success,
)
)
# Commit each turn's work onto the delegation branch, so that
# discarding the branch really does discard the work, and so a
# crash leaves the branch at a known commit (A3).
self._commit_turn(turn)
return DelegationResult(
"capped",
f"turn cap of {self._caps.turns} reached without a handback",
branch=branch,
artifacts=sorted(touched),
)
def _commit_turn(self, turn: int) -> None:
try:
self._git.commit_all(f"Delegation turn {turn + 1}")
except GitError:
# Losing a checkpoint commit does not invalidate the work; the
# canonical log still records what happened.
pass
def _abandon(self, branch: str, base_branch: str) -> None:
try:
self._git.discard_branch(branch, back_to=base_branch)
except GitError:
# The branch is already recorded in the delegation events, so a
# failed cleanup leaves evidence rather than a silent mess.
pass
def _parse_handback(arguments: dict[str, Any]) -> Handback | None:
"""Validated locally rather than trusted: A14's rule for prompt-level
structured output applies to the handback too."""
required = ("changed", "now_true", "next")
if any(not str(arguments.get(key, "")).strip() for key in required):
return None
succeeded = arguments.get("succeeded")
if not isinstance(succeeded, bool):
return None
return Handback(
changed=str(arguments["changed"]).strip(),
now_true=str(arguments["now_true"]).strip(),
next=str(arguments["next"]).strip(),
succeeded=succeeded,
)
+136
View File
@@ -0,0 +1,136 @@
"""Local executor loop (spec A12 escalation triggers, A15 accommodations).
Runs the approved plan with the executor role — the cheap local model doing
mechanical work. Every tool call and result is checkpointed to the canonical
log as it happens (A13), so a crash mid-milestone leaves a resumable record.
Escalation is decided from outside the model (A12): consecutive command
failures and hard caps on turns, never the model's own claim that it is stuck
or finished-enough. Until delegation exists (step 7), an escalation gates to
the human with the failures attached rather than engaging a frontier model.
"""
from dataclasses import dataclass
from typing import Literal
from nisse_dev.canonical import ToolCall, ToolResult
from nisse_dev.llm import LLMClient, LLMError, RoleMap
from nisse_dev.state import StateStore
from nisse_dev.tools import EXECUTOR_TOOLS, MUTATING_TOOLS, Workspace
EXECUTOR_SYSTEM = """You are nisse-dev's executor, doing mechanical work.
Follow the approved plan above exactly. It was written to be executed
verbatim; do not redesign it, expand its scope, or skip steps.
Work by calling tools. Verify your work by running the project's tests.
When every step of the plan is done and the tests pass, reply with a short
plain-text summary and no tool call — that ends the milestone."""
DEFAULT_MAX_TURNS = 24
DEFAULT_FAILURE_LIMIT = 2
# A turn with no tool call means "the plan is done". Believing that before
# any real work has happened would record an empty milestone as a success, so
# the claim is challenged once first.
_NUDGE = (
"\n\nYou have not made any change yet, so the plan cannot be complete. "
"Carry out the next step of the plan with a tool call, or run the tests "
"to show the work is already done."
)
@dataclass(frozen=True)
class ExecutorOutcome:
status: Literal["completed", "capped", "escalate"]
note: str
class LLMExecutor:
def __init__(
self,
client: LLMClient,
store: StateStore,
roles: RoleMap,
workspace: Workspace,
*,
max_turns: int = DEFAULT_MAX_TURNS,
failure_limit: int = DEFAULT_FAILURE_LIMIT,
) -> None:
self._client = client
self._store = store
self._roles = roles
self._workspace = workspace
self._max_turns = max_turns
self._failure_limit = failure_limit
def run(self, milestone: int) -> ExecutorOutcome:
model = self._roles.resolve("executor")
system = (
f"{EXECUTOR_SYSTEM}\n\n{self._workspace.describe()}"
f"\n\nYou are executing milestone {milestone}."
)
consecutive_failures = 0
did_work = False
challenged = False
for _turn in range(self._max_turns):
try:
produced = self._client.complete(
self._store.read_events(),
tools=EXECUTOR_TOOLS,
model=model,
system=system,
)
except LLMError as exc:
return ExecutorOutcome("escalate", f"executor call failed: {exc}")
calls: list[ToolCall] = []
for event in produced:
self._store.append_event(event)
if isinstance(event, ToolCall):
calls.append(event)
if not calls:
if did_work:
return ExecutorOutcome(
"completed", "executor reported the plan done"
)
if not challenged:
# Claiming completion without having touched anything is
# not evidence of completion; ask once before believing it.
challenged = True
system += _NUDGE
continue
return ExecutorOutcome(
"escalate",
"executor claimed the milestone was done without making any change",
)
for call in calls:
if call.tool in MUTATING_TOOLS:
did_work = True
outcome = self._workspace.dispatch(call.tool, call.arguments)
self._store.append_event(
ToolResult(
call_id=call.call_id,
result=outcome.result,
success=outcome.success,
)
)
# Only commands count toward escalation: a failed read is the
# model looking around, a failed command is work not working.
if call.tool == "run_command":
consecutive_failures = (
0 if outcome.success else consecutive_failures + 1
)
if consecutive_failures >= self._failure_limit:
return ExecutorOutcome(
"escalate",
f"{consecutive_failures} consecutive command failures; "
f"last: {outcome.result[:500]}",
)
return ExecutorOutcome(
"capped", f"hit the {self._max_turns}-turn cap for milestone {milestone}"
)
+106
View File
@@ -0,0 +1,106 @@
"""Git operations for the agent's workspace (spec B2 outbound).
Push authentication uses the Gitea service-account token embedded in the
remote URL. The token is never written to disk in the repo config and is
scrubbed from anything this module raises or returns, so a failed push cannot
leak the credential into a transcript, a log line, or a PR comment.
"""
import subprocess
from dataclasses import dataclass
from pathlib import Path
TIMEOUT_S = 120
class GitError(RuntimeError):
pass
@dataclass(frozen=True)
class GitRepo:
root: Path
def _run(self, *args: str, secret: str | None = None) -> str:
try:
completed = subprocess.run(
["git", *args],
cwd=self.root,
capture_output=True,
text=True,
timeout=TIMEOUT_S,
check=False,
)
except (OSError, subprocess.TimeoutExpired) as exc:
raise GitError(f"git {args[0]} failed: {_scrub(str(exc), secret)}") from exc
if completed.returncode != 0:
raise GitError(
f"git {args[0]} failed: "
f"{_scrub(completed.stderr.strip() or completed.stdout.strip(), secret)}"
)
return completed.stdout.strip()
# -- inspection -------------------------------------------------------
def current_branch(self) -> str:
# --show-current rather than rev-parse: it answers correctly on a
# freshly initialised repo, where HEAD points at an unborn branch.
return self._run("branch", "--show-current")
def head_sha(self) -> str:
return self._run("rev-parse", "HEAD")
def has_changes(self) -> bool:
return bool(self._run("status", "--porcelain"))
# -- mutation ---------------------------------------------------------
def init(self, *, default_branch: str = "main") -> None:
self.root.mkdir(parents=True, exist_ok=True)
self._run("init", "-q", "-b", default_branch)
def configure_identity(self, name: str, email: str) -> None:
self._run("config", "user.name", name)
self._run("config", "user.email", email)
def checkout_new_branch(self, name: str) -> None:
self._run("checkout", "-q", "-b", name)
def commit_all(self, message: str) -> str | None:
"""Returns the new commit sha, or None when there was nothing to
commit — an empty milestone is a fact to report, not an error."""
if not self.has_changes():
return None
self._run("add", "-A")
self._run("commit", "-q", "-m", message)
return self.head_sha()
def push(self, remote_url: str, branch: str, *, token: str | None = None) -> None:
# The URL is passed per-invocation rather than stored as a remote, so
# the token never lands in .git/config.
self._run("push", "-q", remote_url, f"HEAD:refs/heads/{branch}", secret=token)
def discard_branch(self, branch: str, *, back_to: str) -> None:
"""Throw away a branch and everything on it.
Used when a delegation aborts: A13 gives a delegation the same
atomicity that reconciliation gives an interrupted transition, and
half-finished work must not survive to be built upon.
"""
self._run("checkout", "-q", "--force", back_to)
self._run("branch", "-q", "-D", branch)
def delete_remote_branch(
self, remote_url: str, branch: str, *, token: str | None = None
) -> None:
self._run("push", "-q", remote_url, f":refs/heads/{branch}", secret=token)
def authenticated_url(base_url: str, repo: str, user: str, token: str) -> str:
host = base_url.split("://", 1)[-1].rstrip("/")
scheme = base_url.split("://", 1)[0] if "://" in base_url else "https"
return f"{scheme}://{user}:{token}@{host}/{repo}.git"
def _scrub(text: str, secret: str | None) -> str:
return text.replace(secret, "***") if secret else text
+193
View File
@@ -0,0 +1,193 @@
"""Gitea client (spec B2).
Outbound: open PRs and post comments through a dedicated service account with
a scoped token. Inbound is deliberately *polled* rather than webhook-driven,
so the agent VM stays outbound-only (B3), and intake is structured JSON —
never routed through a chat bridge.
``find_pull_request`` is the external-truth query that makes opening a PR
idempotent across a crash: boot reconciliation (A3) asks Gitea whether the
effect already happened rather than repeating it.
"""
import os
from collections.abc import Mapping
from dataclasses import dataclass
from typing import Any
import httpx
from nisse_dev.reconcile import Reconciler
from nisse_dev.state import Intent
PR_PAGE_SIZE = 50
MAX_PR_PAGES = 40 # 2000 PRs; beyond that, say so rather than guess
class GiteaError(RuntimeError):
pass
@dataclass(frozen=True)
class PullRequest:
number: int
state: str
head: str
url: str
@dataclass(frozen=True)
class Comment:
id: int
author: str
body: str
created_at: str
class GiteaClient:
def __init__(
self,
base_url: str,
token: str,
*,
timeout_s: float = 30.0,
transport: httpx.BaseTransport | None = None,
) -> None:
self._http = httpx.Client(
base_url=f"{base_url.rstrip('/')}/api/v1",
headers={"Authorization": f"token {token}"},
timeout=timeout_s,
transport=transport,
)
@classmethod
def from_env(cls, env: Mapping[str, str] = os.environ) -> "GiteaClient":
base_url = env.get("NISSE_GITEA_URL")
token = env.get("NISSE_GITEA_TOKEN")
if not base_url or not token:
raise GiteaError("set NISSE_GITEA_URL and NISSE_GITEA_TOKEN")
return cls(base_url, token)
def _request(self, method: str, path: str, **kwargs: Any) -> Any:
try:
response = self._http.request(method, path, **kwargs)
except httpx.HTTPError as exc:
raise GiteaError(f"gitea request failed: {exc}") from exc
if response.status_code >= 400:
raise GiteaError(
f"gitea {method} {path} returned {response.status_code}: "
f"{response.text[:300]}"
)
return response.json() if response.content else None
def whoami(self) -> str:
return str(self._request("GET", "/user")["login"])
def open_pull_request(
self, repo: str, *, head: str, base: str, title: str, body: str
) -> PullRequest:
data = self._request(
"POST",
f"/repos/{repo}/pulls",
json={"head": head, "base": base, "title": title, "body": body},
)
return _pull_request(data)
def find_pull_request(self, repo: str, head: str) -> PullRequest | None:
"""External truth for A3: did the PR for this branch already get
opened? Includes closed PRs — a crash after opening must not be
answered with "no PR is open, so open another".
Pages to the end rather than sampling the first page: a "not found"
that only means "not in the newest 50" would licence a duplicate PR.
If the history is longer than this walk, raise instead of guessing —
the reconciler turns that into "unverifiable" and gates to the human.
"""
for page in range(1, MAX_PR_PAGES + 1):
data = self._request(
"GET",
f"/repos/{repo}/pulls",
params={"state": "all", "limit": PR_PAGE_SIZE, "page": page},
)
items = data or []
for item in items:
if (item.get("head") or {}).get("ref") == head:
return _pull_request(item)
if len(items) < PR_PAGE_SIZE:
return None # reached the end: definitively absent
raise GiteaError(
f"searched {MAX_PR_PAGES * PR_PAGE_SIZE} pull requests in {repo} "
f"without reaching the end; cannot confirm whether {head!r} was "
"already opened"
)
def list_comments(
self, repo: str, index: int, since: str | None = None
) -> list[Comment]:
params: dict[str, str] = {}
if since is not None:
params["since"] = since
data = self._request(
"GET", f"/repos/{repo}/issues/{index}/comments", params=params
)
return [
Comment(
id=item["id"],
author=(item.get("user") or {}).get("login", ""),
body=item.get("body", ""),
created_at=item.get("created_at", ""),
)
for item in data or []
]
def post_comment(self, repo: str, index: int, body: str) -> Comment:
data = self._request(
"POST", f"/repos/{repo}/issues/{index}/comments", json={"body": body}
)
return Comment(
id=data["id"],
author=(data.get("user") or {}).get("login", ""),
body=data.get("body", ""),
created_at=data.get("created_at", ""),
)
def create_issue(self, repo: str, title: str, body: str) -> int:
data = self._request(
"POST", f"/repos/{repo}/issues", json={"title": title, "body": body}
)
return int(data["number"])
def close_pull_request(self, repo: str, index: int) -> None:
self._request("PATCH", f"/repos/{repo}/pulls/{index}", json={"state": "closed"})
def close(self) -> None:
self._http.close()
def _pull_request(data: dict[str, Any]) -> PullRequest:
return PullRequest(
number=data["number"],
state=data.get("state", ""),
head=(data.get("head") or {}).get("ref", ""),
url=data.get("html_url", ""),
)
def pull_request_reconciler(client: GiteaClient, repo: str) -> Reconciler:
"""Reconciler for an ``open_pr`` intent (A3 step 4).
An unreachable Gitea yields None — unverifiable, so the agent gates to
the human rather than risking a duplicate PR. That is the conservative
direction: a delayed decision costs nothing, a repeated side effect does.
"""
def check(intent: Intent) -> bool | None:
branch = intent.payload.get("branch")
if not branch:
return None
try:
return client.find_pull_request(repo, branch) is not None
except GiteaError:
return None
return check
+139 -5
View File
@@ -12,6 +12,7 @@ config change cannot split a task across two planners.
import json import json
import os import os
import time
from collections.abc import Mapping from collections.abc import Mapping
from dataclasses import asdict, dataclass from dataclasses import asdict, dataclass
from typing import Any, ClassVar, Protocol from typing import Any, ClassVar, Protocol
@@ -26,12 +27,21 @@ from nisse_dev.canonical import (
mint_call_id, mint_call_id,
) )
from nisse_dev.render import OpenAIChatRenderer from nisse_dev.render import OpenAIChatRenderer
from nisse_dev.render_anthropic import AnthropicMessagesRenderer
class LLMError(RuntimeError): class LLMError(RuntimeError):
pass pass
# A rate limit is a transient condition, not a bad request. Losing a
# ten-turn delegation to one wastes far more tokens than waiting out the
# window, so retry once before surfacing it (A13's caps still bound the
# total).
RATE_LIMIT_STATUS = 429
RATE_LIMIT_RETRY_S = 20.0
class LLMClient(Protocol): class LLMClient(Protocol):
def complete( def complete(
self, self,
@@ -81,6 +91,7 @@ class LiteLLMClient:
*, *,
timeout_s: float = 120.0, timeout_s: float = 120.0,
transport: httpx.BaseTransport | None = None, transport: httpx.BaseTransport | None = None,
sleep: Any = time.sleep,
) -> None: ) -> None:
self._http = httpx.Client( self._http = httpx.Client(
base_url=base_url.rstrip("/"), base_url=base_url.rstrip("/"),
@@ -89,6 +100,7 @@ class LiteLLMClient:
transport=transport, transport=transport,
) )
self._renderer = OpenAIChatRenderer() self._renderer = OpenAIChatRenderer()
self._sleep = sleep
@classmethod @classmethod
def from_env(cls, env: Mapping[str, str] = os.environ) -> "LiteLLMClient": def from_env(cls, env: Mapping[str, str] = os.environ) -> "LiteLLMClient":
@@ -106,10 +118,9 @@ class LiteLLMClient:
system: str | None = None, system: str | None = None,
) -> list[CanonicalEvent]: ) -> list[CanonicalEvent]:
body = self._renderer.render(events, tools, model, system=system) body = self._renderer.render(events, tools, model, system=system)
try: response = _post_with_rate_limit_retry(
response = self._http.post("/chat/completions", json=body) self._http, "/chat/completions", body, self._sleep
except httpx.HTTPError as exc: )
raise LLMError(f"LLM request failed: {exc}") from exc
if response.status_code != 200: if response.status_code != 200:
raise LLMError( raise LLMError(
f"LLM request returned {response.status_code}: {response.text[:500]}" f"LLM request returned {response.status_code}: {response.text[:500]}"
@@ -131,14 +142,137 @@ class LiteLLMClient:
# gets a locally-minted canonical ID; the provider's is dropped, since # gets a locally-minted canonical ID; the provider's is dropped, since
# every future request is rendered wholly from the canonical log. # every future request is rendered wholly from the canonical log.
for call in message.get("tool_calls") or []: for call in message.get("tool_calls") or []:
raw = call["function"]["arguments"] or "{}"
# Weak models emit truncated or non-JSON argument strings; that is
# a bad model turn, not a client crash (A15).
try:
arguments = json.loads(raw)
except json.JSONDecodeError as exc:
raise LLMError(
f"tool call {call['function']['name']!r} had non-JSON "
f"arguments: {raw[:200]!r}"
) from exc
if not isinstance(arguments, dict):
raise LLMError(
f"tool call {call['function']['name']!r} arguments were "
f"{type(arguments).__name__}, expected object"
)
out.append( out.append(
ToolCall( ToolCall(
call_id=mint_call_id(), call_id=mint_call_id(),
tool=call["function"]["name"], tool=call["function"]["name"],
arguments=json.loads(call["function"]["arguments"] or "{}"), arguments=arguments,
) )
) )
return out return out
def close(self) -> None: def close(self) -> None:
self._http.close() self._http.close()
class AnthropicPassthroughClient:
"""LiteLLM's Anthropic passthrough (A5). Same interface as LiteLLMClient,
so a caller chooses a route rather than a different kind of client.
Delegations take this path because fidelity matters there (A13);
ask_expert and fallbacks use the normalized route, where it does not.
"""
def __init__(
self,
base_url: str,
token: str,
*,
timeout_s: float = 300.0,
max_tokens: int = 4096,
transport: httpx.BaseTransport | None = None,
sleep: Any = time.sleep,
) -> None:
self._http = httpx.Client(
base_url=base_url.rstrip("/"),
headers={
"Authorization": f"Bearer {token}",
"x-api-key": token,
"anthropic-version": "2023-06-01",
},
timeout=timeout_s,
transport=transport,
)
self._renderer = AnthropicMessagesRenderer()
self._max_tokens = max_tokens
self._sleep = sleep
@classmethod
def from_env(
cls, env: Mapping[str, str] = os.environ
) -> "AnthropicPassthroughClient":
base_url = env.get("NISSE_LLM_BASE_URL")
token = env.get("NISSE_LLM_TOKEN")
if not base_url or not token:
raise LLMError("set NISSE_LLM_BASE_URL and NISSE_LLM_TOKEN")
return cls(base_url, token)
def complete(
self,
events: list[StoredEvent],
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
) -> list[CanonicalEvent]:
body = self._renderer.render(
events, tools, model, system=system, max_tokens=self._max_tokens
)
response = _post_with_rate_limit_retry(
self._http, "/v1/messages", body, self._sleep
)
if response.status_code != 200:
raise LLMError(
f"passthrough returned {response.status_code}: {response.text[:500]}"
)
return self._parse(response.json(), requested_model=model)
def _parse(
self, data: dict[str, Any], requested_model: str
) -> list[CanonicalEvent]:
content = data.get("content")
if not isinstance(content, list):
raise LLMError(f"malformed passthrough response: {str(data)[:300]}")
produced_by = data.get("model") or requested_model
out: list[CanonicalEvent] = []
for block in content:
kind = block.get("type")
if kind == "text" and block.get("text"):
out.append(AssistantText(text=block["text"], model=produced_by))
elif kind == "tool_use":
# Provider ids are mapped at the boundary and never used as a
# join key (A10); the canonical log owns identity.
out.append(
ToolCall(
call_id=mint_call_id(),
tool=block["name"],
arguments=block.get("input") or {},
)
)
# Vendor-opaque reasoning blocks (thinking, redacted_thinking) are
# deliberately dropped for now: A10 requires storing them tagged
# with the producing model and never replaying them outside the
# session that made them, and the canonical schema has no event
# for that yet. Nothing requests extended thinking, so none arrive.
return out
def close(self) -> None:
self._http.close()
def _post_with_rate_limit_retry(
http: httpx.Client, path: str, body: dict[str, Any], sleep: Any
) -> httpx.Response:
for attempt in (1, 2):
try:
response = http.post(path, json=body)
except httpx.HTTPError as exc:
raise LLMError(f"LLM request failed: {exc}") from exc
if response.status_code != RATE_LIMIT_STATUS or attempt == 2:
return response
sleep(RATE_LIMIT_RETRY_S)
raise AssertionError("unreachable")
+171
View File
@@ -0,0 +1,171 @@
"""LLM-backed planning agent (spec A4 gating, A11 roles, A13 plan quality).
Gating is structural at the tool layer: the planner is offered exactly two
tools — ``submit_plan``, which posts a plan for approval and ends the turn,
and ``ask_human``, which pauses for an answer. No "proceed" tool exists, so
the model has no path to advance itself past a gate.
Model output is appended to the canonical log as it arrives (A2/A10), so a
crash mid-planning leaves a truthful transcript.
"""
from typing import Any
from nisse_dev.canonical import AssistantText, ToolCall, ToolResult
from nisse_dev.control import AskHuman, PlanningError, SubmitPlan
from nisse_dev.llm import LLMClient, LLMError, RoleMap
from nisse_dev.state import StateStore
# Plans may be executed by a materially weaker model, which cannot
# reconstruct intent (A13).
SYSTEM_PROMPT = """You are nisse-dev, a software engineering agent.
You are planning, not implementing. Produce plans that are explicit and
low-inference: a materially weaker model will execute them and cannot
reconstruct your intent. Name files, commands, and acceptance criteria
concretely; never write "handle the edge cases" or "wire it up".
You must answer by calling exactly one tool:
- submit_plan: your plan is posted for human approval and your turn ends.
- ask_human: ask one question when a decision is genuinely the operator's.
You cannot approve your own plan or start work; a human gate follows every
plan you submit."""
PLANNER_TOOLS: list[dict[str, Any]] = [
{
"type": "function",
"function": {
"name": "submit_plan",
"description": "Post a plan for human approval. Ends your turn.",
"parameters": {
"type": "object",
"properties": {
"type": {
"type": "string",
"description": "Which plan this is, e.g. 'arch' or 'milestone-1'.",
},
"content": {
"type": "string",
"description": "The plan itself, explicit enough to execute verbatim.",
},
},
"required": ["type", "content"],
},
},
},
{
"type": "function",
"function": {
"name": "ask_human",
"description": "Ask the operator one question, then wait.",
"parameters": {
"type": "object",
"properties": {
"question": {"type": "string"},
},
"required": ["question"],
},
},
},
]
_NUDGE = (
"You did not call a tool. Call submit_plan or ask_human now; "
"plain prose is not accepted."
)
class LLMPlanningAgent:
def __init__(
self,
client: LLMClient,
store: StateStore,
roles: RoleMap,
*,
system_prompt: str = SYSTEM_PROMPT,
) -> None:
self._client = client
self._store = store
self._roles = roles
self._system_prompt = system_prompt
def plan(
self,
gate: str,
project: str,
feedback: str | None = None,
answer: str | None = None,
) -> SubmitPlan | AskHuman:
"""Operator feedback and answers are already durable in the canonical
log (the orchestrator appends every inbound message), so they are not
re-injected here; the log is the single source of context."""
model = self._roles.resolve("planner")
instruction = (
f"{self._system_prompt}\n\n"
f"Current task: draft the {gate} plan for project {project!r}."
)
action, first_error = self._try_turn(model, instruction)
if action is not None:
return action
action, second_error = self._try_turn(model, f"{instruction}\n\n{_NUDGE}")
if action is None:
detail = second_error or first_error
raise PlanningError(
f"planner ({model}) produced no submit_plan or ask_human call "
f"after a retry{f': {detail}' if detail else ''}"
)
return action
def _try_turn(
self, model: str, system: str
) -> tuple[SubmitPlan | AskHuman | None, str | None]:
"""A malformed or failed call is a bad turn, not a crash: the caller
retries once and then gates to the human (A12 — judgment about model
output is observable, never the model's own claim)."""
try:
return self._turn(model, system), None
except LLMError as exc:
return None, str(exc)
def _turn(self, model: str, system: str) -> SubmitPlan | AskHuman | None:
events = self._store.read_events()
produced = self._client.complete(
events, tools=PLANNER_TOOLS, model=model, system=system
)
action: SubmitPlan | AskHuman | None = None
for event in produced:
# Append before interpreting: the transcript must be truthful even
# if the call below rejects the arguments.
self._store.append_event(event)
if isinstance(event, AssistantText) or action is not None:
continue
if isinstance(event, ToolCall):
action = self._interpret(event)
if action is not None:
# The gate is this call's outcome. Recording it keeps the
# transcript renderable: providers reject a tool call with
# no result, and the next planning turn replays this log.
self._store.append_event(
ToolResult(
call_id=event.call_id,
result=(
"plan posted for approval; turn ended"
if isinstance(action, SubmitPlan)
else "question posted; awaiting the operator"
),
success=True,
)
)
return action
def _interpret(self, call: ToolCall) -> SubmitPlan | AskHuman | None:
if call.tool == "submit_plan":
content = call.arguments.get("content")
if isinstance(content, str) and content.strip():
return SubmitPlan(content.strip())
elif call.tool == "ask_human":
question = call.arguments.get("question")
if isinstance(question, str) and question.strip():
return AskHuman(question.strip())
return None
+104
View File
@@ -0,0 +1,104 @@
"""Gitea comment intake (spec B2).
Polling, not webhooks: the agent VM stays outbound-only (B3), so nothing
needs to reach in. Intake is structured JSON straight from the API and never
routed through a chat bridge.
Outage posture, per B2: back off exponentially, cap at a few minutes, retry
indefinitely, and log. Polling is pure intake — an outage delays a reaction
but must never fail a task or force a state transition.
"""
import threading
from collections.abc import Callable, Iterable
from nisse_dev.gitea import Comment, GiteaClient, GiteaError
from nisse_dev.oplog import OpLog
DEFAULT_INTERVAL_S = 20.0
INITIAL_BACKOFF_S = 5.0
MAX_BACKOFF_S = 300.0
class CommentPoller:
def __init__(
self,
client: GiteaClient,
repo: str,
on_comment: Callable[[Comment], None],
*,
oplog: OpLog | None = None,
ignore_authors: Iterable[str] = (),
interval_s: float = DEFAULT_INTERVAL_S,
initial_backoff_s: float = INITIAL_BACKOFF_S,
max_backoff_s: float = MAX_BACKOFF_S,
) -> None:
self._client = client
self._repo = repo
self._on_comment = on_comment
self._oplog = oplog
self._ignore = set(ignore_authors)
self._interval_s = interval_s
self._initial_backoff_s = initial_backoff_s
self._max_backoff_s = max_backoff_s
self._seen: set[int] = set()
self._stop = threading.Event()
self._thread: threading.Thread | None = None
def prime(self, index: int) -> None:
"""Mark existing comments as seen without delivering them.
A poller that starts by replaying the whole thread would re-act to
feedback already addressed — including its own earlier replies.
"""
try:
for comment in self._client.list_comments(self._repo, index):
self._seen.add(comment.id)
except GiteaError:
# Nothing seen means the first successful poll delivers the
# backlog once. Better than refusing to start.
pass
def poll_once(self, index: int) -> list[Comment]:
"""One pass; returns what was newly delivered. Separated from the
loop so behaviour is testable without threads or sleeps."""
delivered: list[Comment] = []
for comment in self._client.list_comments(self._repo, index):
if comment.id in self._seen:
continue
self._seen.add(comment.id)
if comment.author in self._ignore:
continue # the agent's own replies are not new input
delivered.append(comment)
self._on_comment(comment)
return delivered
def start(self, index: int) -> None:
self._stop.clear()
self._thread = threading.Thread(target=self._loop, args=(index,), daemon=True)
self._thread.start()
def stop(self) -> None:
self._stop.set()
if self._thread is not None:
self._thread.join(timeout=self._interval_s + 5)
self._thread = None
def _loop(self, index: int) -> None:
backoff = self._initial_backoff_s
while not self._stop.is_set():
try:
self.poll_once(index)
except GiteaError as exc:
if self._oplog is not None:
self._oplog.emit(
"gitea.poll_failed",
cause="gitea unreachable",
error=str(exc)[:300],
retry_in_s=backoff,
)
self._stop.wait(backoff)
backoff = min(backoff * 2, self._max_backoff_s)
continue
backoff = self._initial_backoff_s
self._stop.wait(self._interval_s)
+134
View File
@@ -0,0 +1,134 @@
"""Publishing a finished milestone as a Gitea PR (spec B2, A2/A3).
The ordering here is the whole point. Intent is recorded *before* the first
externally visible act, so a crash anywhere in the sequence leaves a durable
record of what was in flight. Boot reconciliation then asks Gitea whether the
PR exists rather than repeating the push-and-open — the spec's own example of
an idempotent transition reconciled against external truth.
"""
from dataclasses import dataclass
from nisse_dev.git import GitRepo, authenticated_url
from nisse_dev.gitea import (
GiteaClient,
GiteaError,
PullRequest,
pull_request_reconciler,
)
from nisse_dev.reconcile import Reconciler
from nisse_dev.state import StateStore
OPEN_PR_INTENT = "open_pr"
@dataclass(frozen=True)
class PublishResult:
pull_request: PullRequest | None
note: str
class MilestonePublisher:
def __init__(
self,
store: StateStore,
client: GiteaClient,
repo: str,
git_repo: GitRepo,
*,
base_url: str,
account: str,
token: str,
base_branch: str = "main",
) -> None:
self._store = store
self._client = client
self._repo = repo
self._git = git_repo
self._base_branch = base_branch
self._remote = authenticated_url(base_url, repo, account, token)
self._token = token
def reconcilers(self) -> dict[str, Reconciler]:
return {OPEN_PR_INTENT: pull_request_reconciler(self._client, self._repo)}
def branch_for(self, task_id: str, milestone: int) -> str:
# Namespaced by task so re-running a project against the same repo
# cannot collide with an earlier run's branch.
return f"{task_id}/milestone-{milestone}"
def publish(
self,
*,
task_id: str,
milestone: int,
project: str,
plan: str,
service_url: str | None = None,
) -> PublishResult:
branch = self.branch_for(task_id, milestone)
if self._git.current_branch() != branch:
self._git.checkout_new_branch(branch)
sha = self._git.commit_all(f"Milestone {milestone}: {project}")
if sha is None:
# Not an error: the executor may have concluded there was nothing
# to change. Report it and let the operator decide.
return PublishResult(None, "no changes to publish")
intent_id = self._store.begin_intent(
OPEN_PR_INTENT,
{"branch": branch, "milestone": milestone, "sha": sha},
cause=f"publishing milestone {milestone}",
)
self._git.push(self._remote, branch, token=self._token)
pr = self._client.open_pull_request(
self._repo,
head=branch,
base=self._base_branch,
title=f"Milestone {milestone}: {project}",
body=_pr_body(milestone, plan, service_url),
)
self._store.commit_intent(intent_id, cause=f"opened PR #{pr.number}")
return PublishResult(pr, f"opened PR #{pr.number}: {pr.url}")
def comment(self, index: int, body: str) -> None:
"""Reply on the PR. Courtesy only — the pushed commit is the real
answer — so a failure here must not derail the review loop."""
try:
self._client.post_comment(self._repo, index, body)
except GiteaError:
pass
def push_followup(
self, *, task_id: str, milestone: int, message: str
) -> str | None:
"""Push a fix commit onto an open milestone's branch.
No intent is recorded: the PR already exists, so a crash mid-push
costs a repeated push of the same commit, which is harmless. That is
the difference from publish(), where the side effect is a new PR.
"""
branch = self.branch_for(task_id, milestone)
if self._git.current_branch() != branch:
self._git.checkout_new_branch(branch)
sha = self._git.commit_all(message)
if sha is None:
return None
self._git.push(self._remote, branch, token=self._token)
return sha
def _pr_body(milestone: int, plan: str, service_url: str | None) -> str:
parts = [
f"Milestone {milestone}, implemented by nisse-dev against the approved plan.",
"",
"## Approved plan",
"",
plan,
]
if service_url:
# B2: a runnable test service's address belongs in the PR description
# so a reviewer can exercise it. Nothing reports one yet; the browser
# tool (A6) is what will.
parts += ["", "## Running test service", "", service_url]
return "\n".join(parts)
+17 -1
View File
@@ -57,8 +57,24 @@ class OpenAIChatRenderer:
messages: list[dict[str, Any]] = [] messages: list[dict[str, Any]] = []
if system is not None: if system is not None:
messages.append({"role": "system", "content": system}) messages.append({"role": "system", "content": system})
for stored in effective_events(events): kept = effective_events(events)
answered = {e.event.call_id for e in kept if isinstance(e.event, ToolResult)}
for stored in kept:
messages.append(self._message_for(stored.event)) messages.append(self._message_for(stored.event))
# Providers reject a tool call with no result following it. One
# can legitimately be missing: a crash between the call and its
# result (A3), or a turn-ending gate tool. Render a placeholder so
# the transcript stays replayable rather than poisoned.
if isinstance(stored.event, ToolCall) and (
stored.event.call_id not in answered
):
messages.append(
{
"role": "tool",
"tool_call_id": stored.event.call_id,
"content": "[no result recorded]",
}
)
body: dict[str, Any] = {"model": model, "messages": messages} body: dict[str, Any] = {"model": model, "messages": messages}
if tools: if tools:
body["tools"] = tools body["tools"] = tools
+184
View File
@@ -0,0 +1,184 @@
"""Anthropic Messages renderer (spec A10, A13).
The second adapter, and the reason A13 puts normalisation in the renderer
rather than on the wire: the two vendors disagree *structurally*, not
cosmetically.
| | OpenAI-compatible | Anthropic Messages |
|---|---|---|
| system prompt | a message with role `system` | a top-level `system` field |
| tool call | `tool_calls` array on the message | a `tool_use` content block |
| tool result | a message with role `tool` | a `tool_result` block in a **user** message |
| tool schema | `function.parameters` | `input_schema` |
A middlebox that normalised between these would be lossy exactly where a
long-running delegation lives, which is why delegations take the passthrough
route and this renderer builds the vendor-native body.
"""
from typing import Any
from nisse_dev.canonical import (
AssistantText,
Compaction,
DelegationCompleted,
DelegationStarted,
GateOpened,
PlanSubmitted,
StoredEvent,
ToolCall,
ToolResult,
UserMessage,
)
from nisse_dev.render import effective_events
DEFAULT_MAX_TOKENS = 4096
NO_RESULT_PLACEHOLDER = "[no result recorded]"
def to_anthropic_tools(tools: list[dict[str, Any]] | None) -> list[dict[str, Any]]:
"""Convert the OpenAI-shaped manifests the tool modules already define,
so a tool is declared once and rendered per vendor."""
converted: list[dict[str, Any]] = []
for tool in tools or []:
fn = tool.get("function", tool)
converted.append(
{
"name": fn["name"],
"description": fn.get("description", ""),
"input_schema": fn.get(
"parameters", {"type": "object", "properties": {}}
),
}
)
return converted
class AnthropicMessagesRenderer:
def render(
self,
events: list[StoredEvent],
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
*,
max_tokens: int = DEFAULT_MAX_TOKENS,
) -> dict[str, Any]:
kept = effective_events(events)
answered = {e.event.call_id for e in kept if isinstance(e.event, ToolResult)}
blocks: list[tuple[str, dict[str, Any]]] = []
for stored in kept:
blocks.extend(self._blocks_for(stored.event))
# A dangling tool_use is rejected outright by this API — more
# strictly than the OpenAI shape — so a crash between a call and
# its result must not poison the transcript (A3).
if isinstance(stored.event, ToolCall) and (
stored.event.call_id not in answered
):
blocks.append(
(
"user",
{
"type": "tool_result",
"tool_use_id": stored.event.call_id,
"content": NO_RESULT_PLACEHOLDER,
},
)
)
body: dict[str, Any] = {
"model": model,
"max_tokens": max_tokens,
"messages": _coalesce(blocks),
}
if system is not None:
body["system"] = system
if tools:
body["tools"] = to_anthropic_tools(tools)
return body
def _blocks_for(self, event: object) -> list[tuple[str, dict[str, Any]]]:
match event:
case UserMessage(text=text):
return [("user", {"type": "text", "text": text})]
case AssistantText(text=text):
return [("assistant", {"type": "text", "text": text})]
case ToolCall(call_id=call_id, tool=tool, arguments=arguments):
return [
(
"assistant",
{
"type": "tool_use",
"id": call_id,
"name": tool,
"input": arguments,
},
)
]
case ToolResult(call_id=call_id, result=result):
return [
(
"user",
{
"type": "tool_result",
"tool_use_id": call_id,
"content": result,
},
)
]
case PlanSubmitted(plan=plan):
return [("assistant", {"type": "text", "text": plan})]
case GateOpened(gate=gate, reason=reason):
return [
(
"user",
{"type": "text", "text": f"[gate {gate} opened: {reason}]"},
)
]
case DelegationStarted(brief=brief, model=model):
return [
(
"assistant",
{
"type": "text",
"text": f"[delegation started ({model}): {brief}]",
},
)
]
case DelegationCompleted(handback=handback):
return [
(
"assistant",
{"type": "text", "text": f"[delegation handback]\n{handback}"},
)
]
case Compaction(summary=summary):
return [
(
"user",
{
"type": "text",
"text": f"[summary of earlier conversation]\n{summary}",
},
)
]
case _:
raise ValueError(f"unrenderable event: {event!r}")
def _coalesce(
blocks: list[tuple[str, dict[str, Any]]],
) -> list[dict[str, Any]]:
"""Merge adjacent blocks of the same role into one message.
The Messages API expects alternating roles, and a tool_result must sit in
the user message directly after the assistant's tool_use — which happens
naturally once same-role runs are merged.
"""
messages: list[dict[str, Any]] = []
for role, block in blocks:
if messages and messages[-1]["role"] == role:
messages[-1]["content"].append(block)
else:
messages.append({"role": role, "content": [block]})
return messages
+235
View File
@@ -0,0 +1,235 @@
"""Executor tools (spec A15: reduced schema, flat arguments).
Deliberately unsandboxed. The VM is the trust boundary (B3) and tool use is
auto-approved inside an approved unit of work (A4), so per-command
confinement would add failure modes without adding safety. The workspace is
where work happens — the default for relative paths and the command cwd — not
a wall. Protecting agent state from the agent is a mount-layout concern
(A2/B3), not this module's job.
What *is* enforced here is operational: commands cannot hang the loop, cannot
consume the context window with megabytes of build log, and do not inherit
the agent's own credentials.
"""
import os
import shutil
import signal
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Any
# Tools that change or verify something. Reading a file or listing a
# directory is a model looking around, not doing the work, so neither counts
# as evidence that work was carried out.
MUTATING_TOOLS = frozenset({"write_file", "run_command"})
DEFAULT_TIMEOUT_S = 120
MAX_OUTPUT_CHARS = 4000
EXECUTOR_TOOLS: list[dict[str, Any]] = [
{
"type": "function",
"function": {
"name": "read_file",
"description": "Read a UTF-8 text file.",
"parameters": {
"type": "object",
"properties": {"path": {"type": "string"}},
"required": ["path"],
},
},
},
{
"type": "function",
"function": {
"name": "write_file",
"description": "Write a UTF-8 text file, creating parent directories.",
"parameters": {
"type": "object",
"properties": {
"path": {"type": "string"},
"content": {"type": "string"},
},
"required": ["path", "content"],
},
},
},
{
"type": "function",
"function": {
"name": "list_dir",
"description": "List the entries of a directory.",
"parameters": {
"type": "object",
"properties": {"path": {"type": "string"}},
"required": ["path"],
},
},
},
{
"type": "function",
"function": {
"name": "run_command",
"description": (
"Run a shell command in the workspace. Returns its exit code "
"and output; a non-zero exit is a result, not an error."
),
"parameters": {
"type": "object",
"properties": {"command": {"type": "string"}},
"required": ["command"],
},
},
},
]
def _truncate(text: str) -> str:
if len(text) <= MAX_OUTPUT_CHARS:
return text
dropped = len(text) - MAX_OUTPUT_CHARS
return f"{text[:MAX_OUTPUT_CHARS]}\n[... {dropped} characters truncated]"
def _child_env() -> dict[str, str]:
"""The project's build and test code is arbitrary; it has no business
reading the agent's LiteLLM, Matrix, or Gitea credentials."""
return {k: v for k, v in os.environ.items() if not k.startswith("NISSE_")}
@dataclass(frozen=True)
class ToolOutcome:
result: str
success: bool
class Workspace:
def __init__(self, root: Path, *, timeout_s: int = DEFAULT_TIMEOUT_S) -> None:
self.root = Path(root)
self.root.mkdir(parents=True, exist_ok=True)
self._timeout_s = timeout_s
def _resolve(self, path: str) -> Path:
candidate = Path(path)
return candidate if candidate.is_absolute() else self.root / candidate
def _inside(self, target: Path) -> bool:
try:
return target.resolve().is_relative_to(self.root.resolve())
except OSError:
return False
def dispatch(self, tool: str, arguments: dict[str, Any]) -> ToolOutcome:
try:
match tool:
case "read_file":
return self.read_file(str(arguments["path"]))
case "write_file":
return self.write_file(
str(arguments["path"]), str(arguments["content"])
)
case "list_dir":
return self.list_dir(str(arguments.get("path", ".")))
case "run_command":
return self.run_command(str(arguments["command"]))
case _:
return ToolOutcome(f"no such tool: {tool}", success=False)
except KeyError as exc:
return ToolOutcome(f"missing argument {exc} for {tool}", success=False)
def read_file(self, path: str) -> ToolOutcome:
target = self._resolve(path)
try:
return ToolOutcome(_truncate(target.read_text()), success=True)
except (OSError, UnicodeDecodeError) as exc:
return ToolOutcome(f"could not read {path}: {exc}", success=False)
def write_file(self, path: str, content: str) -> ToolOutcome:
target = self._resolve(path)
# Not a sandbox — the VM is the trust boundary (B3). This is about the
# work existing at all: a file written outside the repo is invisible
# to git, so the milestone would publish an empty PR while the model
# truthfully reports success. Models do guess plausible absolute paths
# like /workspace, so say where the workspace actually is.
if not self._inside(target):
return ToolOutcome(
f"refusing to write outside the workspace: {path}. "
f"The workspace root is {self.root}; write there, using "
"relative paths.",
success=False,
)
try:
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(content)
except OSError as exc:
return ToolOutcome(f"could not write {path}: {exc}", success=False)
return ToolOutcome(f"wrote {len(content)} characters to {path}", success=True)
def list_dir(self, path: str) -> ToolOutcome:
target = self._resolve(path)
try:
entries = sorted(
f"{p.name}/" if p.is_dir() else p.name for p in target.iterdir()
)
except OSError as exc:
return ToolOutcome(f"could not list {path}: {exc}", success=False)
return ToolOutcome(_truncate("\n".join(entries) or "(empty)"), success=True)
def describe(self) -> str:
"""Told to the model, so it does not have to guess where it is."""
return (
f"Your workspace is {self.root}. It is a git repository and the "
"only place your work is recorded; commands run there by default. "
"Use relative paths."
)
def run_command(self, command: str) -> ToolOutcome:
try:
# Popen rather than run(): on timeout, run() kills only the direct
# child, and TimeoutExpired carries no pid, so a backgrounded
# grandchild (a dev server, say) would survive. Holding the pid
# lets us kill the whole session.
process = subprocess.Popen(
command,
shell=True,
cwd=self.root,
env=_child_env(),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.DEVNULL,
text=True,
start_new_session=True,
)
except OSError as exc:
return ToolOutcome(f"could not run command: {exc}", success=False)
try:
stdout, stderr = process.communicate(timeout=self._timeout_s)
except subprocess.TimeoutExpired:
self._kill_group(process)
return ToolOutcome(
f"command timed out after {self._timeout_s}s: {command}",
success=False,
)
body = _truncate(
f"exit code: {process.returncode}\nstdout:\n{stdout}\nstderr:\n{stderr}"
)
return ToolOutcome(body, success=process.returncode == 0)
@staticmethod
def _kill_group(process: "subprocess.Popen[str]") -> None:
try:
os.killpg(os.getpgid(process.pid), signal.SIGKILL)
except (ProcessLookupError, PermissionError):
process.kill()
# Drain the pipes of the now-dead group so no zombie is left behind.
try:
process.communicate(timeout=5)
except subprocess.TimeoutExpired:
pass
def reset(self) -> None:
"""Discard workspace contents; used by tests, not by the agent."""
shutil.rmtree(self.root, ignore_errors=True)
self.root.mkdir(parents=True, exist_ok=True)
+210
View File
@@ -0,0 +1,210 @@
"""Plan externalisation to Gitea and approvals arriving on either channel
(spec A4, Part H)."""
import io
from pathlib import Path
from typing import Any
from nisse_dev.approvals import PlanBoard
from nisse_dev.canonical import CanonicalEvent, ToolCall
from nisse_dev.control import (
AWAIT_ARCH_SIGNOFF,
AWAIT_MILESTONE_SIGNOFF,
MILESTONE_PLANNING,
Orchestrator,
)
from nisse_dev.gitea import Comment, GiteaError
from nisse_dev.llm import RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.planner import LLMPlanningAgent
from nisse_dev.state import StateStore
from nisse_dev.transport import InMemoryTransport
ROLES = RoleMap(executor="x", planner="y", delegate="y", consultant="y")
class FakeGitea:
def __init__(self, *, fail: bool = False) -> None:
self.issues: list[dict[str, str]] = []
self.comments: list[tuple[int, str]] = []
self._fail = fail
self._next_id = 100
def create_issue(self, repo: str, title: str, body: str) -> int:
if self._fail:
raise GiteaError("gitea is down")
self.issues.append({"title": title, "body": body})
return len(self.issues)
def post_comment(self, repo: str, index: int, body: str) -> Comment:
if self._fail:
raise GiteaError("gitea is down")
self.comments.append((index, body))
self._next_id += 1
return Comment(
id=self._next_id, author="nisse-dev-testing", body=body, created_at="t"
)
def list_comments(self, repo: str, index: int, since: str | None = None):
return []
class ScriptedClient:
def __init__(self) -> None:
self.calls = 0
def complete(self, events, tools, model, system=None) -> list[CanonicalEvent]:
self.calls += 1
return [
ToolCall(
call_id=f"c{self.calls}",
tool="submit_plan",
arguments={"type": "x", "content": f"plan {self.calls}"},
)
]
def build(
tmp_path: Path, gitea: Any
) -> tuple[Orchestrator, StateStore, InMemoryTransport, PlanBoard]:
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
transport = InMemoryTransport()
board = PlanBoard(gitea, "o/sandbox")
orch = Orchestrator(
store,
transport,
LLMPlanningAgent(ScriptedClient(), store, ROLES),
operators={"don"},
roles=ROLES,
gitea_operators={"dongar"},
board=board,
)
orch.start()
return orch, store, transport, board
def comment(body: str, author: str = "dongar", cid: int = 900) -> Comment:
return Comment(id=cid, author=author, body=body, created_at="t")
def test_project_start_opens_an_issue_and_posts_the_plan(tmp_path: Path) -> None:
gitea = FakeGitea()
_, store, transport, _ = build(tmp_path, gitea)
transport.receive("don", "use demo")
assert gitea.issues[0]["title"] == "nisse-dev: demo"
issue, body = gitea.comments[0]
assert issue == 1
assert "plan 1" in body
assert "approve rev-1" in body # the tag is printed for copy-paste (A4)
# The store keeps a pointer, not the plan record itself (Part H).
assert store.get_state().data["board"]["issue"] == 1
def test_approval_by_gitea_comment_satisfies_the_gate(tmp_path: Path) -> None:
gitea = FakeGitea()
orch, store, _, _ = build(tmp_path, gitea)
orch._transport.receive("don", "use demo")
assert store.get_state().state == AWAIT_ARCH_SIGNOFF
orch.handle_plan_comment(comment("approve rev-1"))
assert store.get_state().state == AWAIT_MILESTONE_SIGNOFF
def test_stale_revision_from_gitea_is_refused(tmp_path: Path) -> None:
gitea = FakeGitea()
orch, store, transport, _ = build(tmp_path, gitea)
transport.receive("don", "use demo")
transport.receive("don", "too vague") # now at rev-2
orch.handle_plan_comment(comment("approve rev-1"))
assert store.get_state().state == AWAIT_ARCH_SIGNOFF # not advanced
orch.handle_plan_comment(comment("approve rev-2", cid=901))
assert store.get_state().state == AWAIT_MILESTONE_SIGNOFF
def test_non_operator_gitea_comment_cannot_approve(tmp_path: Path) -> None:
gitea = FakeGitea()
orch, store, transport, _ = build(tmp_path, gitea)
transport.receive("don", "use demo")
orch.handle_plan_comment(comment("approve rev-1", author="drive-by"))
assert store.get_state().state == AWAIT_ARCH_SIGNOFF
def test_agents_own_comment_cannot_approve_its_plan(tmp_path: Path) -> None:
# The agent posts plans as nisse-dev-testing; that login is not on the
# operator allowlist, so its own comments can never satisfy its gate (A4).
gitea = FakeGitea()
orch, store, transport, _ = build(tmp_path, gitea)
transport.receive("don", "use demo")
orch.handle_plan_comment(comment("approve rev-1", author="nisse-dev-testing"))
assert store.get_state().state == AWAIT_ARCH_SIGNOFF
def test_rejection_by_gitea_comment_replans(tmp_path: Path) -> None:
gitea = FakeGitea()
orch, store, transport, _ = build(tmp_path, gitea)
transport.receive("don", "use demo")
orch.handle_plan_comment(comment("needs more detail on the schema"))
assert store.get_state().state == AWAIT_ARCH_SIGNOFF
assert "plan 2" in transport.outbound[-1].text # replanned
def test_matrix_approval_is_echoed_to_gitea(tmp_path: Path) -> None:
gitea = FakeGitea()
_, _, transport, _ = build(tmp_path, gitea)
transport.receive("don", "use demo")
transport.receive("don", "approve")
echoes = [body for _, body in gitea.comments if "approved by don" in body]
assert echoes, "the Gitea record must not silently disagree with Matrix"
def test_gitea_approval_is_echoed_to_matrix(tmp_path: Path) -> None:
gitea = FakeGitea()
orch, _, transport, _ = build(tmp_path, gitea)
transport.receive("don", "use demo")
orch.handle_plan_comment(comment("approve rev-1"))
assert any("approved by dongar via gitea" in m.text for m in transport.outbound)
def test_gitea_outage_leaves_matrix_a_complete_channel(tmp_path: Path) -> None:
gitea = FakeGitea(fail=True)
_, store, transport, _ = build(tmp_path, gitea)
transport.receive("don", "use demo")
# Planning proceeded and the gate is usable despite Gitea being down.
assert store.get_state().state == AWAIT_ARCH_SIGNOFF
assert any("Matrix-only" in m.text for m in transport.outbound)
transport.receive("don", "approve")
assert store.get_state().state == AWAIT_MILESTONE_SIGNOFF
def test_comments_are_ignored_outside_a_gate(tmp_path: Path) -> None:
gitea = FakeGitea()
orch, store, transport, _ = build(tmp_path, gitea)
transport.receive("don", "use demo")
transport.receive("don", "approve") # arch approved; now at milestone gate
transport.receive("don", "approve") # milestone approved, awaiting start
before = store.get_state().state
orch.handle_plan_comment(comment("approve rev-1", cid=902))
assert store.get_state().state == before
def test_board_records_the_milestone_plan_too(tmp_path: Path) -> None:
gitea = FakeGitea()
_, store, transport, _ = build(tmp_path, gitea)
transport.receive("don", "use demo")
transport.receive("don", "approve")
assert store.get_state().state in (MILESTONE_PLANNING, AWAIT_MILESTONE_SIGNOFF)
posted = [body for _, body in gitea.comments if "milestone-1" in body]
assert posted, "the milestone plan must be externalised as well"
+304
View File
@@ -0,0 +1,304 @@
import io
from pathlib import Path
from typing import Any
import pytest
from nisse_dev.canonical import (
AssistantText,
CanonicalEvent,
DelegationCompleted,
DelegationStarted,
StoredEvent,
ToolCall,
)
from nisse_dev.delegation import (
DELEGATE_TOOLS,
Caps,
Delegation,
)
from nisse_dev.git import GitRepo
from nisse_dev.llm import LLMError
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
from nisse_dev.tools import Workspace
class ScriptedClient:
def __init__(self, turns: list[list[CanonicalEvent]]) -> None:
self._turns = turns
self.calls: list[dict[str, Any]] = []
def complete(
self,
events: list[StoredEvent],
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
) -> list[CanonicalEvent]:
self.calls.append({"tools": tools, "model": model, "system": system})
if not self._turns:
return [AssistantText(text="still thinking", model=model)]
return self._turns.pop(0)
class FailingClient:
def complete(self, *args: Any, **kwargs: Any) -> list[CanonicalEvent]:
raise LLMError("frontier unavailable")
def call(tool: str, cid: str = "c1", **arguments: Any) -> ToolCall:
return ToolCall(call_id=cid, tool=tool, arguments=arguments)
def hand_back(succeeded: bool = True, cid: str = "hb", **overrides: Any) -> ToolCall:
args = {
"changed": "added greet.py",
"now_true": "greet() returns a greeting and its test passes",
"next": "open the PR",
"succeeded": succeeded,
}
args.update(overrides)
return ToolCall(call_id=cid, tool="hand_back", arguments=args)
@pytest.fixture
def store(tmp_path: Path) -> StateStore:
return StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
@pytest.fixture
def repo(tmp_path: Path) -> GitRepo:
r = GitRepo(tmp_path / "work")
r.init()
r.configure_identity("bot", "bot@example.test")
(r.root / "seed.txt").write_text("seed")
r.commit_all("seed")
return r
@pytest.fixture
def ws(repo: GitRepo) -> Workspace:
return Workspace(repo.root, timeout_s=10)
def build(client: Any, store: StateStore, ws: Workspace, repo: GitRepo, **kw: Any):
return Delegation(client, store, ws, repo, model="claude-haiku-4-5", **kw)
def test_successful_delegation_returns_a_structured_handback(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
client = ScriptedClient(
[
[call("write_file", path="greet.py", content="def greet(): ...")],
[hand_back()],
]
)
result = build(client, store, ws, repo).run(brief="add greet", branch="d1")
assert result.usable
assert result.handback is not None
assert "greet.py" in result.handback.changed
assert "what to do next" in result.handback.render().lower()
assert result.artifacts == ["greet.py"]
def test_delegation_works_on_its_own_branch(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
client = ScriptedClient(
[[call("write_file", path="a.py", content="x")], [hand_back()]]
)
build(client, store, ws, repo).run(brief="b", branch="deleg-1")
assert repo.current_branch() == "deleg-1"
def test_abort_discards_the_branch_and_its_work(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# A13: abort must leave nothing to resume on top of.
client = ScriptedClient(
[
[call("write_file", path="half-done.py", content="oops")],
[hand_back(succeeded=False, changed="could not finish")],
]
)
result = build(client, store, ws, repo).run(brief="b", branch="deleg-2")
assert not result.usable
assert repo.current_branch() == "main"
assert not (repo.root / "half-done.py").exists()
def test_turn_cap_is_hard(store: StateStore, ws: Workspace, repo: GitRepo) -> None:
forever = [[call("list_dir", path=".")] for _ in range(20)]
client = ScriptedClient(forever)
result = build(client, store, ws, repo, caps=Caps(turns=3)).run(
brief="b", branch="d"
)
assert result.status == "capped"
assert "turn cap of 3" in result.note
assert len(client.calls) == 3
def test_wall_clock_cap_is_hard(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
ticks = iter([0.0, 1.0, 99.0, 99.0, 99.0])
client = ScriptedClient([[call("list_dir", path=".")] for _ in range(10)])
result = build(
client,
store,
ws,
repo,
caps=Caps(turns=10, wall_clock_s=60.0),
now=lambda: next(ticks),
).run(brief="b", branch="d")
assert result.status == "capped"
assert "wall-clock cap" in result.note
def test_cap_breach_is_not_usable_and_discards_work(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
client = ScriptedClient([[call("write_file", path="partial.py", content="x")]] * 5)
result = build(client, store, ws, repo, caps=Caps(turns=2)).run(
brief="b", branch="d"
)
assert not result.usable # the caller must gate, not resume (A13)
assert not (repo.root / "partial.py").exists()
def test_events_are_checkpointed_as_they_arrive(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
client = ScriptedClient(
[[call("write_file", path="a.py", content="x")], [hand_back()]]
)
build(client, store, ws, repo).run(brief="add a", branch="d")
kinds = [type(e.event).__name__ for e in store.read_events()]
assert kinds == [
"DelegationStarted",
"ToolCall",
"ToolResult",
"ToolCall", # hand_back
"ToolResult",
"DelegationCompleted",
]
def test_composite_events_record_the_brief_caps_and_outcome(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
client = ScriptedClient(
[[call("write_file", path="a.py", content="x")], [hand_back()]]
)
caps = Caps(turns=7, tokens=1234, wall_clock_s=42.0)
build(client, store, ws, repo, caps=caps).run(brief="the brief", branch="d")
events = [e.event for e in store.read_events()]
started = next(e for e in events if isinstance(e, DelegationStarted))
completed = next(e for e in events if isinstance(e, DelegationCompleted))
assert started.brief == "the brief"
assert started.model == "claude-haiku-4-5"
assert started.caps == {"turns": 7, "tokens": 1234, "wall_clock_s": 42.0}
# A16: this pair is the eval record, so it must carry the outcome.
assert completed.exit_status == "handback"
assert "greet()" in completed.handback
def test_hand_back_tool_is_offered_alongside_the_executor_tools() -> None:
names = [t["function"]["name"] for t in DELEGATE_TOOLS]
assert names[-1] == "hand_back"
assert "run_command" in names # full tool access, not a consult (A13)
def test_malformed_handback_is_rejected_and_the_delegation_continues(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
client = ScriptedClient(
[
[hand_back(cid="bad", changed=" ")], # blank field
[call("write_file", path="a.py", content="x")],
[hand_back(cid="good")],
]
)
result = build(client, store, ws, repo).run(brief="b", branch="d")
assert result.usable # recovered rather than accepting a useless note
assert len(client.calls) == 3
def test_non_boolean_success_flag_is_rejected(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
client = ScriptedClient(
[
[call("write_file", path="a.py", content="x")],
[hand_back(succeeded="yes")], # not a bool: unparseable
[hand_back(cid="ok")],
]
)
result = build(client, store, ws, repo).run(brief="b", branch="d")
assert result.usable
assert len(client.calls) == 3
def test_client_failure_is_a_defined_return_not_a_crash(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
result = build(FailingClient(), store, ws, repo).run(brief="b", branch="d")
assert result.status == "failed"
assert "frontier unavailable" in result.note
assert not result.usable
def test_a_delegate_that_never_hands_back_is_capped_not_believed(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# Prose forever: the delegate never reports, so its tree state is unknown.
client = ScriptedClient([])
result = build(client, store, ws, repo, caps=Caps(turns=3)).run(
brief="b", branch="d"
)
assert result.status == "capped"
assert not result.usable
def test_success_claimed_without_any_change_is_refused(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# The handback plus file state is the whole interface back, so a note
# describing work that was never done is worse than no note.
client = ScriptedClient(
[
[hand_back(cid="early")],
[call("write_file", path="real.py", content="x")],
[hand_back(cid="earned")],
]
)
result = build(client, store, ws, repo).run(brief="b", branch="d")
assert result.usable
assert len(client.calls) == 3 # the first claim was sent back
assert (repo.root / "real.py").exists()
def test_failure_may_be_reported_without_doing_work(
store: StateStore, ws: Workspace, repo: GitRepo
) -> None:
# "This sub-task is impossible" is a legitimate finding that requires no
# changes; only a claim of success needs evidence.
client = ScriptedClient(
[[hand_back(succeeded=False, changed="nothing: the API does not exist")]]
)
result = build(client, store, ws, repo).run(brief="b", branch="d")
assert result.status == "failed"
assert not result.usable
assert len(client.calls) == 1 # believed immediately, not challenged
+225
View File
@@ -0,0 +1,225 @@
import io
from pathlib import Path
from typing import Any
import pytest
from nisse_dev.canonical import AssistantText, CanonicalEvent, StoredEvent, ToolCall
from nisse_dev.executor import LLMExecutor
from nisse_dev.llm import LLMError, RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
from nisse_dev.tools import Workspace
ROLES = RoleMap(
executor="llama3.2",
planner="claude-haiku-4-5",
delegate="claude-haiku-4-5",
consultant="claude-haiku-4-5",
)
class ScriptedClient:
def __init__(self, turns: list[list[CanonicalEvent]]) -> None:
self._turns = turns
self.calls: list[dict[str, Any]] = []
def complete(
self,
events: list[StoredEvent],
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
) -> list[CanonicalEvent]:
self.calls.append({"tools": tools, "model": model, "system": system})
if not self._turns:
return [AssistantText(text="done", model=model)]
return self._turns.pop(0)
class FailingClient:
def complete(self, *args: Any, **kwargs: Any) -> list[CanonicalEvent]:
raise LLMError("proxy exploded")
def call(tool: str, call_id: str = "c1", **arguments: Any) -> ToolCall:
return ToolCall(call_id=call_id, tool=tool, arguments=arguments)
@pytest.fixture
def store(tmp_path: Path) -> StateStore:
return StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
@pytest.fixture
def ws(tmp_path: Path) -> Workspace:
return Workspace(tmp_path / "work", timeout_s=10)
def build(client: Any, store: StateStore, ws: Workspace, **kw: Any) -> LLMExecutor:
return LLMExecutor(client, store, ROLES, ws, **kw)
def test_tool_calls_are_executed_and_results_logged(
store: StateStore, ws: Workspace
) -> None:
client = ScriptedClient(
[
[call("write_file", path="app.py", content="print(1)\n")],
[AssistantText(text="wrote the file", model="llama3.2")],
]
)
outcome = build(client, store, ws).run(milestone=1)
assert outcome.status == "completed"
assert (ws.root / "app.py").read_text() == "print(1)\n"
kinds = [type(e.event).__name__ for e in store.read_events()]
assert kinds == ["ToolCall", "ToolResult", "AssistantText"]
def test_uses_executor_role_and_reduced_manifest(
store: StateStore, ws: Workspace
) -> None:
client = ScriptedClient([])
build(client, store, ws).run(milestone=2)
assert client.calls[0]["model"] == "llama3.2" # never the planner
names = [t["function"]["name"] for t in client.calls[0]["tools"]]
assert names == ["read_file", "write_file", "list_dir", "run_command"]
assert "milestone 2" in client.calls[0]["system"]
def test_no_tool_call_after_work_ends_the_milestone(
store: StateStore, ws: Workspace
) -> None:
client = ScriptedClient(
[
[call("write_file", path="a.py", content="x")],
[AssistantText(text="all steps done", model="m")],
]
)
assert build(client, store, ws).run(milestone=1).status == "completed"
def test_completion_claimed_without_any_work_is_challenged(
store: StateStore, ws: Workspace
) -> None:
# A model that answers with prose on turn one has done nothing; believing
# it would record an empty milestone as a success.
client = ScriptedClient(
[
[AssistantText(text="looks done to me", model="m")],
[call("write_file", path="a.py", content="x")],
[AssistantText(text="now it is done", model="m")],
]
)
outcome = build(client, store, ws).run(milestone=1)
assert outcome.status == "completed"
assert (ws.root / "a.py").exists() # the nudge produced actual work
assert "not made any change yet" in client.calls[1]["system"]
def test_persisting_in_an_empty_claim_escalates(
store: StateStore, ws: Workspace
) -> None:
prose = [AssistantText(text="nothing to do", model="m")]
client = ScriptedClient([prose, prose])
outcome = build(client, store, ws).run(milestone=1)
assert outcome.status == "escalate"
assert "without making any change" in outcome.note
def test_two_consecutive_command_failures_escalate(
store: StateStore, ws: Workspace
) -> None:
client = ScriptedClient(
[
[call("run_command", command="exit 1")],
[call("run_command", command="exit 1")],
[call("run_command", command="exit 1")],
]
)
outcome = build(client, store, ws, failure_limit=2).run(milestone=1)
assert outcome.status == "escalate"
assert "2 consecutive command failures" in outcome.note
assert len(client.calls) == 2 # stopped, did not keep trying
def test_a_success_resets_the_failure_count(store: StateStore, ws: Workspace) -> None:
client = ScriptedClient(
[
[call("run_command", command="exit 1")],
[call("run_command", command="true")],
[call("run_command", command="exit 1")],
[AssistantText(text="done", model="m")],
]
)
assert build(client, store, ws, failure_limit=2).run(milestone=1).status == (
"completed"
)
def test_failed_reads_do_not_count_toward_escalation(
store: StateStore, ws: Workspace
) -> None:
# Two failed reads in a row must not trip the failure limit: looking for
# a file that is not there is the model orienting itself, not failing.
client = ScriptedClient(
[
[call("read_file", path="missing-a.txt")],
[call("read_file", path="missing-b.txt")],
[call("write_file", path="a.py", content="x")],
[AssistantText(text="done", model="m")],
]
)
assert build(client, store, ws, failure_limit=2).run(milestone=1).status == (
"completed"
)
def test_reading_alone_is_not_doing_the_work(store: StateStore, ws: Workspace) -> None:
# Exploration is not evidence a milestone was carried out, so a
# completion claim backed only by reads is still challenged.
client = ScriptedClient(
[
[call("list_dir", path=".")],
[AssistantText(text="looks done to me", model="m")],
[AssistantText(text="really, done", model="m")],
]
)
outcome = build(client, store, ws).run(milestone=1)
assert outcome.status == "escalate"
def test_turn_cap_is_enforced(store: StateStore, ws: Workspace) -> None:
forever = [[call("list_dir", path=".")] for _ in range(50)]
client = ScriptedClient(forever)
outcome = build(client, store, ws, max_turns=3).run(milestone=1)
assert outcome.status == "capped"
assert "3-turn cap" in outcome.note
assert len(client.calls) == 3
def test_llm_error_escalates_rather_than_crashing(
store: StateStore, ws: Workspace
) -> None:
outcome = build(FailingClient(), store, ws).run(milestone=1)
assert outcome.status == "escalate"
assert "proxy exploded" in outcome.note
def test_multiple_calls_in_one_turn_all_run(store: StateStore, ws: Workspace) -> None:
client = ScriptedClient(
[
[
call("write_file", call_id="a", path="one.txt", content="1"),
call("write_file", call_id="b", path="two.txt", content="2"),
],
[AssistantText(text="done", model="m")],
]
)
build(client, store, ws).run(milestone=1)
assert (ws.root / "one.txt").exists() and (ws.root / "two.txt").exists()
results = [
e.event for e in store.read_events() if type(e.event).__name__ == "ToolResult"
]
assert {r.call_id for r in results} == {"a", "b"}
+70
View File
@@ -0,0 +1,70 @@
from pathlib import Path
import pytest
from nisse_dev.git import GitError, GitRepo, _scrub, authenticated_url
@pytest.fixture
def repo(tmp_path: Path) -> GitRepo:
r = GitRepo(tmp_path / "work")
r.init()
r.configure_identity("nisse-dev-testing", "nisse@example.test")
return r
def test_init_creates_repo_on_default_branch(repo: GitRepo) -> None:
assert repo.current_branch() == "main"
def test_commit_all_returns_sha_and_branch_switching(repo: GitRepo) -> None:
(repo.root / "a.txt").write_text("a")
sha = repo.commit_all("first")
assert sha is not None
assert sha == repo.head_sha()
repo.checkout_new_branch("milestone-1")
assert repo.current_branch() == "milestone-1"
def test_commit_with_nothing_staged_returns_none(repo: GitRepo) -> None:
(repo.root / "a.txt").write_text("a")
repo.commit_all("first")
# An empty milestone is a fact to report, not an error to raise.
assert repo.commit_all("nothing changed") is None
def test_has_changes_tracks_the_worktree(repo: GitRepo) -> None:
assert not repo.has_changes()
(repo.root / "a.txt").write_text("a")
assert repo.has_changes()
def test_authenticated_url_shape() -> None:
url = authenticated_url("https://gitea.test/", "o/r", "bot", "secret")
assert url == "https://bot:secret@gitea.test/o/r.git"
def test_push_failure_does_not_leak_the_token(repo: GitRepo) -> None:
(repo.root / "a.txt").write_text("a")
repo.commit_all("first")
url = authenticated_url("https://127.0.0.1:1", "o/r", "bot", "sup3rsecret")
with pytest.raises(GitError) as excinfo:
repo.push(url, "main", token="sup3rsecret")
# git redacts credentials from its own messages, but that is git's
# behaviour, not ours; the scrub below is the guarantee we control.
assert "sup3rsecret" not in str(excinfo.value)
def test_scrub_removes_the_token_when_git_does_echo_it() -> None:
assert _scrub("failed to push to https://bot:s3cret@h/o/r.git", "s3cret") == (
"failed to push to https://bot:***@h/o/r.git"
)
assert _scrub("no secret here", None) == "no secret here"
def test_git_error_on_bad_command(tmp_path: Path) -> None:
repo = GitRepo(tmp_path / "not-a-repo")
repo.root.mkdir()
with pytest.raises(GitError):
repo.head_sha()
+202
View File
@@ -0,0 +1,202 @@
import json
import httpx
import pytest
from nisse_dev.gitea import (
PR_PAGE_SIZE,
Comment,
GiteaClient,
GiteaError,
pull_request_reconciler,
)
from nisse_dev.state import Intent
INTENT = Intent(
id=1,
kind="open_pr",
payload={"branch": "milestone-1"},
status="pending",
created_at="t",
resolved_at=None,
)
def client(handler) -> GiteaClient:
return GiteaClient(
"https://gitea.test/", "tok", transport=httpx.MockTransport(handler)
)
def pr_json(number: int, head: str, state: str = "open") -> dict:
return {
"number": number,
"state": state,
"head": {"ref": head},
"html_url": f"https://gitea.test/o/r/pulls/{number}",
}
def test_open_pull_request_posts_expected_body() -> None:
seen: dict = {}
def handler(request: httpx.Request) -> httpx.Response:
seen["url"] = str(request.url)
seen["auth"] = request.headers.get("authorization")
seen["body"] = json.loads(request.content)
return httpx.Response(201, json=pr_json(7, "milestone-1"))
pr = client(handler).open_pull_request(
"o/r", head="milestone-1", base="main", title="T", body="B"
)
assert seen["url"] == "https://gitea.test/api/v1/repos/o/r/pulls"
assert seen["auth"] == "token tok"
assert seen["body"] == {
"head": "milestone-1",
"base": "main",
"title": "T",
"body": "B",
}
assert pr.number == 7
assert pr.head == "milestone-1"
def test_find_pull_request_matches_head_branch() -> None:
handler = lambda request: httpx.Response(200, json=[pr_json(3, "other")])
assert client(handler).find_pull_request("o/r", "milestone-1") is None
def test_find_pull_request_also_finds_closed() -> None:
# A crash after opening must not be answered with "nothing open, open
# another" — the effect already happened.
handler = lambda request: httpx.Response(
200, json=[pr_json(4, "milestone-1", "closed")]
)
found = client(handler).find_pull_request("o/r", "milestone-1")
assert found is not None
assert found.number == 4
def test_find_pull_request_pages_past_the_first_page() -> None:
# The match sits on page 2; sampling only the newest page would report
# "absent" and licence a duplicate PR.
def handler(request: httpx.Request) -> httpx.Response:
page = int(request.url.params.get("page", 1))
if page == 1:
return httpx.Response(
200, json=[pr_json(i, f"other-{i}") for i in range(PR_PAGE_SIZE)]
)
return httpx.Response(200, json=[pr_json(999, "milestone-1", "closed")])
found = client(handler).find_pull_request("o/r", "milestone-1")
assert found is not None
assert found.number == 999
def test_short_page_ends_the_search_definitively() -> None:
calls: list[int] = []
def handler(request: httpx.Request) -> httpx.Response:
calls.append(int(request.url.params.get("page", 1)))
return httpx.Response(200, json=[pr_json(1, "other")])
assert client(handler).find_pull_request("o/r", "milestone-1") is None
assert calls == [1] # one short page proves absence; no further paging
def test_unbounded_history_raises_rather_than_guessing() -> None:
# Never answer "absent" from an incomplete search: the reconciler turns
# this error into "unverifiable" and gates to the human.
handler = lambda request: httpx.Response(
200, json=[pr_json(i, f"other-{i}") for i in range(PR_PAGE_SIZE)]
)
with pytest.raises(GiteaError, match="without reaching the end"):
client(handler).find_pull_request("o/r", "milestone-1")
def test_error_status_raises_gitea_error() -> None:
handler = lambda request: httpx.Response(403, text="forbidden")
with pytest.raises(GiteaError, match="403"):
client(handler).whoami()
def test_list_and_post_comments() -> None:
def handler(request: httpx.Request) -> httpx.Response:
if request.method == "POST":
return httpx.Response(
201,
json={
"id": 9,
"user": {"login": "don"},
"body": "hi",
"created_at": "t",
},
)
return httpx.Response(
200,
json=[
{
"id": 8,
"user": {"login": "don"},
"body": "approve rev-1",
"created_at": "t",
}
],
)
c = client(handler)
assert c.list_comments("o/r", 1) == [
Comment(id=8, author="don", body="approve rev-1", created_at="t")
]
assert c.post_comment("o/r", 1, "hi").id == 9
def test_list_comments_passes_since() -> None:
seen: dict = {}
def handler(request: httpx.Request) -> httpx.Response:
seen["since"] = request.url.params.get("since")
return httpx.Response(200, json=[])
client(handler).list_comments("o/r", 1, since="2026-09-07T00:00:00Z")
assert seen["since"] == "2026-09-07T00:00:00Z"
def test_from_env_requires_config() -> None:
with pytest.raises(GiteaError, match="NISSE_GITEA_URL"):
GiteaClient.from_env({})
# --- reconciler (A3 external truth) --------------------------------------
def test_reconciler_reports_existing_pr_as_done() -> None:
handler = lambda request: httpx.Response(200, json=[pr_json(5, "milestone-1")])
assert pull_request_reconciler(client(handler), "o/r")(INTENT) is True
def test_reconciler_reports_missing_pr_as_not_done() -> None:
handler = lambda request: httpx.Response(200, json=[])
assert pull_request_reconciler(client(handler), "o/r")(INTENT) is False
def test_unreachable_gitea_is_unverifiable_not_a_guess() -> None:
def handler(request: httpx.Request) -> httpx.Response:
raise httpx.ConnectError("refused")
# None gates to the human: a delayed decision is cheap, a duplicate PR
# is not.
assert pull_request_reconciler(client(handler), "o/r")(INTENT) is None
def test_intent_without_a_branch_is_unverifiable() -> None:
handler = lambda request: httpx.Response(200, json=[])
blank = Intent(
id=2,
kind="open_pr",
payload={},
status="pending",
created_at="t",
resolved_at=None,
)
assert pull_request_reconciler(client(handler), "o/r")(blank) is None
+61 -3
View File
@@ -11,16 +11,17 @@ from nisse_dev.canonical import (
UserMessage, UserMessage,
) )
from nisse_dev.envfile import load_env_file from nisse_dev.envfile import load_env_file
from nisse_dev.llm import LiteLLMClient, LLMError, RoleMap from nisse_dev.llm import RATE_LIMIT_RETRY_S, LiteLLMClient, LLMError, RoleMap
EVENTS = [StoredEvent(id=1, ts="t", event=UserMessage(text="hi", source="m:1"))] EVENTS = [StoredEvent(id=1, ts="t", event=UserMessage(text="hi", source="m:1"))]
def make_client(handler) -> LiteLLMClient: def make_client(handler, sleep=lambda _s: None) -> LiteLLMClient:
return LiteLLMClient( return LiteLLMClient(
"http://litellm.test/v1", "http://litellm.test/v1",
"sk-test", "sk-test",
transport=httpx.MockTransport(handler), transport=httpx.MockTransport(handler),
sleep=sleep, # never wait in unit tests
) )
@@ -76,10 +77,67 @@ def test_tool_calls_get_locally_minted_canonical_ids() -> None:
assert event.call_id.startswith("call-") assert event.call_id.startswith("call-")
def test_truncated_tool_arguments_raise_llm_error() -> None:
message = {
"content": None,
"tool_calls": [
{
"id": "c",
"type": "function",
"function": {"name": "submit_plan", "arguments": '{"content": "'},
}
],
}
client = make_client(lambda request: completion(message))
with pytest.raises(LLMError, match="non-JSON arguments"):
client.complete(EVENTS, tools=None, model="local/qwen")
def test_non_object_tool_arguments_raise_llm_error() -> None:
message = {
"content": None,
"tool_calls": [
{
"id": "c",
"type": "function",
"function": {"name": "submit_plan", "arguments": '"just a string"'},
}
],
}
client = make_client(lambda request: completion(message))
with pytest.raises(LLMError, match="expected object"):
client.complete(EVENTS, tools=None, model="local/qwen")
def test_http_error_status_raises_llm_error() -> None: def test_http_error_status_raises_llm_error() -> None:
client = make_client(lambda request: httpx.Response(429, text="rate limited")) client = make_client(lambda request: httpx.Response(500, text="boom"))
with pytest.raises(LLMError, match="500"):
client.complete(EVENTS, tools=None, model="local/qwen")
def test_rate_limit_is_retried_once_then_surfaced() -> None:
# A 429 is transient. Losing a long delegation to one costs far more
# than waiting out the window, but retrying forever would hide a real
# ceiling, so it is retried exactly once.
attempts: list[int] = []
waited: list[float] = []
def handler(request: httpx.Request) -> httpx.Response:
attempts.append(1)
return httpx.Response(429, text="rate limited")
client = make_client(handler, sleep=waited.append)
with pytest.raises(LLMError, match="429"): with pytest.raises(LLMError, match="429"):
client.complete(EVENTS, tools=None, model="local/qwen") client.complete(EVENTS, tools=None, model="local/qwen")
assert len(attempts) == 2
assert waited == [RATE_LIMIT_RETRY_S]
def test_rate_limit_that_clears_succeeds() -> None:
responses = [httpx.Response(429, text="slow down"), completion({"content": "hi"})]
client = make_client(lambda request: responses.pop(0), sleep=lambda _s: None)
[event] = client.complete(EVENTS, tools=None, model="local/qwen")
assert isinstance(event, AssistantText)
def test_connection_failure_raises_llm_error() -> None: def test_connection_failure_raises_llm_error() -> None:
+281
View File
@@ -0,0 +1,281 @@
import io
from pathlib import Path
from typing import Any
import pytest
from nisse_dev.canonical import (
AssistantText,
CanonicalEvent,
PlanSubmitted,
StoredEvent,
ToolCall,
UserMessage,
)
from nisse_dev.control import (
AWAIT_ARCH_SIGNOFF,
AWAIT_MILESTONE_SIGNOFF,
AskHuman,
Orchestrator,
PlanningError,
SubmitPlan,
)
from nisse_dev.llm import RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.planner import PLANNER_TOOLS, LLMPlanningAgent
from nisse_dev.state import StateStore
from nisse_dev.transport import InMemoryTransport
ROLES = RoleMap(
executor="local/qwen",
planner="anthropic/claude-opus-5",
delegate="anthropic/claude-opus-5",
consultant="anthropic/claude-opus-5",
)
class ScriptedClient:
"""Returns canned canonical events per call, recording what it was asked."""
def __init__(self, turns: list[list[CanonicalEvent]]) -> None:
self._turns = turns
self.calls: list[dict[str, Any]] = []
def complete(
self,
events: list[StoredEvent],
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
) -> list[CanonicalEvent]:
self.calls.append(
{"events": events, "tools": tools, "model": model, "system": system}
)
if not self._turns:
return []
return self._turns.pop(0)
def plan_call(content: str, gate: str = "arch") -> ToolCall:
return ToolCall(
call_id="call-1",
tool="submit_plan",
arguments={"type": gate, "content": content},
)
@pytest.fixture
def store(tmp_path: Path) -> StateStore:
return StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
def test_submit_plan_tool_call_becomes_submit_plan(store: StateStore) -> None:
client = ScriptedClient([[plan_call("step 1: create src/app.py")]])
agent = LLMPlanningAgent(client, store, ROLES)
action = agent.plan("arch", "demo")
assert action == SubmitPlan("step 1: create src/app.py")
def test_ask_human_tool_call_becomes_ask_human(store: StateStore) -> None:
client = ScriptedClient(
[
[
ToolCall(
call_id="call-1",
tool="ask_human",
arguments={"question": "postgres or sqlite?"},
)
]
]
)
action = LLMPlanningAgent(client, store, ROLES).plan("arch", "demo")
assert action == AskHuman("postgres or sqlite?")
def test_planner_role_and_tools_are_used(store: StateStore) -> None:
client = ScriptedClient([[plan_call("plan")]])
LLMPlanningAgent(client, store, ROLES).plan("arch", "demo")
call = client.calls[0]
assert call["model"] == "anthropic/claude-opus-5" # never the executor
assert [t["function"]["name"] for t in call["tools"]] == [
"submit_plan",
"ask_human",
]
assert "demo" in call["system"] and "arch" in call["system"]
def test_no_proceed_tool_is_offered() -> None:
names = {tool["function"]["name"] for tool in PLANNER_TOOLS}
assert names == {"submit_plan", "ask_human"} # no self-advancement path
def test_model_output_is_appended_to_the_log(store: StateStore) -> None:
client = ScriptedClient(
[[AssistantText(text="thinking out loud", model="m"), plan_call("the plan")]]
)
LLMPlanningAgent(client, store, ROLES).plan("arch", "demo")
kinds = [type(e.event).__name__ for e in store.read_events()]
assert kinds == ["AssistantText", "ToolCall", "ToolResult"]
def test_gate_tool_call_is_answered_in_the_log(store: StateStore) -> None:
# submit_plan ends the turn and has no natural result, but a tool call
# left dangling makes the next render invalid for the provider.
client = ScriptedClient([[plan_call("the plan")]])
LLMPlanningAgent(client, store, ROLES).plan("arch", "demo")
events = [e.event for e in store.read_events()]
assert [type(e).__name__ for e in events] == ["ToolCall", "ToolResult"]
assert events[1].call_id == events[0].call_id
assert "posted for approval" in events[1].result
def test_prose_only_turn_retries_once_then_raises(store: StateStore) -> None:
client = ScriptedClient(
[
[AssistantText(text="here's what I think", model="m")],
[AssistantText(text="still prose", model="m")],
]
)
with pytest.raises(PlanningError, match="no submit_plan or ask_human"):
LLMPlanningAgent(client, store, ROLES).plan("arch", "demo")
assert len(client.calls) == 2
assert "did not call a tool" in client.calls[1]["system"]
def test_retry_succeeds_after_prose(store: StateStore) -> None:
client = ScriptedClient(
[[AssistantText(text="prose", model="m")], [plan_call("the real plan")]]
)
action = LLMPlanningAgent(client, store, ROLES).plan("arch", "demo")
assert action == SubmitPlan("the real plan")
def test_malformed_tool_arguments_are_rejected(store: StateStore) -> None:
blank = ToolCall(call_id="c", tool="submit_plan", arguments={"content": " "})
client = ScriptedClient([[blank], [blank]])
with pytest.raises(PlanningError):
LLMPlanningAgent(client, store, ROLES).plan("arch", "demo")
def test_planner_sees_prior_transcript(store: StateStore) -> None:
store.append_event(UserMessage(text="use demo", source="transport:@don"))
client = ScriptedClient([[plan_call("plan")]])
LLMPlanningAgent(client, store, ROLES).plan("arch", "demo")
seen = [e.event for e in client.calls[0]["events"]]
assert UserMessage(text="use demo", source="transport:@don") in seen
# --- orchestrator integration (still no network) -------------------------
def build(tmp_path: Path, client: ScriptedClient) -> tuple[Orchestrator, StateStore]:
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
transport = InMemoryTransport()
agent = LLMPlanningAgent(client, store, ROLES)
orch = Orchestrator(store, transport, agent, operators={"don"}, roles=ROLES)
orch.start()
return orch, store, transport
def test_use_project_produces_llm_plan_and_transcript(tmp_path: Path) -> None:
client = ScriptedClient([[plan_call("1. create app.py\n2. add a test")]])
_, store, transport = build(tmp_path, client)
transport.receive("don", "use demo")
request = transport.outbound[-1]
assert request.kind == "approval_request"
assert "create app.py" in request.text
events = [e.event for e in store.read_events()]
kinds = [type(e).__name__ for e in events]
assert kinds == [
"UserMessage",
"ToolCall",
"ToolResult",
"PlanSubmitted",
"GateOpened",
]
submitted = next(e for e in events if isinstance(e, PlanSubmitted))
assert submitted.model == "anthropic/claude-opus-5" # pinned planner
def test_roles_pinned_at_task_start(tmp_path: Path) -> None:
client = ScriptedClient([[plan_call("plan")]])
_, store, transport = build(tmp_path, client)
transport.receive("don", "use demo")
state = store.get_state()
assert state is not None
assert state.data["roles"]["planner"] == "anthropic/claude-opus-5"
def test_planning_failure_gates_to_await_input(tmp_path: Path) -> None:
prose = [AssistantText(text="prose", model="m")]
client = ScriptedClient([prose, prose])
_, store, transport = build(tmp_path, client)
transport.receive("don", "use demo")
state = store.get_state()
assert state is not None
assert state.state == "AWAIT_INPUT"
assert transport.outbound[-1].kind == "question"
def test_operator_can_recover_from_a_planning_failure(tmp_path: Path) -> None:
# Regression: AWAIT_INPUT reached by any path other than ask_human used to
# record no pending item, so every operator reply was refused with "no
# pending question" and the agent was stuck for good.
prose = [AssistantText(text="prose", model="m")]
client = ScriptedClient([prose, prose, [plan_call("recovered plan")]])
_, store, transport = build(tmp_path, client)
transport.receive("don", "use demo")
assert store.get_state().state == "AWAIT_INPUT"
transport.receive("don", "try again, keep it short")
assert store.get_state().state == AWAIT_ARCH_SIGNOFF
assert "recovered plan" in transport.outbound[-1].text
def test_operator_can_recover_after_a_stopped_milestone(tmp_path: Path) -> None:
class StoppedExecutor:
def run(self, milestone: int) -> Any:
from nisse_dev.executor import ExecutorOutcome
return ExecutorOutcome("escalate", "tests failed twice")
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
transport = InMemoryTransport()
client = ScriptedClient([[plan_call("p1")], [plan_call("p2")]])
orch = Orchestrator(
store,
transport,
LLMPlanningAgent(client, store, ROLES),
operators={"don"},
roles=ROLES,
executor=StoppedExecutor(),
)
orch.start()
transport.receive("don", "use demo")
transport.receive("don", "approve")
transport.receive("don", "approve")
transport.receive("don", "start milestone 1")
assert store.get_state().state == "AWAIT_INPUT"
transport.receive("don", "ok, I fixed the dependency")
assert store.get_state().state == AWAIT_MILESTONE_SIGNOFF
assert "start milestone 1" in transport.outbound[-1].text
def test_rejection_feedback_reaches_the_planner_via_the_log(
tmp_path: Path,
) -> None:
client = ScriptedClient([[plan_call("v1")], [plan_call("v2")]])
_, _store, transport = build(tmp_path, client)
transport.receive("don", "use demo")
transport.receive("don", "too vague, name the files")
replan_context = [e.event for e in client.calls[1]["events"]]
assert any(
isinstance(e, UserMessage) and "too vague" in e.text for e in replan_context
)
assert "v2" in transport.outbound[-1].text
+132
View File
@@ -0,0 +1,132 @@
import io
import json
import time
import pytest
from nisse_dev.gitea import Comment, GiteaError
from nisse_dev.oplog import OpLog
from nisse_dev.poller import MAX_BACKOFF_S, CommentPoller
class FakeGitea:
def __init__(self, pages: list[list[Comment]] | None = None) -> None:
self.pages = pages or []
self.calls = 0
self.fail_times = 0
def list_comments(self, repo: str, index: int, since: str | None = None):
self.calls += 1
if self.fail_times > 0:
self.fail_times -= 1
raise GiteaError("gitea unreachable")
if not self.pages:
return []
return self.pages.pop(0)
def comment(cid: int, author: str = "don", body: str = "please fix") -> Comment:
return Comment(id=cid, author=author, body=body, created_at="t")
def build(client, **kw) -> tuple[CommentPoller, list[Comment]]:
seen: list[Comment] = []
poller = CommentPoller(client, "o/r", seen.append, **kw)
return poller, seen
def test_new_comments_are_delivered_once() -> None:
client = FakeGitea([[comment(1)], [comment(1), comment(2)]])
poller, seen = build(client)
poller.poll_once(1)
poller.poll_once(1)
assert [c.id for c in seen] == [1, 2] # id 1 not redelivered
def test_prime_suppresses_the_existing_backlog() -> None:
# Starting a watch must not replay feedback already addressed.
client = FakeGitea([[comment(1), comment(2)], [comment(1), comment(2), comment(3)]])
poller, seen = build(client)
poller.prime(1)
poller.poll_once(1)
assert [c.id for c in seen] == [3]
def test_own_comments_are_ignored() -> None:
client = FakeGitea([[comment(1, author="nisse-dev-testing"), comment(2, "don")]])
poller, seen = build(client, ignore_authors={"nisse-dev-testing"})
poller.poll_once(1)
assert [c.id for c in seen] == [2]
def test_ignored_comments_are_not_redelivered_later() -> None:
client = FakeGitea(
[
[comment(1, author="bot")],
[comment(1, author="bot"), comment(2, "don")],
]
)
poller, seen = build(client, ignore_authors={"bot"})
poller.poll_once(1)
poller.poll_once(1)
assert [c.id for c in seen] == [2]
def test_outage_does_not_raise_out_of_the_loop() -> None:
stream = io.StringIO()
client = FakeGitea([[comment(1)]])
client.fail_times = 2
poller, seen = build(
client,
oplog=OpLog("nisse-dev", stream),
interval_s=0.01,
initial_backoff_s=0.01,
)
poller.start(1)
deadline = time.monotonic() + 5
while not seen and time.monotonic() < deadline:
time.sleep(0.05)
poller.stop()
# It recovered and delivered rather than dying on the outage (B2: an
# outage delays a reaction, never fails a task).
assert [c.id for c in seen] == [1]
assert "gitea.poll_failed" in stream.getvalue()
def test_prime_survives_an_outage() -> None:
client = FakeGitea()
client.fail_times = 1
poller, _ = build(client)
poller.prime(1) # must not raise
def test_backoff_grows_then_caps() -> None:
client = FakeGitea()
client.fail_times = 50
stream = io.StringIO()
poller, _ = build(
client,
oplog=OpLog("nisse-dev", stream),
interval_s=0.001,
initial_backoff_s=0.001,
max_backoff_s=0.008,
)
poller.start(1)
time.sleep(0.3)
poller.stop()
delays = [json.loads(line)["retry_in_s"] for line in stream.getvalue().splitlines()]
assert delays[:4] == [0.001, 0.002, 0.004, 0.008]
assert max(delays) == 0.008 # capped, and retried indefinitely (B2)
assert MAX_BACKOFF_S <= 300.0 # the shipped cap is "a few minutes"
@pytest.mark.parametrize("author", ["don", "someone-else"])
def test_poller_does_not_filter_by_author_itself(author: str) -> None:
# Authorisation belongs to the orchestrator's allowlist, not here: the
# poller is pure intake.
client = FakeGitea([[comment(1, author=author)]])
poller, seen = build(client)
poller.poll_once(1)
assert len(seen) == 1
+196
View File
@@ -0,0 +1,196 @@
"""The B1 loop past EXECUTING: publish, review gate, then next milestone or
project completion."""
import io
from pathlib import Path
from typing import Any
import pytest
from nisse_dev.canonical import AssistantText, CanonicalEvent, StoredEvent, ToolCall
from nisse_dev.control import (
AWAIT_INPUT,
EXECUTING,
IDLE,
PR_OPEN,
Orchestrator,
)
from nisse_dev.executor import ExecutorOutcome
from nisse_dev.gitea import GiteaError, PullRequest
from nisse_dev.llm import RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.planner import LLMPlanningAgent
from nisse_dev.publish import PublishResult
from nisse_dev.state import StateStore
from nisse_dev.transport import InMemoryTransport
ROLES = RoleMap(
executor="llama3.2",
planner="haiku",
delegate="haiku",
consultant="haiku",
)
class ScriptedClient:
def __init__(self, plans: list[str]) -> None:
self._plans = plans
self.calls: list[dict[str, Any]] = []
def complete(
self,
events: list[StoredEvent],
tools: list[dict[str, Any]] | None,
model: str,
system: str | None = None,
) -> list[CanonicalEvent]:
self.calls.append({"system": system})
if not self._plans:
return [AssistantText(text="no more plans", model=model)]
return [
ToolCall(
call_id=f"c{len(self.calls)}",
tool="submit_plan",
arguments={"type": "x", "content": self._plans.pop(0)},
)
]
class DoneExecutor:
def run(self, milestone: int) -> ExecutorOutcome:
return ExecutorOutcome("completed", f"milestone {milestone} done")
class FakePublisher:
def __init__(
self, *, result: PublishResult | None = None, error: Exception | None = None
) -> None:
self._result = result or PublishResult(
PullRequest(number=7, state="open", head="h", url="https://g/pulls/7"),
"opened PR #7: https://g/pulls/7",
)
self._error = error
self.calls: list[dict[str, Any]] = []
def reconcilers(self) -> dict[str, Any]:
return {"open_pr": lambda intent: True}
def publish(self, **kwargs: Any) -> PublishResult:
self.calls.append(kwargs)
if self._error is not None:
raise self._error
return self._result
def build(
tmp_path: Path, publisher: Any, plans: list[str] | None = None
) -> tuple[StateStore, InMemoryTransport]:
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
transport = InMemoryTransport()
client = ScriptedClient(plans or ["arch plan", "m1 plan", "m2 plan"])
orch = Orchestrator(
store,
transport,
LLMPlanningAgent(client, store, ROLES),
operators={"don"},
roles=ROLES,
executor=DoneExecutor(),
publisher=publisher,
)
orch.start()
return store, transport
def run_to_pr(tmp_path: Path, publisher: Any) -> tuple[StateStore, InMemoryTransport]:
store, transport = build(tmp_path, publisher)
transport.receive("don", "use demo")
transport.receive("don", "approve") # arch
transport.receive("don", "approve") # milestone plan
transport.receive("don", "start milestone 1")
return store, transport
def test_completed_milestone_publishes_and_enters_pr_open(tmp_path: Path) -> None:
publisher = FakePublisher()
store, transport = run_to_pr(tmp_path, publisher)
assert store.get_state().state == PR_OPEN
assert publisher.calls[0]["milestone"] == 1
assert publisher.calls[0]["project"] == "demo"
assert "m1 plan" in publisher.calls[0]["plan"] # the approved plan travels
assert "PR #7" in transport.outbound[-1].text
def test_pr_open_records_the_pull_request(tmp_path: Path) -> None:
store, _ = run_to_pr(tmp_path, FakePublisher())
assert store.get_state().data["pr"]["number"] == 7
def test_next_milestone_replans_and_bumps_the_counter(tmp_path: Path) -> None:
store, transport = run_to_pr(tmp_path, FakePublisher())
transport.receive("don", "next milestone")
state = store.get_state()
assert state.state != PR_OPEN
assert state.data["milestone"] == 2
assert "m2 plan" in transport.outbound[-1].text
def test_project_complete_returns_to_idle(tmp_path: Path) -> None:
store, transport = run_to_pr(tmp_path, FakePublisher())
transport.receive("don", "project complete")
state = store.get_state()
assert state.state == IDLE
assert state.project is None
assert "project closed" in transport.outbound[-1].text
def test_unknown_command_in_pr_open_explains_the_options(tmp_path: Path) -> None:
store, transport = run_to_pr(tmp_path, FakePublisher())
transport.receive("don", "looks good!")
assert store.get_state().state == PR_OPEN
assert "next milestone" in transport.outbound[-1].text
def test_no_changes_reports_without_entering_pr_open(tmp_path: Path) -> None:
publisher = FakePublisher(result=PublishResult(None, "no changes to publish"))
store, transport = run_to_pr(tmp_path, publisher)
assert store.get_state().state == EXECUTING
assert transport.outbound[-1].text == "no changes to publish"
def test_publish_failure_gates_to_the_operator(tmp_path: Path) -> None:
publisher = FakePublisher(error=GiteaError("gitea is down"))
store, transport = run_to_pr(tmp_path, publisher)
assert store.get_state().state == AWAIT_INPUT
assert "could not publish" in transport.outbound[-1].text
# And the operator can still get back out (the deadlock regression).
transport.receive("don", "gitea is back")
assert store.get_state().state != AWAIT_INPUT
def test_publisher_reconciler_is_registered_at_boot(tmp_path: Path) -> None:
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
store.begin_intent("open_pr", {"branch": "b"}, cause="crash")
transport = InMemoryTransport()
orch = Orchestrator(
store,
transport,
LLMPlanningAgent(ScriptedClient([]), store, ROLES),
operators={"don"},
roles=ROLES,
publisher=FakePublisher(),
)
orch.start()
# The publisher's reconciler answered "already opened", so boot resumed
# instead of gating on an unknown intent kind.
assert store.pending_intents() == []
assert store.get_state().state != AWAIT_INPUT
@pytest.mark.parametrize("command", ["next milestone", "project complete"])
def test_pr_open_commands_are_operator_only(tmp_path: Path, command: str) -> None:
store, transport = run_to_pr(tmp_path, FakePublisher())
transport.receive("mallory", command)
assert store.get_state().state == PR_OPEN
+158
View File
@@ -0,0 +1,158 @@
import io
from pathlib import Path
from typing import Any
import pytest
from nisse_dev.git import GitRepo
from nisse_dev.gitea import GiteaError, PullRequest
from nisse_dev.oplog import OpLog
from nisse_dev.publish import OPEN_PR_INTENT, MilestonePublisher
from nisse_dev.state import StateStore
class FakeGitea:
def __init__(self, *, fail: bool = False) -> None:
self.opened: list[dict[str, Any]] = []
self.existing: list[str] = []
self._fail = fail
def open_pull_request(
self, repo: str, *, head: str, base: str, title: str, body: str
) -> PullRequest:
if self._fail:
raise GiteaError("gitea is down")
self.opened.append(
{"repo": repo, "head": head, "base": base, "title": title, "body": body}
)
self.existing.append(head)
return PullRequest(
number=len(self.opened), state="open", head=head, url=f"u/{head}"
)
def find_pull_request(self, repo: str, head: str) -> PullRequest | None:
if head in self.existing:
return PullRequest(number=1, state="open", head=head, url="u")
return None
class FakeGit:
"""Records what would have been pushed; no network."""
def __init__(self, root: Path, *, changes: bool = True) -> None:
self.repo = GitRepo(root)
self.repo.init()
self.repo.configure_identity("bot", "bot@example.test")
if changes:
(root / "work.txt").write_text("done")
self.pushed: list[str] = []
def current_branch(self) -> str:
return self.repo.current_branch()
def checkout_new_branch(self, name: str) -> None:
self.repo.checkout_new_branch(name)
def commit_all(self, message: str) -> str | None:
return self.repo.commit_all(message)
def push(self, remote_url: str, branch: str, *, token: str | None = None) -> None:
self.pushed.append(branch)
@pytest.fixture
def store(tmp_path: Path) -> StateStore:
return StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
def build(store: StateStore, git: Any, client: Any) -> MilestonePublisher:
return MilestonePublisher(
store,
client,
"o/sandbox",
git,
base_url="https://gitea.test",
account="bot",
token="tok",
)
def test_publish_commits_pushes_and_opens_a_pr(
store: StateStore, tmp_path: Path
) -> None:
git = FakeGit(tmp_path / "work")
client = FakeGitea()
result = build(store, git, client).publish(
task_id="demo-abc", milestone=2, project="demo", plan="the plan"
)
assert git.pushed == ["demo-abc/milestone-2"]
assert client.opened[0]["head"] == "demo-abc/milestone-2"
assert client.opened[0]["base"] == "main"
assert "the plan" in client.opened[0]["body"]
assert result.pull_request is not None
assert "opened PR #1" in result.note
def test_intent_is_recorded_then_committed(store: StateStore, tmp_path: Path) -> None:
build(store, FakeGit(tmp_path / "work"), FakeGitea()).publish(
task_id="t", milestone=1, project="demo", plan="p"
)
# Nothing left pending: the intent opened before the push was settled
# once the PR existed.
assert store.pending_intents() == []
def test_failure_after_intent_leaves_it_pending_for_reconciliation(
store: StateStore, tmp_path: Path
) -> None:
publisher = build(store, FakeGit(tmp_path / "work"), FakeGitea(fail=True))
with pytest.raises(GiteaError):
publisher.publish(task_id="t", milestone=1, project="demo", plan="p")
[pending] = store.pending_intents()
assert pending.kind == OPEN_PR_INTENT
assert pending.payload["branch"] == "t/milestone-1"
assert pending.payload["sha"]
def test_nothing_to_commit_is_reported_not_raised(
store: StateStore, tmp_path: Path
) -> None:
git = FakeGit(tmp_path / "work", changes=False)
client = FakeGitea()
result = build(store, git, client).publish(
task_id="t", milestone=1, project="demo", plan="p"
)
assert result.pull_request is None
assert result.note == "no changes to publish"
assert client.opened == []
# No externally visible act was attempted, so no intent was opened.
assert store.pending_intents() == []
def test_branch_is_namespaced_by_task(store: StateStore, tmp_path: Path) -> None:
publisher = build(store, FakeGit(tmp_path / "work"), FakeGitea())
assert publisher.branch_for("greet-9f2", 3) == "greet-9f2/milestone-3"
def test_reconciler_settles_an_intent_left_by_a_crash(
store: StateStore, tmp_path: Path
) -> None:
# Simulate the dangerous window: the PR was opened, but the process died
# before the intent could be committed. On the next boot the reconciler
# must say "already done" so the milestone is not published twice.
client = FakeGitea()
publisher = build(store, FakeGit(tmp_path / "work"), client)
intent_id = store.begin_intent(
OPEN_PR_INTENT, {"branch": "t/milestone-1"}, cause="crash simulation"
)
client.existing.append("t/milestone-1")
reconcilers = publisher.reconcilers()
assert set(reconcilers) == {OPEN_PR_INTENT}
[pending] = store.pending_intents()
assert reconcilers[OPEN_PR_INTENT](pending) is True
store.commit_intent(intent_id, cause="verified")
assert client.opened == [] # nothing re-published
+27
View File
@@ -71,6 +71,33 @@ def test_compaction_replaces_its_range_only() -> None:
assert contents == ["kept", "[summary of earlier conversation]\nthe old stuff"] assert contents == ["kept", "[summary of earlier conversation]\nthe old stuff"]
def test_tool_call_without_a_result_gets_a_placeholder() -> None:
# Providers reject a dangling tool_use; this happens after a crash (A3)
# or with turn-ending gate tools like submit_plan.
events = stored(
[
UserMessage(text="go", source="m:1"),
ToolCall(call_id="call-x", tool="submit_plan", arguments={}),
]
)
body = OpenAIChatRenderer().render(events, tools=None, model="q")
assert [m["role"] for m in body["messages"]] == ["user", "assistant", "tool"]
assert body["messages"][2]["tool_call_id"] == "call-x"
assert body["messages"][2]["content"] == "[no result recorded]"
def test_answered_tool_call_gets_no_placeholder() -> None:
events = stored(
[
ToolCall(call_id="call-x", tool="run_command", arguments={}),
ToolResult(call_id="call-x", result="ok", success=True),
]
)
body = OpenAIChatRenderer().render(events, tools=None, model="q")
assert [m["role"] for m in body["messages"]] == ["assistant", "tool"]
assert body["messages"][1]["content"] == "ok"
def test_gate_event_renders_as_context() -> None: def test_gate_event_renders_as_context() -> None:
events = stored([GateOpened(gate="arch", reason="plan submitted (rev-1)")]) events = stored([GateOpened(gate="arch", reason="plan submitted (rev-1)")])
body = OpenAIChatRenderer().render(events, tools=None, model="q") body = OpenAIChatRenderer().render(events, tools=None, model="q")
+196
View File
@@ -0,0 +1,196 @@
"""The Anthropic Messages shape differs structurally from the OpenAI one;
these tests pin the differences that a normalising middlebox would blur."""
import httpx
import pytest
from nisse_dev.canonical import (
AssistantText,
Compaction,
StoredEvent,
ToolCall,
ToolResult,
UserMessage,
)
from nisse_dev.llm import AnthropicPassthroughClient, LLMError
from nisse_dev.render_anthropic import (
AnthropicMessagesRenderer,
to_anthropic_tools,
)
TOOLS = [
{
"type": "function",
"function": {
"name": "run_command",
"description": "Run a shell command.",
"parameters": {
"type": "object",
"properties": {"command": {"type": "string"}},
"required": ["command"],
},
},
}
]
def stored(events: list) -> list[StoredEvent]:
return [StoredEvent(id=i + 1, ts="t", event=e) for i, e in enumerate(events)]
def render(events: list, **kw) -> dict:
return AnthropicMessagesRenderer().render(
stored(events), kw.pop("tools", None), "claude-haiku-4-5", **kw
)
def test_system_is_a_top_level_field_not_a_message() -> None:
body = render([UserMessage(text="hi", source="m")], system="you are nisse")
assert body["system"] == "you are nisse"
assert all(m["role"] != "system" for m in body["messages"])
def test_max_tokens_is_always_present() -> None:
# The Messages API rejects a request without it.
assert render([UserMessage(text="hi", source="m")])["max_tokens"] > 0
def test_tool_call_is_a_content_block_not_an_array() -> None:
body = render(
[ToolCall(call_id="call-1", tool="run_command", arguments={"command": "ls"})]
)
block = body["messages"][0]["content"][0]
assert block["type"] == "tool_use"
assert block["id"] == "call-1"
assert block["input"] == {"command": "ls"}
assert "tool_calls" not in body["messages"][0]
def test_tool_result_is_a_user_block_directly_after_the_call() -> None:
body = render(
[
ToolCall(call_id="call-1", tool="run_command", arguments={}),
ToolResult(call_id="call-1", result="exit code: 0", success=True),
]
)
assistant, user = body["messages"]
assert assistant["role"] == "assistant"
assert assistant["content"][0]["type"] == "tool_use"
assert user["role"] == "user" # not a "tool" role, as OpenAI would have
assert user["content"][0] == {
"type": "tool_result",
"tool_use_id": "call-1",
"content": "exit code: 0",
}
def test_adjacent_same_role_events_are_merged() -> None:
body = render(
[
UserMessage(text="one", source="m"),
UserMessage(text="two", source="m"),
AssistantText(text="reply", model="m"),
]
)
assert [m["role"] for m in body["messages"]] == ["user", "assistant"]
assert [b["text"] for b in body["messages"][0]["content"]] == ["one", "two"]
def test_dangling_tool_call_gets_a_placeholder_result() -> None:
body = render([ToolCall(call_id="call-x", tool="submit_plan", arguments={})])
assert [m["role"] for m in body["messages"]] == ["assistant", "user"]
assert body["messages"][1]["content"][0]["tool_use_id"] == "call-x"
def test_tools_are_converted_to_input_schema() -> None:
[tool] = to_anthropic_tools(TOOLS)
assert tool["name"] == "run_command"
assert tool["input_schema"]["properties"] == {"command": {"type": "string"}}
assert "parameters" not in tool
assert "function" not in tool
def test_compaction_still_replaces_its_range() -> None:
events = stored(
[
UserMessage(text="old", source="m"),
UserMessage(text="kept", source="m"),
]
)
events.append(
StoredEvent(
id=3,
ts="t",
event=Compaction(summary="summary", model="m", replaces=[1, 1]),
)
)
body = AnthropicMessagesRenderer().render(events, None, "claude-haiku-4-5")
texts = [b["text"] for m in body["messages"] for b in m["content"]]
assert "old" not in texts
assert any("summary" in t for t in texts)
# --- passthrough client --------------------------------------------------
def client(handler) -> AnthropicPassthroughClient:
return AnthropicPassthroughClient(
"https://litellm.test", "tok", transport=httpx.MockTransport(handler)
)
def test_client_posts_to_the_messages_route_with_version_header() -> None:
seen: dict = {}
def handler(request: httpx.Request) -> httpx.Response:
seen["url"] = str(request.url)
seen["version"] = request.headers.get("anthropic-version")
return httpx.Response(
200,
json={
"model": "claude-haiku-4-5",
"content": [{"type": "text", "text": "hi"}],
},
)
events = stored([UserMessage(text="hello", source="m")])
[event] = client(handler).complete(events, None, "claude-haiku-4-5")
assert seen["url"] == "https://litellm.test/v1/messages"
assert seen["version"] == "2023-06-01"
assert event == AssistantText(text="hi", model="claude-haiku-4-5")
def test_client_maps_tool_use_to_local_call_ids() -> None:
response = {
"model": "claude-haiku-4-5",
"content": [
{"type": "text", "text": "running it"},
{
"type": "tool_use",
"id": "toolu_vendor_xyz",
"name": "run_command",
"input": {"command": "pytest"},
},
],
}
events = stored([UserMessage(text="go", source="m")])
text, call = client(lambda r: httpx.Response(200, json=response)).complete(
events, TOOLS, "claude-haiku-4-5"
)
assert isinstance(text, AssistantText)
assert isinstance(call, ToolCall)
assert call.arguments == {"command": "pytest"}
assert call.call_id != "toolu_vendor_xyz" # provider id never a join key
assert call.call_id.startswith("call-")
def test_client_errors_are_llm_errors() -> None:
events = stored([UserMessage(text="go", source="m")])
with pytest.raises(LLMError, match="400"):
client(lambda r: httpx.Response(400, text="bad")).complete(
events, None, "claude-haiku-4-5"
)
with pytest.raises(LLMError, match="malformed"):
client(lambda r: httpx.Response(200, json={"oops": 1})).complete(
events, None, "claude-haiku-4-5"
)
+184
View File
@@ -0,0 +1,184 @@
"""Reacting to PR review comments while in PR_OPEN (spec B2)."""
import io
from pathlib import Path
from typing import Any
from nisse_dev.canonical import CanonicalEvent, StoredEvent, ToolCall
from nisse_dev.control import AWAIT_INPUT, PR_OPEN, Orchestrator
from nisse_dev.executor import ExecutorOutcome
from nisse_dev.gitea import Comment, GiteaError, PullRequest
from nisse_dev.llm import RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.planner import LLMPlanningAgent
from nisse_dev.publish import PublishResult
from nisse_dev.state import StateStore
from nisse_dev.transport import InMemoryTransport
ROLES = RoleMap(executor="x", planner="y", delegate="y", consultant="y")
class ScriptedClient:
def __init__(self) -> None:
self.calls = 0
def complete(self, events, tools, model, system=None) -> list[CanonicalEvent]:
self.calls += 1
return [
ToolCall(
call_id=f"c{self.calls}",
tool="submit_plan",
arguments={"type": "x", "content": f"plan {self.calls}"},
)
]
class RecordingExecutor:
def __init__(self, outcomes: list[str] | None = None) -> None:
self.runs: list[int] = []
self.seen_events: list[list[StoredEvent]] = []
# One entry per run, so a test can let the milestone succeed and
# only the review response fail.
self._outcomes = outcomes or []
self.store: StateStore | None = None
def run(self, milestone: int) -> ExecutorOutcome:
self.runs.append(milestone)
if self.store is not None:
self.seen_events.append(self.store.read_events())
status = self._outcomes.pop(0) if self._outcomes else "completed"
return ExecutorOutcome(status, f"executor said {status}")
class RecordingPublisher:
def __init__(self, *, sha: str | None = "abc12345", error: Exception | None = None):
self._sha = sha
self._error = error
self.followups: list[dict[str, Any]] = []
self.comments: list[tuple[int, str]] = []
def reconcilers(self) -> dict[str, Any]:
return {}
def publish(self, **kwargs: Any) -> PublishResult:
return PublishResult(
PullRequest(number=42, state="open", head="h", url="u"), "opened PR #42: u"
)
def push_followup(self, **kwargs: Any) -> str | None:
self.followups.append(kwargs)
if self._error is not None:
raise self._error
return self._sha
def comment(self, index: int, body: str) -> None:
self.comments.append((index, body))
def build(
tmp_path: Path,
executor: Any,
publisher: Any,
gitea_operators: set[str] | None = None,
) -> tuple[Orchestrator, StateStore, InMemoryTransport]:
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
if hasattr(executor, "store"):
executor.store = store
transport = InMemoryTransport()
orch = Orchestrator(
store,
transport,
LLMPlanningAgent(ScriptedClient(), store, ROLES),
operators={"don"},
roles=ROLES,
executor=executor,
publisher=publisher,
gitea_operators=gitea_operators if gitea_operators is not None else {"don"},
)
orch.start()
transport.receive("don", "use demo")
transport.receive("don", "approve")
transport.receive("don", "approve")
transport.receive("don", "start milestone 1")
return orch, store, transport
def review(
cid: int = 5, author: str = "don", body: str = "rename the module"
) -> Comment:
return Comment(id=cid, author=author, body=body, created_at="t")
def test_review_comment_triggers_a_fix_and_a_push(tmp_path: Path) -> None:
executor, publisher = RecordingExecutor(), RecordingPublisher()
orch, store, transport = build(tmp_path, executor, publisher)
assert store.get_state().state == PR_OPEN
orch.handle_review_comment(review())
assert executor.runs == [1, 1] # milestone run, then the review response
assert publisher.followups[0]["milestone"] == 1
assert "abc12345" in transport.outbound[-1].text
assert publisher.comments[0][0] == 42 # replied on the PR
def test_comment_enters_the_transcript_before_the_executor_runs(
tmp_path: Path,
) -> None:
executor, publisher = RecordingExecutor(), RecordingPublisher()
orch, _, _ = build(tmp_path, executor, publisher)
orch.handle_review_comment(review(body="please add a test"))
# The executor must see the feedback as conversation, not as an
# out-of-band instruction it was never shown.
texts = [getattr(e.event, "text", "") for e in executor.seen_events[-1]]
assert any("please add a test" in t for t in texts)
def test_non_operator_comments_are_ignored(tmp_path: Path) -> None:
executor, publisher = RecordingExecutor(), RecordingPublisher()
orch, store, _ = build(tmp_path, executor, publisher)
orch.handle_review_comment(review(author="drive-by"))
assert executor.runs == [1] # only the original milestone run
assert publisher.followups == []
assert store.get_state().state == PR_OPEN
def test_comments_outside_pr_open_are_ignored(tmp_path: Path) -> None:
executor, publisher = RecordingExecutor(), RecordingPublisher()
orch, _store, transport = build(tmp_path, executor, publisher)
transport.receive("don", "project complete") # back to IDLE
orch.handle_review_comment(review())
assert executor.runs == [1]
def test_failed_review_response_gates_to_the_operator(tmp_path: Path) -> None:
# Milestone succeeds; the response to the review comment does not.
executor = RecordingExecutor(outcomes=["completed", "escalate"])
publisher = RecordingPublisher()
orch, store, transport = build(tmp_path, executor, publisher)
orch.handle_review_comment(review())
assert store.get_state().state == AWAIT_INPUT
assert "could not address the review comment" in transport.outbound[-1].text
assert publisher.followups == []
def test_no_code_change_needed_is_reported(tmp_path: Path) -> None:
executor, publisher = RecordingExecutor(), RecordingPublisher(sha=None)
orch, _, transport = build(tmp_path, executor, publisher)
orch.handle_review_comment(review(body="just curious, why sqlite?"))
assert "no code change was needed" in transport.outbound[-1].text
def test_push_failure_is_reported_without_losing_pr_open(tmp_path: Path) -> None:
executor = RecordingExecutor()
publisher = RecordingPublisher(error=GiteaError("push rejected"))
orch, store, transport = build(tmp_path, executor, publisher)
orch.handle_review_comment(review())
assert "could not push the fix" in transport.outbound[-1].text
assert store.get_state().state == PR_OPEN # still reviewable, can retry
+158
View File
@@ -0,0 +1,158 @@
import os
import time
from pathlib import Path
import pytest
from nisse_dev.tools import EXECUTOR_TOOLS, MAX_OUTPUT_CHARS, Workspace
@pytest.fixture
def ws(tmp_path: Path) -> Workspace:
return Workspace(tmp_path / "work", timeout_s=10)
def test_manifest_is_small_and_flat() -> None:
# A15: fewer tools, flatter arguments for the local executor.
names = [t["function"]["name"] for t in EXECUTOR_TOOLS]
assert names == ["read_file", "write_file", "list_dir", "run_command"]
for tool in EXECUTOR_TOOLS:
props = tool["function"]["parameters"]["properties"]
assert all(p["type"] == "string" for p in props.values())
def test_write_then_read_round_trip(ws: Workspace) -> None:
assert ws.write_file("src/app.py", "print('hi')\n").success
outcome = ws.read_file("src/app.py")
assert outcome.success
assert outcome.result == "print('hi')\n"
def test_relative_paths_land_in_the_workspace(ws: Workspace) -> None:
ws.write_file("notes.txt", "x")
assert (ws.root / "notes.txt").exists()
def test_read_missing_file_is_a_failed_result_not_an_exception(
ws: Workspace,
) -> None:
outcome = ws.read_file("nope.txt")
assert not outcome.success
assert "could not read" in outcome.result
def test_list_dir(ws: Workspace) -> None:
ws.write_file("a.txt", "a")
ws.write_file("sub/b.txt", "b")
outcome = ws.list_dir(".")
assert outcome.success
assert outcome.result.splitlines() == ["a.txt", "sub/"]
def test_list_empty_dir(ws: Workspace) -> None:
assert ws.list_dir(".").result == "(empty)"
def test_run_command_success_reports_exit_code_and_stdout(ws: Workspace) -> None:
outcome = ws.run_command("echo hello")
assert outcome.success
assert "exit code: 0" in outcome.result
assert "hello" in outcome.result
def test_nonzero_exit_is_a_result_not_an_error(ws: Workspace) -> None:
outcome = ws.run_command("exit 3")
assert not outcome.success # feeds the A12 failure count
assert "exit code: 3" in outcome.result
def test_command_runs_in_the_workspace(ws: Workspace) -> None:
ws.write_file("marker.txt", "here")
assert "marker.txt" in ws.run_command("ls").result
def test_command_timeout_is_reported(tmp_path: Path) -> None:
ws = Workspace(tmp_path / "work", timeout_s=1)
outcome = ws.run_command("sleep 30")
assert not outcome.success
assert "timed out" in outcome.result
def _is_running(pid: int) -> bool:
"""os.kill(pid, 0) is not enough: a killed orphan stays a zombie until
something reaps it, and no init does that inside a CI container. Read the
process state instead."""
try:
stat = Path(f"/proc/{pid}/stat").read_text()
except FileNotFoundError:
return False
# State is the first field after the parenthesised comm.
return stat.rsplit(")", 1)[1].split()[0] != "Z"
def test_timeout_kills_the_whole_process_group(tmp_path: Path) -> None:
ws = Workspace(tmp_path / "work", timeout_s=1)
pidfile = ws.root / "child.pid"
# A backgrounded grandchild must not outlive the timeout.
ws.run_command(f"(sleep 30 & echo $! > {pidfile}); sleep 30")
assert pidfile.exists()
pid = int(pidfile.read_text().strip())
assert _is_running(os.getpid()), "liveness check is broken, not the kill"
# SIGKILL to a process group is asynchronous, and the grandchild is an
# orphan by now, so its exit is not synchronised with our return. Wait
# for death rather than assuming it is instantaneous.
deadline = time.monotonic() + 5
while _is_running(pid) and time.monotonic() < deadline:
time.sleep(0.05)
assert not _is_running(pid)
def test_agent_credentials_are_not_inherited(
ws: Workspace, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setenv("NISSE_LLM_TOKEN", "sk-secret")
monkeypatch.setenv("PATH", os.environ["PATH"])
outcome = ws.run_command("echo [${NISSE_LLM_TOKEN}]")
assert "sk-secret" not in outcome.result
assert "[]" in outcome.result
def test_stdin_is_closed_so_prompts_fail_fast(ws: Workspace) -> None:
outcome = ws.run_command("read line && echo got")
assert not outcome.success
assert "got" not in outcome.result
def test_long_output_is_truncated(ws: Workspace) -> None:
outcome = ws.run_command("printf 'x%.0s' $(seq 1 20000)")
assert len(outcome.result) < MAX_OUTPUT_CHARS + 200
assert "truncated" in outcome.result
def test_dispatch_unknown_tool_and_missing_argument(ws: Workspace) -> None:
assert not ws.dispatch("fly", {}).success
missing = ws.dispatch("read_file", {})
assert not missing.success
assert "missing argument" in missing.result
def test_write_outside_the_workspace_is_refused(ws: Workspace, tmp_path: Path) -> None:
# Not sandboxing: a file written outside the repo is invisible to git, so
# the work would vanish while the model truthfully reported success.
# Models really do invent absolute paths like /workspace.
stray = tmp_path / "elsewhere" / "slugify.py"
outcome = ws.write_file(str(stray), "x")
assert not outcome.success
assert "outside the workspace" in outcome.result
assert str(ws.root) in outcome.result # tells it where to write instead
assert not stray.exists()
def test_escaping_via_parent_traversal_is_refused(ws: Workspace) -> None:
outcome = ws.write_file("../escaped.txt", "x")
assert not outcome.success
assert not (ws.root.parent / "escaped.txt").exists()
def test_describe_names_the_root(ws: Workspace) -> None:
assert str(ws.root) in ws.describe()
+89
View File
@@ -0,0 +1,89 @@
"""A real delegation over the Anthropic passthrough: scoped sub-task in,
working code and a structured handback out."""
import io
import os
from pathlib import Path
import pytest
from nisse_dev.delegation import Caps, Delegation
from nisse_dev.envfile import load_env_file
from nisse_dev.git import GitRepo
from nisse_dev.llm import AnthropicPassthroughClient
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
from nisse_dev.tools import Workspace
load_env_file(Path(__file__).resolve().parents[2] / ".env")
SENTINEL = "NISSE_LIVE_PLANNER_MODEL"
pytestmark = [
pytest.mark.integration,
pytest.mark.skipif(
not os.environ.get(SENTINEL),
reason=f"no live frontier route configured ({SENTINEL} unset)",
),
]
BRIEF = """Create `slugify.py` in the workspace root containing a function
`slugify(text)` that lowercases the text and replaces every run of
non-alphanumeric characters with a single hyphen, with no leading or trailing
hyphen. Then create `test_slugify.py` with at least two assertions covering
it, and run `python -m pytest test_slugify.py -q` until it passes."""
def build(tmp_path: Path, caps: Caps) -> tuple[Delegation, GitRepo, StateStore]:
repo = GitRepo(tmp_path / "work")
repo.init()
repo.configure_identity("nisse-dev-testing", "n@t.test")
(repo.root / "README.md").write_text("sandbox\n")
repo.commit_all("seed")
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
delegation = Delegation(
AnthropicPassthroughClient.from_env(),
store,
Workspace(repo.root, timeout_s=60),
repo,
model=os.environ[SENTINEL],
caps=caps,
)
return delegation, repo, store
def test_delegate_completes_a_scoped_subtask(tmp_path: Path) -> None:
delegation, repo, _store = build(tmp_path, Caps(turns=12, wall_clock_s=300.0))
result = delegation.run(brief=BRIEF, branch="live/delegation-1")
assert result.usable, f"delegation did not hand back: {result.status} {result.note}"
assert (repo.root / "slugify.py").exists()
# The handback is the entire interface back, so it must actually say
# something in each part rather than be structurally present but empty.
handback = result.handback
assert handback is not None
assert len(handback.changed) > 10
assert len(handback.now_true) > 10
assert len(handback.next) > 5
# The delegate's own work is on its branch, and it really runs.
assert repo.current_branch() == "live/delegation-1"
check = Workspace(repo.root, timeout_s=60).run_command(
"python -m pytest test_slugify.py -q"
)
assert check.success, f"delegate's own tests do not pass: {check.result[:400]}"
def test_a_cap_breach_discards_the_work(tmp_path: Path) -> None:
# One turn cannot finish the brief, so this exercises the abort path
# against a real model rather than a scripted one.
delegation, repo, _ = build(tmp_path, Caps(turns=1, wall_clock_s=120.0))
result = delegation.run(brief=BRIEF, branch="live/delegation-2")
assert not result.usable
assert result.status == "capped"
assert repo.current_branch() == "main"
assert not (repo.root / "slugify.py").exists()
+108
View File
@@ -0,0 +1,108 @@
"""Live Gitea round trip against the sandbox repo: push a branch, open a PR,
comment, and prove the A3 reconciler reads external truth correctly.
Cleans up after itself (closes the PR, deletes the branch) so repeated runs
do not silt up the sandbox.
"""
import os
import uuid
from pathlib import Path
import pytest
from nisse_dev.envfile import load_env_file
from nisse_dev.git import GitError, GitRepo, authenticated_url
from nisse_dev.gitea import GiteaClient, pull_request_reconciler
from nisse_dev.state import Intent
load_env_file(Path(__file__).resolve().parents[2] / ".env")
SENTINEL = "NISSE_GITEA_URL"
ALSO_REQUIRED = ("NISSE_GITEA_TOKEN", "NISSE_GITEA_TEST_REPO")
pytestmark = [
pytest.mark.integration,
pytest.mark.skipif(
not os.environ.get(SENTINEL),
reason=f"no live Gitea configured ({SENTINEL} unset)",
),
]
def intent_for(branch: str) -> Intent:
return Intent(
id=1,
kind="open_pr",
payload={"branch": branch},
status="pending",
created_at="t",
resolved_at=None,
)
def test_push_open_pr_and_reconcile(tmp_path: Path) -> None:
missing = [var for var in ALSO_REQUIRED if not os.environ.get(var)]
assert not missing, (
f"{SENTINEL} is set, so the live Gitea path is expected to work, "
f"but these are missing: {', '.join(missing)}"
)
base_url = os.environ[SENTINEL]
token = os.environ["NISSE_GITEA_TOKEN"]
repo_path = os.environ["NISSE_GITEA_TEST_REPO"]
client = GiteaClient.from_env()
account = client.whoami()
assert "testing" in account, (
f"live tests must run as a testing account, not {account}"
)
branch = f"live-test-{uuid.uuid4().hex[:8]}"
reconcile = pull_request_reconciler(client, repo_path)
# Before the PR exists, external truth says the effect has not happened.
assert reconcile(intent_for(branch)) is False
repo = GitRepo(tmp_path / "work")
repo.init()
repo.configure_identity(account, f"{account}@example.test")
(repo.root / f"{branch}.txt").write_text("live test artifact\n")
assert repo.commit_all("Live test commit") is not None
url = authenticated_url(base_url, repo_path, account, token)
pr = None
try:
repo.push(url, branch, token=token)
pr = client.open_pull_request(
repo_path,
head=branch,
base="main",
title=f"live test {branch}",
body="Opened by the nisse-dev live test suite.",
)
assert pr.head == branch
client.post_comment(repo_path, pr.number, "live test comment")
bodies = [c.body for c in client.list_comments(repo_path, pr.number)]
assert "live test comment" in bodies
# Now the same reconciler must see the effect and refuse to repeat it.
assert reconcile(intent_for(branch)) is True
# A4 is only real if the server enforces it. The agent's own token
# must not be able to merge the PR it just opened, and must not be
# able to push to the protected default branch at all.
merged = client._http.post(
f"/repos/{repo_path}/pulls/{pr.number}/merge", json={"Do": "merge"}
)
assert merged.status_code >= 400, (
"the agent merged its own PR: branch protection is not enforcing "
"the human gate"
)
with pytest.raises(GitError):
repo.push(url, "main", token=token)
finally:
if pr is not None:
client.close_pull_request(repo_path, pr.number)
repo.delete_remote_branch(url, branch, token=token)
client.close()
+22
View File
@@ -33,6 +33,27 @@ pytestmark = [
] ]
def _assert_testing_identity(transport: MatrixTransport, config: MatrixConfig) -> None:
"""Two invariants that only a live homeserver can check.
The configured user id must be the token's real owner, because the
transport filters its own messages by comparing sender to that value —
a mismatch would make the bot read its own posts as operator input.
And tests must run as a testing identity, so a stray live run can never
act as the production agent.
"""
whoami = transport._http.get("/account/whoami")
whoami.raise_for_status()
actual = whoami.json()["user_id"]
assert actual == config.user_id, (
f"NISSE_MATRIX_USER_ID is {config.user_id} but the token belongs to "
f"{actual}; the self-message filter would not match"
)
assert "testing" in actual, (
f"live tests must run as a testing identity, not {actual}"
)
def test_send_lands_in_test_room() -> None: def test_send_lands_in_test_room() -> None:
missing = [var for var in ALSO_REQUIRED if not os.environ.get(var)] missing = [var for var in ALSO_REQUIRED if not os.environ.get(var)]
assert not missing, ( assert not missing, (
@@ -43,6 +64,7 @@ def test_send_lands_in_test_room() -> None:
marker = f"transport smoke test {uuid.uuid4().hex}" marker = f"transport smoke test {uuid.uuid4().hex}"
transport = MatrixTransport(config) transport = MatrixTransport(config)
try: try:
_assert_testing_identity(transport, config)
transport.send_message(marker) transport.send_message(marker)
response = transport._http.get( response = transport._http.get(
f"/rooms/{config.room_id}/messages", f"/rooms/{config.room_id}/messages",
+107
View File
@@ -0,0 +1,107 @@
"""End-to-end milestone against real models: plan with the planner role,
approve through the gate, then let the executor role actually create and test
a file on disk.
Gated on NISSE_LIVE_PLANNER_MODEL like the planning test — it needs a model
that holds the tool-call contract on both ends.
"""
import io
import os
from pathlib import Path
import pytest
from nisse_dev.control import (
AWAIT_ARCH_SIGNOFF,
AWAIT_INPUT,
AWAIT_MILESTONE_SIGNOFF,
EXECUTING,
Orchestrator,
)
from nisse_dev.envfile import load_env_file
from nisse_dev.executor import LLMExecutor
from nisse_dev.llm import LiteLLMClient, RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.planner import LLMPlanningAgent
from nisse_dev.state import StateStore
from nisse_dev.tools import Workspace
from nisse_dev.transport import InMemoryTransport
load_env_file(Path(__file__).resolve().parents[2] / ".env")
SENTINEL = "NISSE_LIVE_PLANNER_MODEL"
pytestmark = [
pytest.mark.integration,
pytest.mark.skipif(
not os.environ.get(SENTINEL),
reason=f"no live planner route configured ({SENTINEL} unset)",
),
]
def _drive_to_execution(
store: StateStore, transport: InMemoryTransport, max_steps: int = 10
) -> None:
"""Play the operator. A scripted approve/approve/start sequence is not
robust against a live model: ask_human is a legitimate move at any gate,
and a question shifts the state machine under a fixed script. So respond
to whatever the agent actually did."""
for _ in range(max_steps):
state = store.get_state()
assert state is not None
if state.state == EXECUTING:
return
gate = state.data.get("gate") or {}
if state.state == AWAIT_INPUT:
transport.receive("don", "Use your best judgement and proceed.")
elif state.state == AWAIT_MILESTONE_SIGNOFF and gate.get("approved"):
transport.receive("don", "start milestone 1")
elif state.state in (AWAIT_ARCH_SIGNOFF, AWAIT_MILESTONE_SIGNOFF):
transport.receive("don", "approve")
else:
raise AssertionError(f"unexpected state {state.state}")
raise AssertionError(
f"never reached execution; last message: {transport.outbound[-1]}"
)
def test_plan_approve_execute_produces_a_real_file(tmp_path: Path) -> None:
assert os.environ.get("NISSE_LLM_TOKEN"), (
f"{SENTINEL} is set, so the live path is expected to work, "
"but NISSE_LLM_TOKEN is missing"
)
# Both roles use the capable model here: the point is the whole gated
# loop over a real proxy, not local-model competence (measured separately).
model = os.environ[SENTINEL]
roles = RoleMap(executor=model, planner=model, delegate=model, consultant=model)
client = LiteLLMClient.from_env()
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
workspace = Workspace(tmp_path / "work", timeout_s=60)
transport = InMemoryTransport()
orchestrator = Orchestrator(
store,
transport,
LLMPlanningAgent(client, store, roles),
operators={"don"},
roles=roles,
executor=LLMExecutor(client, store, roles, workspace, max_turns=10),
)
orchestrator.start()
transport.receive(
"don",
"use greet. Milestone 1: create greet.py in the workspace root defining "
"greet(name) returning 'Hello, <name>!', and verify it by running python.",
)
_drive_to_execution(store, transport)
state = store.get_state()
assert state is not None
assert state.state in (EXECUTING, AWAIT_INPUT)
created = list(workspace.root.rglob("*.py"))
assert created, f"executor created no Python file; said: {transport.outbound[-1]}"
combined = "\n".join(p.read_text() for p in created)
assert "def greet" in combined
+108
View File
@@ -0,0 +1,108 @@
"""Live Anthropic passthrough: the vendor-native route delegations will use.
Proves the renderer's body is accepted as-is, that a tool call round-trips
through the canonical log, and — per A5 — that prompt caching survives the
passthrough rather than being flattened by the proxy.
"""
import io
import os
from pathlib import Path
import httpx
import pytest
from nisse_dev.canonical import AssistantText, ToolCall, ToolResult, UserMessage
from nisse_dev.envfile import load_env_file
from nisse_dev.llm import AnthropicPassthroughClient
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore
from nisse_dev.tools import EXECUTOR_TOOLS
load_env_file(Path(__file__).resolve().parents[2] / ".env")
SENTINEL = "NISSE_LIVE_PLANNER_MODEL"
pytestmark = [
pytest.mark.integration,
pytest.mark.skipif(
not os.environ.get(SENTINEL),
reason=f"no live frontier route configured ({SENTINEL} unset)",
),
]
def store_with(tmp_path: Path, *events) -> StateStore:
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
for event in events:
store.append_event(event)
return store
def test_passthrough_completes_a_plain_turn(tmp_path: Path) -> None:
store = store_with(
tmp_path, UserMessage(text="Reply with exactly: pong", source="live")
)
result = AnthropicPassthroughClient.from_env().complete(
store.read_events(), None, os.environ[SENTINEL], system="Be terse."
)
texts = [e.text for e in result if isinstance(e, AssistantText)]
assert texts and "pong" in texts[0].lower()
def test_passthrough_round_trips_a_tool_call(tmp_path: Path) -> None:
store = store_with(
tmp_path,
UserMessage(
text="List the files in the current directory using your tools.",
source="live",
),
)
client = AnthropicPassthroughClient.from_env()
produced = client.complete(
store.read_events(),
EXECUTOR_TOOLS,
os.environ[SENTINEL],
system="Use a tool. Do not answer in prose.",
)
calls = [e for e in produced if isinstance(e, ToolCall)]
assert calls, f"expected a tool call, got {produced!r}"
# Feed the result back: the follow-up render must be accepted, which is
# what proves tool_result placement is right for this vendor.
for event in produced:
store.append_event(event)
store.append_event(
ToolResult(call_id=calls[0].call_id, result="app.py\ntests/", success=True)
)
follow_up = client.complete(
store.read_events(), EXECUTOR_TOOLS, os.environ[SENTINEL]
)
assert follow_up, "the vendor rejected or ignored the tool_result turn"
def test_prompt_caching_survives_the_passthrough() -> None:
# A5 asks for this to be verified rather than assumed: a proxy that
# flattened the request would silently drop cache control and turn a
# cost optimisation into a per-call expense.
base = os.environ["NISSE_LLM_BASE_URL"].rstrip("/")
token = os.environ["NISSE_LLM_TOKEN"]
response = httpx.post(
f"{base}/v1/messages",
headers={
"Authorization": f"Bearer {token}",
"x-api-key": token,
"anthropic-version": "2023-06-01",
},
json={
"model": os.environ[SENTINEL],
"max_tokens": 16,
"messages": [{"role": "user", "content": "Say pong."}],
},
timeout=120,
)
response.raise_for_status()
usage = response.json().get("usage", {})
assert "cache_read_input_tokens" in usage, (
f"cache accounting absent from passthrough usage: {usage}"
)
+64
View File
@@ -0,0 +1,64 @@
"""Live planning turn: a real model must answer with a submit_plan or
ask_human tool call, not prose.
Deliberately opt-in via NISSE_LIVE_PLANNER_MODEL rather than the usual
NISSE_LLM_BASE_URL sentinel. Planning escalates to a frontier model
unconditionally (A12), and the small local model does not hold the tool-call
contract reliably — measured, see the PR — so this test needs a frontier
route that does not exist yet. Set the variable to the model to plan with
once one is configured; until then this skips on purpose, not by accident.
"""
import io
import os
from pathlib import Path
import pytest
from nisse_dev.canonical import UserMessage
from nisse_dev.control import AskHuman, SubmitPlan
from nisse_dev.envfile import load_env_file
from nisse_dev.llm import LiteLLMClient, RoleMap
from nisse_dev.oplog import OpLog
from nisse_dev.planner import LLMPlanningAgent
from nisse_dev.state import StateStore
load_env_file(Path(__file__).resolve().parents[2] / ".env")
SENTINEL = "NISSE_LIVE_PLANNER_MODEL"
pytestmark = [
pytest.mark.integration,
pytest.mark.skipif(
not os.environ.get(SENTINEL),
reason=f"no frontier planner route configured ({SENTINEL} unset)",
),
]
def test_live_planning_turn_returns_a_tool_action(tmp_path: Path) -> None:
assert os.environ.get("NISSE_LLM_TOKEN"), (
f"{SENTINEL} is set, so the live planning path is expected to work, "
"but NISSE_LLM_TOKEN is missing"
)
model = os.environ[SENTINEL]
roles = RoleMap(executor=model, planner=model, delegate=model, consultant=model)
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO()))
store.append_event(
UserMessage(
text=(
"Project 'hello-cli': a Python script that prints a greeting. "
"Draft the architecture plan."
),
source="live-test",
)
)
agent = LLMPlanningAgent(LiteLLMClient.from_env(), store, roles)
action = agent.plan("arch", "hello-cli")
assert isinstance(action, SubmitPlan | AskHuman)
text = action.content if isinstance(action, SubmitPlan) else action.question
assert text.strip(), "tool call carried empty content"
# The turn must be durable regardless of what the model said (A2/A10).
assert len(store.read_events()) > 1