From be04c2854278ba29e35f5d1e038e8080f89975be Mon Sep 17 00:00:00 2001 From: Don Garrett Date: Fri, 4 Sep 2026 21:56:23 +0000 Subject: [PATCH] Add Nisse build specification Seeds nisse-core as the home for system-level design documentation. The spec covers the shared library (Part A), both agents (Parts B and C), and the supporting services, build order, and open decisions. nisse-core holds this because the document describes the shared design rather than either agent's implementation; the repo gains code later, at the extraction step in the build order. Co-Authored-By: Claude Opus 5 --- docs/Architecture-Build-Spec.md | 802 ++++++++++++++++++++++++++++++++ 1 file changed, 802 insertions(+) create mode 100644 docs/Architecture-Build-Spec.md diff --git a/docs/Architecture-Build-Spec.md b/docs/Architecture-Build-Spec.md new file mode 100644 index 0000000..70feef8 --- /dev/null +++ b/docs/Architecture-Build-Spec.md @@ -0,0 +1,802 @@ +# Nisse — Build Specification + +A build-oriented spec for two self-hosted AI agents sharing a common library. This +document states **requirements per component** so major pieces can be broken out and +built independently. It is deliberately lighter on the supporting services (Matrix, +Gitea, Vikunja, Outline, LiteLLM) and heaviest on the **agent library** and the +**two agents**, which is where the real design work lives. + +For deeper rationale behind any decision here, see the companion `Architecture.md`. + +**Document history.** The Tiered Inference amendment has been folded into this +document and no longer exists separately: its A-series components appear as A10–A16, +its network and credential revisions in A5 and B3, and its revised build order as +Part G. The inference-hardware question it raised has since been resolved in +principle (Part H: a 4090-class host). + +--- + +## 0. Naming + +The project is **Nisse**, after the Nordic farm spirit that keeps the place running +overnight and expects to be treated well in return. Three artifacts: + +| Artifact | Name | Role | +|---|---|---| +| Shared library | `nisse-core` | The spine both agents sit on (Part A) | +| Dev Agent | `nisse-dev` | Plans and implements software projects (Part B) | +| Sysadmin Agent | `nisse-ops` | Operates infrastructure (Part C) | + +Throughout this document, **Dev Agent** and **Sysadmin Agent** name the *roles* in +conceptual discussion; `nisse-dev` and `nisse-ops` name the *artifacts* — repos, +container images, VM hostnames, Matrix identities, and service accounts. Use the +same string across all of those: the VM, the container, the Caddy host, the Matrix +account, and the Bitwarden machine account for a given agent all share one name, so +that grepping the infrastructure for `nisse-ops` finds everything. + +Published under a personal GitHub account rather than an org; `github.com/nisse` is +squatted and GitHub does not reclaim inactive usernames. The repo names carry the +identity. `nisse-core` is free on PyPI if packages are ever published. + +--- + +## 0.1 What We're Building + +Two agents: + +1. **Dev Agent** — `nisse-dev` (build first) — plans and implements software + projects, produces Gitea PRs for human review, reacts to review comments and + commits. +2. **Sysadmin Agent** — `nisse-ops` (follow-on) — operates and maintains + infrastructure, driven by tickets, gated on high-consequence actions. + +Both are thin applications over a shared **`nisse-core`** library. The library is +*extracted from the Dev Agent after it works* — not designed up front for two +consumers that don't exist yet. Build the Dev Agent as if the library were internal, +then lift the seams that proved real. + +Whether the three artifacts live in one repository or three is **an open decision** +(Part H). It changes what the extraction step in the build order concretely means. + +### Guiding principles (apply to everything below) + +- **Human-gated autonomy.** Agents work independently once told to start, but never + cross key boundaries (start implementation, apply infrastructure change) without + explicit sign-off. +- **No per-command approval.** Within an approved unit of work, execution is + unattended. Control lives at the *boundaries*, not per action. +- **Isolation matched to blast radius.** Dev Agent is disposable and sealed; Sysadmin + Agent is trusted with live-infrastructure reach. They never share a running + instance. +- **Stateless image, stateful volumes.** Containers are replaceable; durable state + lives on named volumes and must survive both redeploys and unexpected reboots. +- **Structured channels for machines, prose channels for humans.** Never route + machine event intake through human-facing formatters (chat bridges). +- **Own the stack.** Everything self-hostable stays inside the network where practical. + +--- + +## PART A — THE AGENT LIBRARY (`nisse-core`) + +The shared spine. Every component here is consumed by both agents. This is the part +to get right. + +### A1. Transport + +**Purpose:** all human⇄agent conversation — commands in, questions/approvals/updates +out. + +Requirements: +- A `Transport` interface the orchestrator depends on; **no platform SDK imported + directly** anywhere else. Methods (at least): `send_message`, + `send_approval_request`, `ask_question`, `on_message` (inbound callback/stream). +- **`MatrixTransport`** is the first and primary implementation. Self-hosted + homeserver (Synapse or Conduit) keeps the control plane in-network. Must support + threaded conversations (for plan discussion) and per-agent identities. +- Interface must make adding `TelegramTransport` / `SlackTransport` later a matter of + one new class + config, no orchestrator changes. +- Both agents run as **distinct identities** (`@nisse-dev`, `@nisse-ops`) in separate + rooms on the same homeserver. + +Explicitly *not* required: rich message formatting, custom clients. Existing Matrix +clients (Element) are the UI. + +**Matrix is strictly a UI channel.** Although the homeserver is self-hosted and +therefore not "remote," room history must not become load-bearing context. The +canonical log (A10) is the sole source of truth. + +### A2. State Store and Durability *(critical path — build first)* + +**Purpose:** durable, crash-safe persistence of everything that must outlive a +process restart. This is what makes the whole system safe to redeploy and reboot; it +blocks everything else. + +Must persist: active project/task, per-project state-machine position, approvals +granted, pending questions, session continuity, (dev) git clone locations. + +Disposable (never on the state volume): agent code, build artifacts/test containers, +shipped-out logs. + +Requirements: +- **Crash-atomic writes.** SQLite in **WAL mode** (default choice) or atomic + file writes (`write-temp → fsync → rename`). A power cut mid-write must never + produce torn state. +- **Checkpoint after every state transition**, with `fsync`. No "flush later" — + in-memory state that hasn't hit disk is assumed lost. +- **Durable intent log.** Before a unit of work starts, record intent, so a restart + knows what was in flight. +- **Idempotent transitions reconciled against external truth.** On any operation with + an external side effect (open PR, push branch), restart must detect the effect + already happened (query Gitea/git) rather than repeat it. +- **`schema_version`** field from the first commit, with a migration hook, so a newer + image can read older state. + +### A3. Boot-Time Reconciliation + +**Purpose:** turn an ambiguous crash into a safe, known state. + +Every startup (clean or post-crash), before accepting operator commands: +1. Open state store (WAL recovery is automatic). +2. Run any pending schema migration. +3. Read intent log; look for a transition that began but never committed. +4. Reconcile against external truth (Gitea/git for the dev agent; relevant services + for the sysadmin agent). +5. **Interrupted mid-delegation (A13) — apply the deterministic resume rule.** + **Auto-resume** the delegation — re-render its transcript from the canonical log + and continue the loop, posting a notice (not a question) to transport — when the + cut is clean: the last durable event is a completed `tool_result` or assistant + turn, the delegation branch sits at the expected commit, no cap is breached, and + external truth matches the log. **Gate to `AWAIT_INPUT`** when the last event is a + `tool_call` with no recorded result whose effect cannot be verified by inspection + (tree diff for edits, Gitea query for pushes), or on any branch/log mismatch. The + judgment is an observable check, never model self-assessment — the same reasoning + as A12. +6. Any other interruption → move to **`AWAIT_INPUT`** and post to transport + (e.g. "restarted mid-milestone-2, branch at commit abc123, continue or re-plan?"). + Never blind-resume. +7. Otherwise resume the last committed state. + +### A4. Control Loop, State Machine, and Gating Primitives + +**Purpose:** the generic engine that drives an agent through gated work. Each agent +supplies its own concrete states; the primitives are shared. + +Requirements: +- A **state machine** framework: named states, transitions, and human gates. Each + gate blocks until an explicit operator action arrives via transport. +- **Gating is structural, not advisory.** The mechanism (used by both agents): + - Expose a **`submit_plan(type, content)`** tool to the model. Calling it posts the + plan for approval and **ends the agent's turn.** The agent cannot self-advance. + - Expose an **`ask_human(question)`** tool — non-terminal: pause, post question, + block until answered, resume. + - **Do not** expose any "proceed" / "start" tool. Advancement past a gate happens + only when the orchestrator receives operator approval and hands over the next + task. The agent has no path to approve itself. +- **Approval sources.** A gate waits on an approval event from any registered + **`ApprovalSource`** — the Matrix transport (A1), the Gitea comment poller (B2), or + the Vikunja comment stream (C4). First valid approval wins and is consumed; the + gate records which channel satisfied it, and the orchestrator echoes the outcome to + the other channels so the records converge. Each source authenticates the approver + against a per-channel operator allowlist (Matrix MXID, Gitea username, Vikunja + username). +- **Approvals bind to a plan revision, never to "the current gate."** Every submitted + plan revision carries a monotonically-increasing tag (`rev-N`). An approval naming + a stale revision is rejected with a notice, never applied to a newer one — this is + what makes multi-channel approval race-free. +- **One consistent approval UI across all channels.** Approval is always the typed + word `approve`, attached to a specific revision: in Matrix, a reply to the plan + message (the reply relation carries the binding; a bare `approve` in the room binds + to the latest revision only when unambiguous, otherwise the agent asks); in Gitea + and Vikunja, a comment `approve rev-N`, with the tag printed in the plan comment + for copy-paste. Any other reply/comment is `reject `. Reactions/emoji are + never an approval path. Approvals are consumed at first sight; later edits to an + approval comment are ignored and logged. +- **Autonomy inside an approved unit of work:** all tool/command use auto-approved + (Claude Agent SDK `bypassPermissions` or an allow-all `canUseTool`). Escalation to + the human happens *only* via `ask_human`. +- The loop must survive restart mid-work (relies on A2/A3). + +### A5. LLM Access *(tiered)* + +**Purpose:** reach both a locally-hosted model and frontier models through one +interface, with the frontier vendor swappable by config. + +A locally-hosted model drives the control loop and performs mechanical work; frontier +models are engaged for work that exceeds local capability. Two requirements drive the +tiering: + +1. **No remotely-held state.** Conversation and task state lives on the agent VM. No + provider holds anything the system depends on to resume. +2. **Vendor fluidity.** The frontier tier should be swappable — Anthropic, OpenAI, + Google, or a vendor that does not exist yet — as a configuration change, not a + refactor. + +These two are only in tension if portability is required *within* a session. Making +frontier engagements **ephemeral and self-contained** (A13) allows vendor-specific +session data to be used freely inside a delegation while keeping the durable record +vendor-neutral. + +Requirements: +- **One client interface**, `complete(canonical_log, tools, model) -> canonical_events`, + present at the **first real LLM call** — not introduced as a later cutover. The same + reasoning that produced the `Transport` abstraction. +- **LiteLLM in front of everything, from the start.** The agent VM gets exactly one + LLM destination; LiteLLM fans out to the local inference host and to frontier + vendors behind it. This also resolves the RFC1918 conflict in B3 — a LAN-resident + inference host would otherwise need a second hole punched in the egress allowlist. +- **Passthrough for the vendor driving a given session** (`/v1/messages` for + Anthropic), so vendor-specific features are not lost on the path where they matter — + delegations (A13) always take this path. **Normalized routing for everything else**; + the canonical log (A10) reconciles the difference. +- **Fallback chains configured at LiteLLM.** Rate limits and outages differ per + vendor; a frontier failure should not kill a task. +- **No vendor credentials on the agent VM.** LiteLLM holds every key, sourced from + Bitwarden machine accounts (A7); the agent holds one LiteLLM token. Adding a vendor + does not touch the agent VM, and a compromised agent leaks no provider keys. +- Config from env/`.env`, **nothing hardcoded**: base URL, auth token, and the role + mapping (A11). Changing any of it is an env change + restart, nothing more. +- **Budget caps and spend alerts live on the LiteLLM virtual key**, not in agent code. + +### A6. Browser Tool + +**Purpose:** drive web UIs. Shared implementation; each agent points it at a +different network with different trust. + +Requirements: +- **Playwright** (headless Chromium), exposed to the model as tool actions: + navigate, click, type, read DOM, screenshot. Screenshot + accessibility-tree loop + so the model can operate UIs it wasn't trained on. +- Runs **inside the agent's VM**. Network reachability is set by the agent's + deployment, not the tool. +- Must support authenticated sessions (for the sysadmin agent's admin UIs) while + being usable without auth (dev agent's test apps). + +### A7. Secrets Client + +**Purpose:** fetch credentials at runtime instead of baking them into images. + +Requirements: +- Client for **Bitwarden/Vaultwarden** via **machine account / Secrets Manager** (or + `bw` CLI + API key). Not the personal vault. +- Reads only from a **dedicated, minimally-scoped machine account** per agent. +- A single **bootstrap token** (the machine-account credential) is the one secret + that lives on the box: tight file perms, never in image or repo. +- (Deferred, not first cut: dynamic/short-lived secrets via Vault/OpenBao. Leave the + interface able to grow toward this; don't build it yet.) + +### A8. SSO Identity + +**Purpose:** each agent authenticates as itself. + +Requirements: +- Each agent is a dedicated **LLDAP** principal (`svc-nisse-dev`, `svc-nisse-ops`) in + a tightly-scoped **Authelia** group. Never the operator's account. +- Support an **unattended-auth path**: because interactive MFA is impossible for a + headless agent, allow a narrow **IP-scoped, group-scoped, target-scoped** + single-factor exception. Keep it specific on all three axes. + +### A9. Logging + +**Purpose:** observability without bloating state. + +Requirements: structured logs to journald / external log stack, **not** kept +in-container. Every gate crossing, tool call, and escalation is logged. + +### A10. Canonical Transcript Schema + +The durable record is a **canonical event log** in SQLite, not a provider-shaped +`messages` array. Provider request bodies are rendered at call time from this log. + +| Event | Payload | +|---|---| +| `user_message` | text, source (Matrix event ID) | +| `assistant_text` | text, model that produced it | +| `tool_call` | canonical call ID, tool name, arguments | +| `tool_result` | canonical call ID, result, success/failure | +| `plan_submitted` | plan body, authoring model | +| `gate_opened` | gate type, reason | +| `delegation_started` | brief, target role, resolved model, caps | +| `delegation_completed` | handback note, artifacts, tokens, duration, exit status | +| `compaction` | summary, producing model, range of events replaced | + +Rules: +- **Canonical tool-call IDs are minted locally.** Provider IDs are mapped at the + boundary and never used as a join key. Providers differ in both ID format and in + where tool calls sit structurally (content blocks vs. a separate array on the + assistant message). +- **No provider-side handles for content.** If a Files API is used, store the bytes or + a content-addressed local path; upload or inline at render time. A provider file ID + must never be the only reference to something. +- **Vendor-opaque reasoning artifacts are stored, tagged with the producing model, and + never replayed outside the session that produced them.** This covers Anthropic + signed thinking blocks, OpenAI encrypted reasoning items, and Gemini thought + signatures. Inside a delegation, replay per that vendor's rules. On delegation exit, + the artifacts become archival only. A "session" here is the **delegation**, not the + process lifetime: a delegation resumed after a crash (A3) is the same session and + may replay its own artifacts. +- **Token budgeting is a render-time concern.** Context window and tokenizer are + properties of the model being called. Compute budget when building the request, not + when writing the log. +- **Compaction summaries are first-class log events** with the producing model + recorded, so a tier change does not silently inherit opaquely-compacted history. +- **Prompt cache breakpoints are placed by the renderer and never stored.** A tier + change means a cold cache — a cost event, not a correctness event. + +### A11. Role Configuration + +Model identity is expressed as **roles**, resolved to concrete model strings in config: + +```yaml +roles: + executor: local/ # drives the loop, mechanical work + planner: # authors plans + delegate: # scoped sub-task execution + consultant: # advisory, no repo access +``` + +"Best model this week" is a config edit. **The resolved mapping is pinned at task +start** and recorded in the log, so a mid-task config change cannot split a task +across two planners. + +### A12. Escalation Triggers + +The local model does **not** decide when it needs help. Small models are poorly +calibrated about their own competence and will escalate either constantly or never. +Triggers are observable from outside the model: + +1. **Phase policy.** Planning and architecture escalate unconditionally. Mechanical + edits stay local. +2. **Failure count.** Local attempts; tests fail twice; escalate with the failures + attached to the brief. + +A model-invoked `ask_expert` (A14) exists in addition to these, never as a +replacement, and carries a per-task budget so a confused executor cannot spend twenty +consults circling. + +### A13. Delegation + +Escalation is **delegation**: a frontier model receives a scoped sub-task, works +autonomously with full tool access, and hands back. It is not a consult, and the local +model does not interpret or re-derive the frontier model's reasoning. + +**Harness — one, not one per vendor.** Do not adopt each vendor's agent SDK to obtain +their best agentic behavior; that recreates the lock-in this design avoids, relocated. +One loop, one cap regime, one handback contract. + +**Normalization lives in the A10 renderer, not on the wire.** Vendors disagree +structurally about tool calling (content blocks vs. `tool_calls` arrays, tool-result +placement, streaming deltas) and about reasoning artifacts (signed thinking blocks, +encrypted reasoning items, thought signatures) — and a normalizing middlebox is lossy +exactly where a long-running delegation lives. So: + +- **Delegations run over LiteLLM's passthrough route** (`/v1/messages` for Anthropic, + per A5), with the A10 renderer producing the vendor-native request body. The + renderer is a per-vendor adapter by definition — it already has to know each wire + shape to render the canonical log. +- **LiteLLM's normalized route is reserved for calls where fidelity is cheap:** + `ask_expert` (A14), fallback chains, and trialing a new vendor before its renderer + adapter exists. +- **Adding a vendor** therefore means writing one renderer adapter — or provisionally + accepting LiteLLM's normalization for that vendor and validating it against the A16 + eval set before trusting it with delegations. A day-one build needs exactly one + adapter: the vendor actually in use. + +**Scoping and authority.** Delegation sits **inside** the plan gate, not around it. A +frontier sub-agent that acts freely would bypass `submit_plan` (A4), which is the +structural control the design rests on. + +- **Planning is itself a delegation** that produces a plan and stops. Human approves. +- **Execution delegations are scoped to a sub-task the approved plan already + contains.** + +The frontier model is smart twice; neither time is it authorized to expand the work. + +Because plans may be executed by a materially weaker model, the planner system prompt +must require **explicit, low-inference plans** — the executor cannot reconstruct +intent. + +**Isolation and caps.** +- **Branch at delegation start.** Abort discards the branch. This gives delegations the + same atomicity property that boot-time reconciliation (A3) gives interrupted state + transitions. +- **Hard caps on turns, tokens, and wall clock.** +- **Checkpoint delegation events as they arrive.** Every `tool_call` / `tool_result` / + assistant turn inside a delegation is durably written to the canonical log when it + happens (per A2), never batched at handback. This is what makes a delegation + resumable after a crash — the deterministic resume rule lives in A3. +- **Defined failure return.** On cap breach or explicit failure, the local loop gates + to `AWAIT_INPUT` rather than resuming on top of a half-edited tree. + +**Handback.** The final delegation action is a **structured handback note written for a +weaker reader**, not a human-facing summary: + +- what changed +- what is now true that was not before +- what to do next + +That note plus the resulting file state is the entire interface back to the local loop. +The delegation's internal transcript is stored alongside the composite event, +vendor-tagged, and never replayed into anything. + +#### Rejected alternative: ask-and-apply + +An alternative was considered in which the local model asks a frontier model for advice +and applies the answer itself. **Rejected.** + +Its principal advantage was blast radius — the frontier model never touches the repo, +and everything flows through a gated local loop. That was explicitly not a concern +here. + +Without it, the comparison is lopsided: + +- Delegation lets the frontier model read the code, run the tests, and see the + failures. Ask-and-apply forces it to reason blind from whatever the local model + thought to include in the brief, and small models write poor briefs. +- The cost advantage is illusory: one delegation that succeeds beats three consult + round-trips that do not. +- Any variant where the local model *interprets* rather than applies verbatim + reintroduces the capability gap downstream of the expensive call. + +The diff-application machinery, verbatim-fidelity rules, and local-model-as-verifier +design are all dropped. + +### A14. `ask_expert` (retained, minimal) + +One narrow case survives delegation: **questions requiring no repo access.** "Which of +these two approaches is better," "why does this library behave this way," "is this API +deprecated." + +- Single stateless call, text answer. +- No diff application, no fidelity concerns, no session state. +- Reuses the delegation brief format. +- Structured output contracted **at the prompt level, not the API level** — vendors + differ on JSON mode and schema enforcement. Parse, validate against a local schema, + retry once. + +Expected implementation size: small. Do not let it grow into a second escalation +subsystem. + +### A15. Local Model Accommodations + +- **Constrained decoding for tool-call JSON.** +- **Reduced tool schema for the executor role:** fewer tools, flatter arguments, no + deeply nested optionals. Maintain **two tool manifests keyed on role**, rendered at + request time like everything else. + +### A16. Delegation Evaluation Capture + +Capture a **consult/delegation eval set** from day one. Each `delegation_started` / +`delegation_completed` pair (A10) is already a clean input-output record; the +additional cost is marking outcomes as good or bad. + +Without this, "best model this week" is decided by benchmark chatter, and swapping the +frontier tier becomes something to avoid because the effect is unmeasurable. With it, a +new model is a config change and a re-run. + +--- + +## PART B — THE DEV AGENT / `nisse-dev` (build first) + +A thin app over `nisse-core`. Plans and implements software projects; produces PRs. + +### B1. Workflow (state machine) + +``` +IDLE + → [use ] clone repo, begin architecture planning +PLANNING_ARCH + → agent drafts architecture + implementation plan → submit_plan +AWAIT_ARCH_SIGNOFF ← human gate (approve / reject+feedback) + → +MILESTONE_PLANNING + → agent drafts plan for a specific milestone → submit_plan +AWAIT_MILESTONE_SIGNOFF ← human gate + → [start milestone ] +EXECUTING autonomous; no per-command approval + → may enter AWAIT_INPUT (ask_human) any time it's unsure +PR_OPEN + → open PR; react to review comments and commits (review loop) + → on acceptance → next milestone, or project complete +``` + +### B2. Requirements + +- **One active project at a time.** Operator selects it. +- **Operator commands** (over transport): `use `, `approve`, + `reject `, `start milestone `, `status`, plus free text during + planning (forwarded to the agent as answers). +- **Plan gating** per A4: architecture plan and each milestone plan require sign-off + before the next phase. Enforced structurally. Plans and approvals are externalized + to Gitea issue comments (Part H, decided); Matrix and Gitea are both approval + channels, with revision binding and the uniform `approve` UI per A4. +- **Autonomous execution** within a started milestone: builds, runs, tests without + per-command approval; asks questions any time via `ask_human`. +- **Gitea integration:** + - *Outbound:* `git` + Gitea API to push branches and open PRs, via a dedicated + Gitea **service account** with a scoped token. Record any runnable test service's + URL/port in the PR description. + - *Inbound:* react to PR comments, commits, pushes. **Poll** the Gitea API + (15–30s) rather than take webhooks, to keep the VM outbound-only. Intake is + structured JSON — never via a chat bridge. + - *On Gitea unavailability:* the poller backs off (exponential, capped at a few + minutes) and retries indefinitely, logging the outage. Polling is pure intake — + an outage delays reactions but never fails a task or forces a state transition. + Outbound Gitea operations that fail mid-flight are already covered by the + intent log and boot-time reconciliation (A2/A3). +- **Browser (A6)** scoped to the **internal test network only** — used to exercise + web apps it built and reproduce bugs. **No SSO needed** for this. + +### B3. Deployment & Isolation + +- **Dedicated VM** (Proxmox), not a bare container — the agent builds and runs Docker + containers, so it needs its own real Docker daemon. (Docker-in-a-container via + socket-mount ≈ host root; via dind = nested-storage pain. The VM boundary is the + correct trust boundary for unattended arbitrary command execution.) +- **Sealed to start:** egress allow-list (LiteLLM, Gitea, package registries), **no + ingress**. Egress denial must include **RFC1918 except the specific LiteLLM/Gitea + IPs** — sealed from the internet must also mean no reach to NAS/Proxmox/UDM. + - **The LAN-resident inference host is not a second exception.** It sits behind + LiteLLM (A5), so the allowlist keeps exactly one LLM destination regardless of how + many models or vendors are in play. +- **No Tailscale** on this VM (it routes around firewall policy). Manage via Proxmox + console / jump host. +- **Own VLAN from day one** (even a VLAN of one) and test services on **predictable + ports** (e.g. 18000–18099), so opening access later is a firewall edit, not a + rebuild. (Sealed for now; may relax later.) +- Test containers on a dedicated **internal Docker network with no route out**. +- Build-daemon storage on a **separate mount** from the state volume, so + `docker system prune` between milestones can't touch agent state. + +--- + +## PART C — THE SYSADMIN AGENT / `nisse-ops` (follow-on) + +A thin app over the extracted `nisse-core`. Operates infrastructure; driven by +tickets; gated on consequences. Built after the Dev Agent works. + +### C1. How it differs (not reducible to config) + +| Axis | Dev Agent | Sysadmin Agent | +|---|---|---| +| Rollback | revert snapshot / discard branch | none for `zfs destroy` or a firewall lockout | +| Trust model | allow-all in sandbox | tiered **allow / deny / confirm** policy | +| Work shape | long milestones → a PR | short reactive tasks + scheduled checks → live changes | +| Driver | Gitea projects | **Vikunja tickets** | +| Placement | sealed, own VLAN, no Tailscale | Tailscale, credentials, broad reach | + +Running both roles in one instance would collapse isolation — build them as separate +deployments over the same library. + +### C2. Permission Policy Engine *(the defining addition)* + +Tiered policy, evaluated on every action: +- **Read-only telemetry** (metrics, logs, status) — free. +- **Reversible service actions** (restart a container) — allowed. +- **High-consequence actions** (storage, networking, auth) — **confirm-first** over + transport before executing. + +This is the sysadmin analog of the dev agent's plan gate: the human boundary sits at +irreversible / high-blast-radius actions. + +Special cases fold into the tiers: +- **Browser actions against admin UIs default to confirm-first.** Changes to + **network gear (UniFi) are the most conservatively gated action in the system** — a + bad remote UI click can sever the path used to fix it. +- **Action-triggering tickets are operator-scoped** (C4). Auto-generated tickets + (e.g. from a monitoring alert) route through confirm-first, never drive unattended. + +### C3. Interfaces + +- **Browser (A6):** into admin UIs. **Prefer real APIs where they exist** (Proxmox, + TrueNAS good; UniFi partial); use the browser only for genuinely UI-only gaps — + it's the least stable interface. +- **SSO (A8):** logs into admin UIs as `svc-nisse-ops`. This account is a high-value + credential; its blast radius equals its Authelia scope, so scope it minimally. +- **Secrets (A7):** UI/SSO logins and service API tokens from Bitwarden. +- **Event intake:** monitoring feeds the orchestrator as **structured JSON** + (directly or via Alertmanager), never through a Matrix bridge. The agent posts + curated, state-aware updates to `#ops` ("Gitea 502ing, restarted, back up"). + +### C4. Ticket-Driven Control (Vikunja) + +The sysadmin analog of "dev agent driven by Gitea." +- **Vikunja ticket = task of record** (durable, has status/queue/history). **Matrix = + live conversation about it** (questions, confirm-first prompts, progress). +- **Flow:** ticket filed → agent picks up → discuss/confirm over Matrix → update + status as it progresses → close with result. +- **Mechanics:** Vikunja REST API + webhooks (same two-direction, JSON-only pattern + as Gitea); dedicated service account, scoped token from Bitwarden. Verify the + installed version's webhooks fire on comment creation; if not, poll comments as + B2 polls Gitea. +- **Approvals:** Vikunja is an `ApprovalSource` (A4) for plan gates on its own + tickets, via `approve rev-N` comments. Live **confirm-first** prompts (C2) stay + Matrix-primary — polling latency makes a ticket comment a poor interactive + channel — with the outcome written back to the ticket as the record. +- **Trust:** a driving ticket is an instruction channel into an infrastructure- + changing agent → action-triggering ticket creation is operator-scoped (C2). + +### C5. Deployment & Isolation + +- Longer-lived, **stateful VM** (not disposable). +- Has **Tailscale** + scoped credentials to reach infrastructure across both sites. +- Updated **in place** via the container pipeline, **restart gated on an idle flag**. + +--- + +## PART D — NOTES / KNOWLEDGE (Outline) + +Shared knowledge store for both agents (sysadmin-primary). Not a message bus — see +note below. + +Requirements: +- **Read — runbooks as context.** Procedures written in Outline are fetched before + acting, constraining the agent toward *your* procedures over improvised ones. +- **Write — durable record.** Agent writes change logs / incident notes / findings + back to Outline: searchable, off-box, and an extra externalized record that + survives volume loss. +- Access via Outline **REST API**, dedicated service account, token from Bitwarden. + +**Deliberately rejected:** using a shared notes store (Outline *or* an Obsidian vault) +as an **agent-to-agent message bus**. It's unstructured, race-prone (last-write-wins, +no delivery semantics), and a shared read-write store both agents touch would +undermine the dev/sysadmin isolation boundary. Agent coordination, if ever needed, +goes through the **human**, **Matrix**, or a **Vikunja ticket** handed from one agent +to another — or a real message queue if genuine multi-agent async ever arrives. + +--- + +## PART E — SUPPORTING SERVICES (installed) + +**Status: all supporting services are installed and running.** What remains for each +is per-agent *configuration* (service accounts, scoped tokens, rooms), created as +each agent is built — not standing up the service itself. + +- **Matrix homeserver** (Synapse/Conduit): ✅ installed. *Remaining per-agent:* create + the two bot identities (`@nisse-dev`, `@nisse-ops`), one room per agent + `#ops`. +- **Gitea:** ✅ installed. *Remaining per-agent:* `nisse-dev` service account + scoped + token; enable native-JSON webhooks or confirm API polling access. +- **LiteLLM proxy:** ✅ installed, and **on the path from the first LLM call** (A5) — + it is the agent VM's single LLM destination, providing per-agent virtual keys with + budget caps, central logging, vendor swap, fallback routing, and the Anthropic + passthrough. *Remaining per-agent:* mint a virtual key; configure routes for the + local inference host and each frontier vendor, with keys sourced from Bitwarden. +- **Local inference host:** 🔜 **decided in principle, hardware pending** — a + 4090-class server (Part H). Interim: a Jetson Nano running Ollama validates the + plumbing, and the `executor` role maps to a Haiku-class frontier model via LiteLLM + until the server arrives. Everything in A5, A12, and A15 assumes the eventual host + sits on the LAN behind LiteLLM. +- **Vikunja:** ✅ installed. *Remaining (sysadmin):* service account + scoped token; + webhook/poll config (Part C4). +- **Outline:** ✅ installed. *Remaining per-agent:* service account + API token + (Part D). +- **Bitwarden/Vaultwarden:** ✅ installed. *Remaining per-agent:* dedicated + minimally-scoped machine account + bootstrap token (A7). +- **LLDAP + Authelia:** ✅ installed. *Remaining per-agent:* create `svc-nisse-dev` / + `svc-nisse-ops` principals, scoped groups, and the narrow unattended-auth exception + (A8). + +--- + +## PART F — DEPLOYMENT MODEL (both agents) + +- **Containers + persistent volumes.** Stateless image; named volumes carry state + across redeploys (and allow optional persistence between deployments). +- **Image delivery:** Gitea Actions → Gitea OCI registry → Watchtower for the *pull*. +- **Gate the restart, not the pull:** an idle/"safe-to-restart" flag; Watchtower must + never restart mid-work. *Unplanned* interruptions (reboot/crash) are handled by the + durability layers (A2/A3), not by restart timing. +- **Survive reboots:** VM + Docker service set to start on boot; `restart: + unless-stopped`; ZFS snapshots of the state dataset (scheduled + on each approval). +- **Terraform + cloud-init:** deferred. When adopted, `bpg/proxmox` + cloud-init, + VM definitions as code, clean `destroy`/`apply`; the dev agent becomes a candidate + for immutable rebuilds. + +--- + +## PART G — BUILD ORDER + +1. **State store + durability (A2/A3)** — critical path; everything sits on it. +2. **Transport (A1, Matrix) + control loop/gating (A4)** with a **stub agent** that + just echoes plans — get the approval loop feeling right before any real LLM. +3. **LiteLLM + LLM client interface (A5).** The interface must exist at the **first + real LLM call**, not as a later cutover. Mint a virtual key with a budget cap; + verify prompt caching survives the passthrough. +4. **Canonical log schema and renderer (A10).** Before any transcript accumulates in a + provider-shaped form that would have to be migrated later. +5. **Local executor loop (A11/A12/A15)** — role resolution pinned at task start, phase + policy, failure-count escalation, reduced tool manifest, constrained decoding. +6. **Gitea layer** — polling intake, PR creation, comment/commit handling. +7. **Delegation harness + plan-gate integration (A13/A14).** Begin capturing the eval + set here (A16); the records are free once the composite events exist. +8. **Browser tool (A6)** for `nisse-dev` — test apps on the sealed network. +9. First **genuinely unattended milestone** (budget caps + ZFS rollback earn their keep + here). Crash-atomic writes and reconcile-on-boot must be in place first. +10. **Extract `nisse-core`**; build the **Sysadmin Agent** (`nisse-ops`): policy + engine (C2), SSO + scoped browser (A8/A6), secrets (A7), Vikunja control (C4), + Outline runbooks (D), scheduled checks, alert intake. + - Under a monorepo, extraction is a **package split** within one tree: move code + behind a package boundary, no release machinery required. + - Under three repos, it is a **repo split plus a version-pinning contract**: + `nisse-core` gets tagged releases, and `nisse-dev` pins a version rather than + tracking `main`, or every core change becomes a coordinated two-repo bump. + This is the step the open repo-layout decision (Part H) actually bites on. + +--- + +## PART H — OPEN DECISIONS + +- **Inference hardware — resolved in principle, no longer blocking.** Target: a + **4090-class host (24GB VRAM)** running a **24–32B open model** at 4–5-bit + quantization (candidates: Devstral Small 24B, Qwen3-32B / Qwen3-30B-A3B, + GPT-OSS-20B). Models of this class are competent at tool calling and mechanical + edits, which puts the system in "tiering genuinely load-bearing" territory rather + than "local model as dispatcher." Consequences: + - **Context, not weights, is the VRAM pressure.** A 32B at 4-bit leaves roughly + 5GB for KV cache, and agent loops render long contexts. Model size vs. context + length vs. KV-cache quantization is an empirical trade — tune it, and the + thresholds that depend on local capability (A12 failure count, A14 consult + budget, A15 manifest size), at build-order step 5 on the real hardware. + - **Interim posture:** until the server exists, the `executor` role points at a + cheap frontier model (Haiku-class) through LiteLLM — a config entry under A11, + nothing structural, so the build order does not stall. The Jetson Nano validates + plumbing (Ollama → LiteLLM → agent) only; nothing learned from models it can run + transfers to threshold tuning. + - If the 24GB class proves insufficient in practice, additional hardware is the + fallback — the role indirection (A11) makes that swap invisible to the agents. + +The rest resolve when reached, not now. + +- **Repository layout:** one monorepo with three packages vs. three separate repos. + Three repos match the build order and give each artifact its own CI and release + cadence, at the cost of version-pinning `nisse-core` from Gitea and coordinating + bumps every time the shared interface moves. A monorepo is less friction while + `nisse-core` is still being extracted from a working `nisse-dev` and its seams are + still moving. Splitting later is easy; merging later is not. Decide by step 10. +- **Sysadmin browser access to network gear:** standing vs. per-task hand-off. The + lock-yourself-out risk argues for per-task. +- **Policy action taxonomy (C2):** the concrete list of which operations are + allow vs. confirm. Best decided with the agent half-built in front of you. +- **Approved-plan externalization — decided: Gitea-backed from the start.** Plans are + posted as issue comments on the project repo; the approval comment (author, + timestamp, revision tag) is the durable off-box record, authenticated by Gitea's + own auth. The state store keeps only a pointer ("gate 3 satisfied by comment 8841"), + so boot-time reconciliation (A3) can re-derive gate state from Gitea even after + state-volume loss — external truth wins (A2). Approval mechanics and revision + binding are specified in A4. + +--- + +## Repository Layout + +``` +nisse-core/ # shared library (extracted after nisse-dev works) + transport/ # matrix | telegram | slack (A1) + state/ # persistence, reconcile, migration (A2/A3) + control/ # loop, state machine, gating, ask_human (A4) + llm/ # client interface, role resolution (A5) + log/ # canonical event log + renderer (A10) + delegation/ # delegation harness, ask_expert (C1/C3) + browser/ # Playwright tool (A6) + secrets/ # Bitwarden machine-account client (A7) + sso/ # LLDAP identity + auth path (A8) + logging/ # structured operational logs (A9) + +nisse-dev/ # app: sealed VM (Part B) + workflow (arch/milestone gating), Gitea PR layer, allow-all sandbox + browser → internal test network only + +nisse-ops/ # app: separate host, Tailscale (Part C) + policy engine, tiered confirmation, scheduled checks + Vikunja ticket intake, Outline runbooks/records + browser → admin UIs (API-first, UI fallback), confirm-first gating + +infra/ # (future) Terraform + cloud-init (Part F) + nisse-dev.tf | nisse-ops.tf | templates/ +``` + +Shown as three sibling trees. Under a monorepo they become three packages under one +root; under three repos, three checkouts. Either way the module boundaries are the +same, which is the point of deciding this late (Part H). + +`log/` carries the canonical event log and its renderer (A10); `delegation/` carries +the delegation harness and `ask_expert` (A13/A14).