Revise the deployment model: an agent host and tool VMs

Agents run as containers on one stable agent host that keeps their
state and secrets and never runs a command a model chose. Model-chosen
commands run on disposable tool VMs over SSH: a build VM per project
for nisse-dev, its own tool VM for nisse-ops. Secrets are a set per
agent, handed out per operation by the harness and redacted from tool
output. MCP servers stay on the agent host as the one exception.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

docs/Architecture.md

@@ -1,8 +1,9 @@
 # Nisse — Architecture
 
-**Status: agreed (PR #83).** How and in what order it gets built is in
+**Status: agreed (PR #83); deployment model revised (PR #90).** How and in what
-`docs/Development-Plan.md`. It replaced `docs/Architecture-Build-Spec.md`, which
+order it gets built is in `docs/Development-Plan.md`. It replaced
-remains in git history; its section letters (A–H) are not reused here.
+`docs/Architecture-Build-Spec.md`, which remains in git history; its section letters
+(A–H) are not reused here.
 
 This document says what Nisse is, how its parts fit together, and why. It records
 decisions and the reasons for them. It doesn't record status or sequencing.
@@ -34,7 +35,7 @@ agents, one string is used everywhere a site appears, including MCP server
 instances (X3).
 
 All artifacts live as packages in **one repository, `nisse`**. The repository
-boundary was never a trust boundary in this design (the VM is, X1); a monorepo
+boundary was never a trust boundary in this design (the hosts are, X1); a monorepo
 avoids tripled admin for a solo operator and keeps `nisse-core` easy to change
 while its seams are still settling.
 
@@ -54,7 +55,7 @@ The agents run **local-first** (P7): a model on the operator's own inference
 hardware does the work, and frontier models are called in when it needs help.
 
 The agents share one library, `nisse-core` (Part L). Each agent adds what is
-specific to its job: `nisse-dev` works on software projects in a sealed VM
+specific to its job: `nisse-dev` works on software projects on sealed build VMs
 (Part D); `nisse-ops` operates live infrastructure across two sites (Part O).
 
 ### P2. Threat model
@@ -132,6 +133,8 @@ truly irreversible also sits behind a hard wall.
 - Where a tool offers a **dry run** or a preview, the agent uses it before the
   real action.
 - Build artifacts and caches are **disposable**; nothing depends on them.
+- **Tool VMs are disposable** (X1): a broken one is re-created, not repaired. What
+  must survive lives on the agent host.
 
 ### P6. Structural, not model judgment
 
@@ -163,9 +166,10 @@ safety, such as choosing which ready work item to switch to (L1).
 
 ### P8. Carried principles
 
-- **Isolation matched to blast radius.** `nisse-dev` is disposable and sealed;
+- **Isolation matched to blast radius.** `nisse-dev`'s work runs on disposable,
-  `nisse-ops` holds live-infrastructure reach. They never share a running
+  sealed build VMs; `nisse-ops` holds live-infrastructure reach. They never share a
-  instance, and every agent gets its own VM (X1).
+  running instance, and no command a model chose runs where agents keep their state
+  and secrets (X1, L13).
 - **Stateless images, stateful volumes.** Containers are replaceable; durable
   state lives on volumes that survive redeploys and unexpected reboots.
 - **Structured channels for machines, prose channels for humans.** Machine intake
@@ -196,6 +200,12 @@ Every agent's work is organised into three kinds of unit.
   deduplicate: a later finding for the same problem attaches to the open work item
   instead of opening a second (a stable fingerprint per finding).
 - **The operator is always a source** of work items.
+- **Another agent can be a source.** A request from one agent to another (for
+  example, `nisse-dev` asking `nisse-ops` for a build VM) arrives through the
+  receiving agent's normal intake, such as a ticket, and becomes a work item there.
+  It carries no authority of its own: the receiving agent applies its own rules
+  (P3, L3) exactly as for any other work item, and the requester waits on it like
+  any external dependency.
 - **A work item is the unit of state:** its conversation, plans and approvals,
   pending questions, pinned role mapping (L9) and transcript (L7) all belong to it.
   Its ID is minted locally when it opens and doubles as its correlation ID in logs.
@@ -517,11 +527,11 @@ is fixed.
 
 ### L9. Models, roles and helpers
 
-**LiteLLM in front of everything.** Each agent VM has exactly one LLM destination:
+**LiteLLM in front of everything.** Each agent has exactly one LLM destination:
 LiteLLM, which fans out to the local inference host and to frontier vendors. It
 holds every vendor key (the agent holds one LiteLLM token), enforces budget caps
 per virtual key, logs centrally, and routes fallbacks. Adding a vendor never
-touches an agent VM.
+touches an agent.
 
 **One client interface:** `complete(events, tools, model) → events`, over the
 canonical log. Delegations use LiteLLM's passthrough route with a per-vendor
@@ -589,7 +599,7 @@ later operator-written procedures.
   each agent's memory, and `procedures` (the operator's). Sharing grants read
   access only; concurrent writers never exist.
 - **`nisse-dev` gets private space only.** Shared infrastructure documents are a
-  map of exactly what its sealed network keeps it from (D7).
+  map of exactly what its sealed network keeps its builds from (D7).
 - **Each agent authenticates as its own Outline user,** with its own API token. A
   token from the operator's account would silently void every permission here.
 - **Collections and documents are identified by ID** from configuration and state,
@@ -597,7 +607,7 @@ later operator-written procedures.
 
 **Never an instruction channel, never an approval channel.** Work is never read
 out of the knowledge store, and an agent never writes there to make another agent
-act. Coordination between agents goes through the operator, Matrix, or a ticket.
+act. Coordination between agents goes through the operator, Matrix, or a ticket (L1).
 
 **Corrections go through conversation.** The operator tells the agent, which
 records the correction verbatim (L6). The operator keeps write permission as a
@@ -635,17 +645,28 @@ contents, dependencies, web pages, alert payloads, log lines.
   can bias which ready item is chosen next; the switch notice makes it visible.
 - **Each agent is checked against the lethal trifecta** — private data, untrusted
   content, and a way to send data out. Damage needs all three. `nisse-dev` holds
-  little private data and can reach only LiteLLM, Gitea and package registries;
+  little private data, and the build VMs where untrusted content meets command
-  `nisse-ops` has all three legs and relies on its hard walls and step sequences
+  execution hold no credential beyond their project's (D7); `nisse-ops` has all
-  (O2).
+  three legs and relies on its hard walls and step sequences (O2).
 
 ### L12. Secrets, identity and logging
 
-**Secrets.** Fetched at run time from Vaultwarden machine accounts, never baked
+**Secrets.**
-into images. One minimally scoped machine account per agent and one per MCP server
+- **Each agent has its own set of secrets,** held on the agent host (X1) and
-instance (X3); each has a single bootstrap token on its own host, and nothing
+  mounted only into that agent's container. No agent can read another's. They are
-else is stored there. Only credentials go in secret stores; every other setting
+  never baked into images.
-lives in reviewable configuration files.
+- **An agent may give its set structure.** `nisse-dev` namespaces it by project
+  (D7); `nisse-ops` may do the same by site. The framework only needs names.
+- **The harness hands secrets out per operation.** Which secrets a command or tool
+  call receives, and how (environment or stdin, never a command line), is decided
+  by code and configuration, never by a model.
+- **Every value handed out is redacted** from tool output before it reaches the
+  transcript, a model or a log. A test that prints its environment must not put a
+  key into a frontier request.
+- **MCP server instances** (X3) each have their own minimally scoped credentials,
+  given only to that instance.
+- **Only credentials go in secret stores;** every other setting lives in reviewable
+  configuration files.
 
 **Identity.** Each agent is its own LLDAP principal (`svc-nisse-dev`,
 `svc-nisse-ops`) in a tightly scoped Authelia group, never the operator's account.
@@ -659,6 +680,33 @@ event, work item, cause). Every classifier verdict, escalation and external effe
 is logged. **Logs are diagnostic, never load-bearing:** nothing reads them to
 derive state.
 
+### L13. Tools and execution
+
+Tool calls come in three kinds, and the framework allows no others.
+
+| Kind | Examples | Where it runs |
+|---|---|---|
+| **Control tools** | `submit_plan`, `ask_human`, `finish`, `ask_expert` | In the harness. They act only on the agent's own state and conversation |
+| **Execution tools** | Running a command; reading, writing and listing files | On a remote host, over SSH, through `execute(host, …)` |
+| **MCP tools** | Proxmox, UniFi, Gitea APIs (X3) | In MCP server containers beside the agent |
+
+- **`execute(host, command, *, secrets, stdin)` is the only way to run anything.**
+  There is no local execution path in the framework, so the agent host never runs a
+  command a model chose. File tools are SSH operations too.
+- **The harness binds the host,** not the model. For `nisse-dev` it comes from the
+  work item: that project's build VM (D7). Where an agent must let the model choose
+  (`nisse-ops` running a command on its tool VM or on a device), the model picks
+  from the hosts the harness allows for that work item; any other host is refused
+  before a connection is made.
+- **MCP tools are the one exception,** made deliberately: their servers run fixed
+  code against an API, and the model chooses only arguments. They pass through the
+  action classes (P3) and the classifier (L4) like everything else, and their output
+  is redacted (L12).
+- **Tests use SSH too:** a throwaway `sshd` container stands in for a tool VM, so the
+  code path under test is the one used in production.
+- **The action classifier is not a tool.** It is a model call made by the harness
+  and executes nothing, so it runs on the agent host.
+
 ---
 
 ## Part D — `nisse-dev`
@@ -708,12 +756,22 @@ agent's state (L8).
 
 ### D3. The workspace
 
+**Each project has its own build VM** (X1), where its workspace lives and its
+commands run.
+- **Kept between work items,** so installed toolchains and package caches stay
+  warm, and isolated from every other project's.
+- **Collected when idle.** A build VM whose project has been idle long enough may be
+  destroyed, and is re-created when work on the project resumes.
+- **Re-created from a template** plus the toolchains recorded for the project (see
+  below). Work pushed to Gitea survives; a commit not yet pushed may be lost when a
+  VM is wiped, and that is accepted.
+
 **A fresh, clean workspace at `use <project>`.**
 - Fetch from the repository, and check out a clean tree at the latest base branch.
 - Anything left in a previous workspace (uncommitted changes, local branches,
   unpushed commits) is **set aside**, never silently discarded or carried over,
   and reported to the operator.
-- Build artifacts and caches never survive into a new task.
+- Build artifacts and caches in the workspace never survive into a new task.
 
 The milestones of one work item share that workspace, since they build on each
 other, often before earlier PRs are merged.
@@ -724,12 +782,12 @@ the project needs (from its files, e.g. `Cargo.toml`, `pyproject.toml`,
 installed **before any planning**. The environment's toolchain versions become
 facts in the transcript (L6), so plans match reality.
 
-**The agent installs what it needs on its own VM.** The VM is disposable and is
+**The agent installs what it needs on the project's build VM.** The VM is
-the boundary (D7), so installing packages and toolchains there is class 2 (P3): a
+disposable and is the boundary (D7), so installing packages and toolchains there
-contained side effect. Every install is recorded as a fact (L6) and in the
+is class 2 (P3): a contained side effect. Every install is recorded as a fact (L6)
-project's scope memory, so the environment can be rebuilt; toolchains that keep
+and in the project's scope memory, so the VM can be rebuilt; toolchains that keep
-being needed are baked into the VM image (X4). Installing on any host other than
+being needed are baked into the template (X4). Installing on any other host stays
-its own stays class 3.
+class 3.
 
 **Repairing the environment is in scope.** Within a task, the agent may clean
 caches and build artifacts or reinstall project dependencies to fix a broken
@@ -746,6 +804,12 @@ a corrupt `.pyc`.
   the work produced). Anything left out is reported.
 - **"Changes to publish" are counted from where the milestone started,** never
   from a possibly stale remote branch.
+- **Pushes are harness operations on the build VM.** The harness runs `git push`
+  there, handing the push key over stdin to a short-lived `ssh-agent` for that one
+  push (L12); the model's commands never receive it. Nothing is synced through the
+  agent host: Gitea is the only copy that has to survive.
+- **Pull requests are opened from the agent's container** with its Gitea API token,
+  which never reaches a build VM.
 - **Publishing is idempotent** (L8): an open PR for the branch is adopted and
   updated, never duplicated.
 - **A milestone that completes without changes is done, not failed,** when it
@@ -779,29 +843,47 @@ a corrupt `.pyc`.
 | Class | Examples |
 |---|---|
 | 1 | Reading the repo, running tests and builds in the workspace or a test container, querying Gitea |
-| 2 (in scope) | Editing the workspace, committing to milestone branches, pushing them, opening and updating PRs, commenting on its issues and PRs, cleaning caches, installing project dependencies, packages and toolchains on its own VM |
+| 2 (in scope) | Editing the workspace, committing to milestone branches, pushing them, opening and updating PRs, commenting on its issues and PRs, cleaning caches, installing project dependencies, packages and toolchains on the project's build VM |
-| 3 | Merging to the default branch (unless the trust dial allows), deleting remote branches or others' work, changing repository settings, anything outside the work item's repo or its own VM |
+| 3 | Merging to the default branch (unless the trust dial allows), deleting remote branches or others' work, changing repository settings, anything outside the work item's repo or its project's build VM |
 
 Its scope sources are the approved architecture, the build plan and each
 milestone's marking.
 
 ### D7. Deployment and isolation
 
-- **Its own VM** (X1), with a real Docker daemon, because it builds and runs
+**The agent** runs as a container on the agent host (X1), holding its state, its
-  containers. The VM is the trust boundary for unattended command execution.
+secrets and its Gitea API token. It reaches LiteLLM, Gitea, Matrix, and SSH to its
-- **Sealed:** its egress allowlist (X2) permits only LiteLLM, Gitea and package
+build VMs, and nothing else. It never runs a command a model chose (L13).
-  sources (language registries and OS package mirrors); there is no ingress. Denial covers RFC1918 (except the specific
+
-  LiteLLM and Gitea addresses) and the tailnet's `100.64.0.0/10`, so it can't reach
+**Build VMs,** one per project (D3), are where the work happens:
-  the NAS, Proxmox or the gateways. The inference host sits behind LiteLLM, so it is
+- **A real Docker daemon,** because projects build and run containers. The VM is
-  not a second exception.
+  the trust boundary for unattended command execution.
-- **No Tailscale** on this VM; it would route around the firewall.
+- **Sealed:** egress allows package sources (language registries and OS package
-- **Its own VLAN,** and test services on predictable ports (e.g. 18000–18099), so
+  mirrors) and Gitea's git endpoint; there is no ingress except SSH from the agent
-  opening access later is a firewall change.
+  host. Denial covers RFC1918 (except those specific addresses), the agent host,
+  and the tailnet's `100.64.0.0/10`, so a build can't reach the NAS, Proxmox, the
+  gateways or the agent's state.
+- **A project may add egress its tests need,** for example LiteLLM and a Matrix test
+  homeserver for the `nisse` project itself. Additions are declared in the
+  project's configuration and reviewed like any allowlist (X2).
+- **No Tailscale** on build VMs; it would route around the firewall.
+- **On their own VLAN,** with test services on predictable ports (e.g.
+  18000–18099), so opening access later is a firewall change.
 - **Test containers on an internal Docker network with no route out.**
-- **Build storage on a separate mount** from the state volume.
 - **A browser** (a shared Playwright tool, headless) **reaches only the internal
   test network,** to exercise web apps the agent built and reproduce bugs. No SSO.
 
+**Secrets are namespaced by project** (L12), e.g. `nisse/litellm-key`,
+`nisse/gitea-test-token`. A build VM's commands receive only their own project's
+secrets, and each secret's configuration says whether the model's commands may see
+it (a test credential the project's tests need) or only harness operations use it
+(the push key). Anything the model's commands receive must be scoped for testing,
+never able to touch production.
+
+**Provisioning.** Build VMs are created and destroyed by Terraform from the
+template. Eventually `nisse-dev` asks `nisse-ops` for them, as an agent-to-agent
+request (L1); until then the operator provisions them.
+
 ---
 
 ## Part O — `nisse-ops`
@@ -849,8 +931,8 @@ exactly the steps written. This is `nisse-ops`' scope source for the classifier
 - **Approvals expire.** A plan not started within the expiry window must be approved
   again.
 - **Policy is small, conservative, and data:** a versioned file declaring each MCP
-  instance's privilege, protected targets (the NAS, gateways, the agents' own
+  instance's privilege, protected targets (the NAS, gateways, the agent host and
-  hosts), and each elevated tool's target argument. Anything unclassified is
+  tool VMs), and each elevated tool's target argument. Anything unclassified is
   treated as elevated and protected. Server-declared tool hints can make a tool
   stricter, never looser. A model may propose entries for a new server as a PR; no
   classification happens at run time.
@@ -894,40 +976,66 @@ a routine fix approved by hand often enough).
 
 ### O6. Deployment
 
-- **Its own long-lived, stateful VM** (X1).
+- **A container on the agent host** (X1), with its MCP servers beside it (X3).
-- **Tailscale** and scoped credentials to reach infrastructure at both sites.
+- **Its own tool VM,** long-lived, for network-management tools and the commands it
-- **Its MCP servers run on its own Docker host;** its egress allowlist therefore
+  runs (L13). It holds Tailscale and the reach those commands need at both sites;
-  includes the Proxmox and UniFi endpoints at both sites.
+  like any tool VM it can be rebuilt from its template.
+- **Its egress allowlist** includes the Proxmox and UniFi endpoints at both sites,
+  for its MCP servers.
+- **Accepts provisioning requests** from other agents (L1): creating and
+  destroying build VMs for `nisse-dev` through Terraform, under its own rules (O2).
 - **Updated in place,** with restarts gated on an idle flag (X4).
 
 ---
 
 ## Part X — Deployment model (all agents)
 
-### X1. Every agent gets its own VM
+### X1. The agent host and tool VMs
 
-No agent runs as a container on a shared host, and no two agents share a VM.
+**One agent host,** a stable, long-lived VM.
-- **The VM is the isolation boundary between agents:** a compromised or confused
+- **Each agent runs on it as a container,** with its own volume (state, transcripts,
-  agent can't reach another's state, credentials or processes.
+  memory), its own secrets (L12) and its own network. No agent can reach another's
-- **Per-agent attribution:** everything an agent holds is issued to its own
+  state, credentials or processes.
-  identity, so every upstream action traces to one agent.
+- **Its MCP servers run beside it** as containers (X3).
+- **It never runs a command a model chose** (L13). That is what makes it safe for
+  agents to share it: nothing on it executes untrusted input.
 
-### X2. Egress policy per agent instance
+**Tool VMs** are where model-chosen commands run, reached only over SSH from the
+agent host. Each agent has the tool VMs its work needs:
+- `nisse-dev`: one build VM per project, collected when idle and re-created on
+  demand (D3, D7);
+- `nisse-ops`: its own tool VM, for network-management tools (O6);
+- a purely conversational agent: none.
 
-- **Default-deny allowlist per agent instance,** declared as a versioned artifact
+Tool VMs are **disposable** (P5): each is built from a template plus recorded
-  next to the agent's definition, each entry justified by the component that
+installs, and nothing on one has to survive.
-  needs it.
+
-- **Declared in one place, enforced in several:** at the VM, the Docker network,
+**Per-agent attribution:** everything an agent holds is issued to its own
-  and the UniFi firewall. Network-layer enforcement survives compromise of the
+identity, so every upstream action traces to one agent.
-  agent's host.
+
-- **The allowlist is the union** of what the agent and its MCP servers reach.
+### X2. Egress policy
-- **The tailnet is `100.64.0.0/10`,** not RFC1918; deny it explicitly.
+
+- **Default-deny allowlists,** declared as versioned artifacts next to the agent's
+  definition, each entry justified by the component that needs it.
+- **For each agent container,** the union of what the agent and its MCP servers
+  reach, enforced by its Docker network and the agent host's firewall. The UniFi
+  firewall sees the agent host as one address, so these allowlists rest on the
+  host; that is acceptable because nothing on the host executes untrusted input.
+- **For each tool VM,** enforced at the VM, its Docker network and the UniFi
+  firewall on its VLAN. Network-layer enforcement survives compromise of the VM,
+  which matters here because this is where untrusted input meets command execution.
+- **Tool VMs never reach the agent host,** except as replies on the SSH sessions it
+  opens.
+- **The tailnet is `100.64.0.0/10`,** not RFC1918; deny it explicitly wherever it
+  isn't needed.
 
 ### X3. MCP servers
 
-- **One instance per agent VM,** as containers on that agent's Docker host. A shared
+- **One instance per agent,** as containers beside it on the agent host. A shared
   process would mean a shared credential, losing attribution in the upstream API's
   own audit log. Duplication is nearly free for stateless servers.
+- **The one exception to "execution happens on tool VMs"** (L13): a server runs
+  fixed code against an API, and the model supplies only arguments.
 - **One instance per (service, site, privilege).** The site is part of the server's
   identity, so a wrong-site action needs a tool that isn't mounted, not just a wrong
   argument. Read-only and elevated instances have separate credentials, each from
@@ -937,17 +1045,19 @@ No agent runs as a container on a shared host, and no two agents share a VM.
   extra round trips for a local driver, no discovery of capabilities not already
   sought, opaque arguments, and a dynamic tool list that defeats prompt caching.
 
-### X4. Containers, images and restarts
+### X4. Containers, images, VMs and restarts
 
-- **Stateless images, named volumes** for state.
+- **Stateless images, named volumes** for state, on the agent host.
 - **Delivery:** Gitea Actions → Gitea's OCI registry → Watchtower pulls.
 - **Gate the restart, not the pull:** an idle flag; Watchtower never restarts an
   agent mid-work. Unplanned interruptions are handled by recovery (L8).
 - **Third-party MCP server images are digest-pinned** and updated deliberately,
   never by Watchtower: they hold root-equivalent credentials.
+- **Tool VMs come from templates,** provisioned by Terraform (`bpg/proxmox` and
+  cloud-init). Installs that keep recurring are baked into the template.
 - **Survive reboots:** VMs and Docker start on boot, containers restart unless
-  stopped, and the state dataset has ZFS snapshots (scheduled, and on each
+  stopped, and the agent host's state dataset has ZFS snapshots (scheduled, and on
-  approval).
+  each approval). Tool VMs are not snapshotted; they are rebuilt.
 
 ### X5. Supporting services
 
@@ -981,10 +1091,15 @@ it uses; nothing is shared between agents.
 - **When a work item must have an external record.** Plan-driven dev work and
   ticket-driven ops work already do; whether every item with an approval must.
 - **Fallback browser access to network gear:** standing, or handed off per task.
-- **Relocating stateful MCP servers** off agent VMs, one at a time, if one ever
+- **Relocating stateful MCP servers** off the agent host, one at a time, if one
-  justifies losing per-agent attribution.
+  ever justifies losing per-agent attribution.
-- **Infrastructure as code** (Terraform with `bpg/proxmox` and cloud-init) for agent
+- **Where secrets come from.** Each agent's set lives on the agent host (L12); whether
-  VMs.
+  it is filled by hand, synced from Vaultwarden, or written by Terraform.
+- **How `nisse-ops`' MCP servers reach the second site** from the agent host
+  (Tailscale on the host, with other agents' networks denied the tailnet, or a
+  subnet route), since Tailscale on the host serves every container on it.
+- **How long idle before a build VM is collected,** and whether collection needs the
+  operator's word.
 
 ---
 
@@ -1008,9 +1123,11 @@ history):
 - **Documents as the dev workflow (D2),** a fresh workspace per task and toolchain
   preflight (D3), and git hygiene learned from the live runs (D4).
 - **Conversation (L5)** is a first-class requirement, not a command interpreter.
+- **Deployment (X1, L13),** revised in PR #90: one agent host with each agent as a
+  container, and disposable tool VMs (a build VM per project for `nisse-dev`) where
+  every model-chosen command runs over SSH, replacing a VM per agent.
 - **Kept, largely as they were:** the work model, durability and reconciliation,
   LiteLLM and the canonical log, delegation, the knowledge store, prompt injection,
-  secrets and identity, `nisse-ops`' approved step sequences, and the deployment
+  identity, and `nisse-ops`' approved step sequences.
-  model.
 - **Moved to the Development Plan (to be written):** build order, supporting-service
   status, and interim arrangements.