Merge pull request 'Add a bring-up script and a transcript dump; let nisse-dev install on its own VM (M0.2)' (#86) from bringup-script into main

docs/Architecture.md

@@ -93,7 +93,7 @@ Every action any agent can take falls into one of three classes:
 |---|---|---|---|
 | **1. No side effects** | Observes; changes nothing outside a disposable sandbox | Reading files, querying APIs, searching, viewing logs and metrics, running a build or tests in a throwaway environment | Always allowed |
 | **2. Contained side effects** | Changes something, but the change stays within the agent's own area or has a known undo | Editing the workspace, committing to the agent's branch, opening a PR, writing the agent's own notes, starting a test container; for ops, a step with a known undo inside an approved plan | Allowed when in scope; logged and reported |
-| **3. Dangerous** | Irreversible, visible beyond the agent's own area, or outside what was agreed | Merging to a default branch, deleting data, changing a firewall, installing software on a host, contacting third parties, spending beyond a budget | Needs discussion, unless an approved plan names the exact action |
+| **3. Dangerous** | Irreversible, visible beyond the agent's own area, or outside what was agreed | Merging to a default branch, deleting data, changing a firewall, installing software on a host the agent doesn't own, contacting third parties, spending beyond a budget | Needs discussion, unless an approved plan names the exact action |
 
 **"In scope"** means within what the operator agreed to: the work item's task,
 an approved plan, a pre-approved milestone (D2), or an approved step sequence
@@ -716,12 +716,15 @@ other, often before earlier PRs are merged.
 **Toolchain preflight.** When the workspace is prepared, the agent determines what
 the project needs (from its files, e.g. `Cargo.toml`, `pyproject.toml`,
 `package.json`, or a declaration in the repo) and checks it is available. A gap is
-reported **before any planning**, with the options. The environment's toolchain
+installed **before any planning**. The environment's toolchain versions become
-versions become facts in the transcript (L6), so plans match reality.
+facts in the transcript (L6), so plans match reality.
 
-**Toolchains come from the environment, not from the agent improvising.**
+**The agent installs what it needs on its own VM.** The VM is disposable and is
-Installing software is class 3 (P3). How toolchains are provided (project dev
+the boundary (D7), so installing packages and toolchains there is class 2 (P3): a
-containers on the VM's Docker daemon, or the VM image) is an open question (Q).
+contained side effect. Every install is recorded as a fact (L6) and in the
+project's scope memory, so the environment can be rebuilt; toolchains that keep
+being needed are baked into the VM image (X4). Installing on any host other than
+its own stays class 3.
 
 **Repairing the environment is in scope.** Within a task, the agent may clean
 caches and build artifacts or reinstall project dependencies to fix a broken
@@ -771,8 +774,8 @@ a corrupt `.pyc`.
 | Class | Examples |
 |---|---|
 | 1 | Reading the repo, running tests and builds in the workspace or a test container, querying Gitea |
-| 2 (in scope) | Editing the workspace, committing to milestone branches, pushing them, opening and updating PRs, commenting on its issues and PRs, cleaning caches, installing project dependencies from package registries |
+| 2 (in scope) | Editing the workspace, committing to milestone branches, pushing them, opening and updating PRs, commenting on its issues and PRs, cleaning caches, installing project dependencies, packages and toolchains on its own VM |
-| 3 | Merging to the default branch (unless the trust dial allows), deleting remote branches or others' work, installing system software, changing repository settings, anything outside the work item's repo |
+| 3 | Merging to the default branch (unless the trust dial allows), deleting remote branches or others' work, changing repository settings, anything outside the work item's repo or its own VM |
 
 Its scope sources are the approved architecture, the build plan and each
 milestone's marking.
@@ -782,7 +785,7 @@ milestone's marking.
 - **Its own VM** (X1), with a real Docker daemon, because it builds and runs
   containers. The VM is the trust boundary for unattended command execution.
 - **Sealed:** its egress allowlist (X2) permits only LiteLLM, Gitea and package
-  registries; there is no ingress. Denial covers RFC1918 (except the specific
+  sources (language registries and OS package mirrors); there is no ingress. Denial covers RFC1918 (except the specific
   LiteLLM and Gitea addresses) and the tailnet's `100.64.0.0/10`, so it can't reach
   the NAS, Proxmox or the gateways. The inference host sits behind LiteLLM, so it is
   not a second exception.
@@ -959,8 +962,6 @@ it uses; nothing is shared between agents.
   way a misread must default to asking (L5).
 - **How far the local driver goes.** Which kinds of work it handles well and where
   a frontier driver is needed, measured on real tasks, not assumed (P7).
-- **Providing toolchains** for `nisse-dev`: project dev containers on its Docker
-  daemon, or the VM image, or both (D3).
 - **One local model for every agent.** The inference host keeps one model resident;
   different models per agent would reload weights on every switch. Preferred: one
   model good enough for every local role. Fallback: agents take turns holding the

docs/Development-Plan.md

@@ -80,10 +80,9 @@ test Matrix room, a budget-capped LiteLLM key; configuration in `~/nisse-e2e/`.
 - **A bring-up run on this box ends each phase** where it can be run live. What it
   finds is filed as issues and fixed before the next phase starts, or explicitly
   deferred.
-- **Bring-up guards on this box** (it isn't the sealed VM): the agent runs as an
+- **Bring-up runs on this development VM,** as root, like production. Guards: it
-  unprivileged user, touches only test resources, and keeps today's structural
+  touches only test resources, and keeps today's structural gates on in its
-  gates on in its bring-up configuration until the classifier and trust dial are
+  bring-up configuration until the classifier and trust dial are proven (Phase 3).
-  proven (Phase 3).
 - **This plan is revised as we go:** a milestone's scope may move once work shows
   its real shape, recorded here in the same PR.
 
@@ -100,10 +99,15 @@ test Matrix room, a budget-capped LiteLLM key; configuration in `~/nisse-e2e/`.
   section table this plan carried for the purpose.
 - Acceptance: `grep` finds no reference to the old spec or its section numbers.
 
-**M0.2 A bring-up script, and an unprivileged agent user on this box.** *Pre-approved.*
+**M0.2 A bring-up script.** *Pre-approved.*
-- One script: set aside the previous run's state, workspace and log; start the agent
+- `scripts/bringup.sh`: set aside the previous run's state, workspace and log;
-  as an unprivileged user; collect logs and the transcript afterwards.
+  export the ref under test; start the agent as a systemd unit, so it outlives the
-- Acceptance: a smoke run starts, answers `status`, and stops cleanly as that user.
+  shell that started it; collect the log and a readable transcript
+  (`python -m nisse_dev.dump`) afterwards.
+- The agent runs as root on this development VM, as it will on its own VM in
+  production (D7), so installing packages works as it will there.
+- Acceptance: a smoke run starts, answers `status`, stops cleanly, and `collect`
+  produces the log and transcript.
 
 ### Phase 1 — Foundations the new loop needs
 
@@ -211,11 +215,13 @@ on, then deliberately loosened in the bring-up configuration.
 
 ### Phase 4 — The environment
 
-**M4.1 Toolchain preflight.** *Pre-approved.* Detect, check, report before planning;
+**M4.1 Toolchain preflight.** *Pre-approved.* Detect what the project needs, check
-toolchain facts in the transcript (D3).
+it, and install what's missing before planning; toolchain facts in the transcript
+(D3).
 
-**M4.2 Providing toolchains.** *Needs discussion.* Decide dev containers vs the VM
+**M4.2 Recording installs.** *Pre-approved.* Every package or toolchain the agent
-image (Q), then build it.
+installs is recorded as a fact and in the project's scope memory, so the
+environment can be rebuilt; recurring ones are noted for the VM image (D3, X4).
 
 **M4.3 Environment repair.** *Pre-approved.* The driver may clean caches and
 reinstall project dependencies within a task, and reports it (D3).

nisse-dev/src/nisse_dev/dump.py

@@ -0,0 +1,96 @@
+"""A readable dump of a state store, for bring-up runs and debugging.
+
+    python -m nisse_dev.dump STATE_DB [--full]
+
+Opens the store read-only (SQLite `mode=ro`), so it is safe to run against a
+live agent's database: it never migrates, writes or locks it. Prints the
+agent's state, pending intents, and the canonical log, one event per line;
+long texts are shortened unless `--full` is given.
+"""
+
+import argparse
+import json
+import sqlite3
+import sys
+from pathlib import Path
+from typing import Any
+
+SHORT = 160  # characters of each text shown without --full
+
+
+def _connect(path: Path) -> sqlite3.Connection:
+    conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
+    conn.row_factory = sqlite3.Row
+    return conn
+
+
+def _clip(text: str, full: bool) -> str:
+    text = text.replace("\n", " ⏎ ") if not full else text
+    if full or len(text) <= SHORT:
+        return text
+    return text[:SHORT] + f" … [{len(text) - SHORT} more]"
+
+
+def _event_line(kind: str, payload: dict[str, Any], full: bool) -> str:
+    match kind:
+        case "user_message":
+            return f"USER [{payload.get('source', '')}]: {_clip(payload['text'], full)}"
+        case "assistant_text":
+            return f"ASSISTANT ({payload.get('model', '')}): {_clip(payload['text'], full)}"
+        case "tool_call":
+            args = json.dumps(payload.get("arguments", {}), ensure_ascii=False)
+            return f"CALL {payload['tool']} {_clip(args, full)}"
+        case "tool_result":
+            mark = "ok" if payload.get("success") else "FAILED"
+            return f"  result ({mark}): {_clip(str(payload.get('result', '')), full)}"
+        case "plan_submitted":
+            plan = payload["plan"]
+            return f"PLAN ({payload.get('model', '')}, {len(plan)} chars): {_clip(plan, full)}"
+        case "gate_opened":
+            return f"GATE {payload.get('gate', '')}: {payload.get('reason', '')}"
+        case _:
+            return f"{kind.upper()}: {_clip(json.dumps(payload, ensure_ascii=False), full)}"
+
+
+def render(path: Path, *, full: bool = False) -> str:
+    conn = _connect(path)
+    lines: list[str] = []
+    state = conn.execute("SELECT * FROM agent_state WHERE id = 1").fetchone()
+    if state is None:
+        lines.append("state: (none)")
+    else:
+        data = json.loads(state["data"])
+        gate = data.get("gate") or {}
+        lines.append(
+            f"state: {state['state']}  project: {state['project']}  "
+            f"task: {data.get('task_id')}  gate: "
+            + (f"{gate.get('name')} rev-{gate.get('rev')}" if gate else "none")
+        )
+    pending = conn.execute(
+        "SELECT id, kind, payload FROM intent_log WHERE status = 'pending'"
+    ).fetchall()
+    for row in pending:
+        lines.append(f"pending intent #{row['id']} {row['kind']}: {row['payload']}")
+    lines.append("")
+    for row in conn.execute("SELECT * FROM canonical_events ORDER BY id"):
+        payload = json.loads(row["payload"])
+        time = row["ts"][11:19]
+        lines.append(f"#{row['id']} {time} {_event_line(row['type'], payload, full)}")
+    conn.close()
+    return "\n".join(lines)
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("state_db", type=Path)
+    parser.add_argument("--full", action="store_true", help="don't shorten texts")
+    args = parser.parse_args(argv)
+    if not args.state_db.exists():
+        print(f"no state store at {args.state_db}", file=sys.stderr)
+        return 1
+    print(render(args.state_db, full=args.full))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

nisse-dev/tests/test_dump.py

@@ -0,0 +1,59 @@
+import io
+from pathlib import Path
+
+from nisse_dev.canonical import PlanSubmitted, ToolCall, ToolResult, UserMessage
+from nisse_dev.dump import main, render
+from nisse_dev.oplog import OpLog
+from nisse_dev.state import StateStore
+
+
+def store_with_a_run(path: Path) -> None:
+    store = StateStore(path, oplog=OpLog("nisse-dev", io.StringIO()))
+    store.transition(
+        "AWAIT_ARCH_SIGNOFF",
+        cause="test",
+        project="demo",
+        data={"task_id": "demo-1", "gate": {"name": "arch", "rev": 1}},
+    )
+    store.append_event(UserMessage(text="Add a truncate helper", source="matrix"))
+    store.append_event(PlanSubmitted(plan="x" * 500, model="opus"))
+    store.append_event(
+        ToolCall(call_id="c1", tool="run_command", arguments={"command": "ls"})
+    )
+    store.append_event(ToolResult(call_id="c1", result="README.md", success=False))
+    store.begin_intent("open_pr", {"branch": "b"}, cause="test")
+    store.close()
+
+
+def test_the_dump_shows_state_intents_and_each_event(tmp_path: Path) -> None:
+    db = tmp_path / "state.db"
+    store_with_a_run(db)
+    text = render(db)
+    assert (
+        "state: AWAIT_ARCH_SIGNOFF  project: demo  task: demo-1  gate: arch rev-1"
+        in text
+    )
+    assert "pending intent #1 open_pr" in text
+    assert "USER [matrix]: Add a truncate helper" in text
+    assert "PLAN (opus, 500 chars): " in text and "[340 more]" in text
+    assert 'CALL run_command {"command": "ls"}' in text
+    assert "result (FAILED): README.md" in text
+
+
+def test_full_shows_long_texts_whole(tmp_path: Path) -> None:
+    db = tmp_path / "state.db"
+    store_with_a_run(db)
+    assert "x" * 500 in render(db, full=True)
+
+
+def test_the_dump_never_writes_to_the_store(tmp_path: Path) -> None:
+    db = tmp_path / "state.db"
+    store_with_a_run(db)
+    before = db.read_bytes()
+    render(db)
+    assert db.read_bytes() == before
+
+
+def test_a_missing_store_is_reported(tmp_path: Path, capsys) -> None:
+    assert main([str(tmp_path / "nope.db")]) == 1
+    assert "no state store" in capsys.readouterr().err

scripts/bringup.sh

@@ -0,0 +1,103 @@
+#!/usr/bin/env bash
+# Bring-up runs of nisse-dev on this development VM (Development Plan M0.2).
+#
+# The agent runs as root, as it will on its own VM in production (D7): it may
+# install what it needs. It touches only test resources (the sandbox repo, the
+# test Matrix room, a budget-capped LiteLLM key), from the run's config. It
+# runs as a transient systemd unit, so it outlives the shell that started it.
+#
+#   scripts/bringup.sh start [REF]    set the last run aside, export REF (default
+#                                     HEAD) and start the agent
+#   scripts/bringup.sh stop           stop it cleanly (SIGTERM)
+#   scripts/bringup.sh status         the unit, and the agent's state
+#   scripts/bringup.sh logs           follow the agent's log
+#   scripts/bringup.sh collect        save the log and a readable transcript
+#                                     into the current run's directory
+#
+# Layout, under $E2E (default /root/nisse-e2e):
+#   nisse-dev.yaml, .env    the run config and its secrets (made by hand once)
+#   code/                   the exported code under test, with its own .venv
+#   state.db, project/      the agent's state and workspace
+#   run-info, collected/    what was run, and the output of `collect`
+#   runs/<time>/            earlier runs, set aside by `start`
+set -euo pipefail
+
+E2E=${NISSE_E2E:-/root/nisse-e2e}
+SRC=${NISSE_SRC:-/root/nisse}            # repository to export code from
+CODE=$E2E/code
+UNIT=nisse-bringup
+UV=${UV:-$(command -v uv || echo /snap/bin/uv)}
+
+die() { echo "bringup: $*" >&2; exit 1; }
+running() { systemctl is-active --quiet "$UNIT"; }
+
+cmd_start() {
+    local ref=${1:-HEAD} sha
+    [ -f "$E2E/nisse-dev.yaml" ] && [ -f "$E2E/.env" ] || die "no config in $E2E"
+    running && die "already running; stop it first"
+    sha=$(git -C "$SRC" rev-parse --verify "$ref^{commit}") || die "unknown ref $ref"
+
+    # Set the previous run aside, whole: state, workspace, collected output.
+    local items=() item aside
+    for item in "$E2E"/state.db* "$E2E/project" "$E2E/collected" "$E2E/run-info"; do
+        [ -e "$item" ] && items+=("$item")
+    done
+    if [ ${#items[@]} -gt 0 ]; then
+        aside=$E2E/runs/$(date +%Y%m%d-%H%M%S)
+        mkdir -p "$aside"
+        mv "${items[@]}" "$aside/"
+        echo "previous run set aside in $aside"
+    fi
+
+    # Export the code under test, so edits to the working tree can't change a
+    # running agent. Uncommitted changes are not part of a run.
+    rm -rf "$CODE"
+    mkdir -p "$CODE"
+    git -C "$SRC" archive "$sha" | tar -x -C "$CODE"
+    (cd "$CODE" && "$UV" sync --quiet --all-packages)
+    printf 'ref: %s\ncommit: %s\nstarted: %s\n' "$ref" "$sha" "$(date -Is)" > "$E2E/run-info"
+
+    systemctl reset-failed "$UNIT" 2>/dev/null || true
+    systemd-run --quiet --unit="$UNIT" --collect \
+        --property=WorkingDirectory="$CODE" \
+        --property=KillSignal=SIGTERM --property=TimeoutStopSec=120 \
+        --setenv=NISSE_CONFIG="$E2E/nisse-dev.yaml" \
+        --setenv=NISSE_ENV="$E2E/.env" \
+        "$CODE/.venv/bin/nisse-dev"
+    echo "started $UNIT at ${sha:0:8} ($ref)"
+}
+
+cmd_stop() {
+    running || { echo "not running"; return 0; }
+    systemctl stop "$UNIT"   # SIGTERM: the agent shuts down cleanly (main.py)
+    echo "stopped"
+}
+
+cmd_status() {
+    if running; then echo "unit: running"; else echo "unit: not running"; fi
+    [ -f "$E2E/run-info" ] && sed 's/^/  /' "$E2E/run-info"
+    if [ -f "$E2E/state.db" ] && [ -x "$CODE/.venv/bin/python" ]; then
+        "$CODE/.venv/bin/python" -m nisse_dev.dump "$E2E/state.db" | head -n 3
+    fi
+}
+
+cmd_logs() { journalctl -u "$UNIT" -f -o cat; }
+
+cmd_collect() {
+    [ -f "$E2E/run-info" ] || die "no run to collect"
+    local out=$E2E/collected since
+    since=$(sed -n 's/^started: //p' "$E2E/run-info")
+    mkdir -p "$out"
+    journalctl -u "$UNIT" --since "$since" -o cat > "$out/agent.log"
+    "$CODE/.venv/bin/python" -m nisse_dev.dump --full "$E2E/state.db" > "$out/transcript.txt"
+    echo "collected into $out"
+}
+
+case "${1:-}" in
+    start) shift; cmd_start "$@" ;;
+    stop) cmd_stop ;;
+    status) cmd_status ;;
+    logs) cmd_logs ;;
+    collect) cmd_collect ;;
+    *) sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
+esac