New projects from conversation (M2.5, PR B)

- propose_project(name, purpose, decisions) in a conversation goes up for
  a go-ahead (checked: a slug, not an existing project, no trailing
  question).
- On go, the provisioning checklist (D1): a repository with a README,
  protection on its default branch, the agent as a collaborator, a build
  VM, a deploy key, and the project entry as YAML with <...> to fill in.
- On "done", the agent reads its configuration again and adds the new
  project live (build_loop_agent(load=...); only additions), then checks
  what was provided (D4, verified, not assumed): permissions and
  protection as it sees them, then on a throwaway branch with an empty
  commit, a branch push must succeed and a push to the default branch and
  merging its own PR must be refused. The check's PR is closed and its
  branch deleted. Failures are said per item; nothing starts.
- Checked: the tracking issue (the distilled decisions, carrying the new
  item's ID), and the project's item, agreed and labelled `design`, whose
  first milestone is ARCHITECTURE.md and the build plan. A conversation
  waiting on provisioning doesn't close as idle.
- The walk replays exactly from its seed (item IDs too), and event IDs
  stay unique across simulated restarts, as a server's are.
- The test config holds output to 8192 tokens, under the test key's
  per-request cost cap.

Tests: new-project unit tests (checklist and its YAML parse as config, done
before the entry, what isn't set up, a checked project's record and first
PR, an existing name), protection enforced by a hook on the fake forge,
the walk proposing projects and saying done, and an e2e new project in a
repository the test operator provisions from the agent's checklist.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

.gitea/workflows/ci.yml

@@ -54,44 +54,3 @@ jobs:
           # key) and deletes it afterwards.
           NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }}
         run: uv run pytest -m "integration and not e2e" -rs
-
-  e2e:
-    # The end-to-end suite (scripts/e2e.sh): the whole agent against real
-    # services, every role on a local model, so it spends nothing but keeps
-    # the RTX 4090 busy for ~25 minutes. On PRs only, as its own job so the
-    # checks above report first; a newer push to the same PR cancels the
-    # run it supersedes. Locally, run only what you're writing (-k) or
-    # diagnosing (--lf).
-    if: github.event_name != 'push'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    concurrency:
-      group: e2e-${{ github.head_ref || github.ref }}
-      cancel-in-progress: true
-    steps:
-      - uses: actions/checkout@v4
-
-      - name: Install uv
-        run: |
-          curl -LsSf https://astral.sh/uv/install.sh | sh
-          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
-
-      - name: Sync workspace
-        run: uv sync --all-packages
-
-      - name: Install sshd
-        # The build VM is a throwaway sshd here (scripts/e2e.sh --local).
-        run: |
-          SUDO=$(command -v sudo || true)
-          $SUDO apt-get update -qq
-          $SUDO apt-get install -y -qq openssh-server openssh-client > /dev/null
-          $SUDO mkdir -p /run/sshd
-
-      - name: End-to-end tests
-        env:
-          NISSE_LLM_TOKEN: ${{ secrets.NISSE_LLM_TOKEN }}
-          NISSE_MATRIX_TOKEN: ${{ secrets.NISSE_MATRIX_TOKEN }}
-          NISSE_GITEA_TOKEN: ${{ secrets.NISSE_GITEA_TOKEN }}
-          NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }}
-          NISSE_E2E_RUNS: /tmp/nisse-e2e-runs
-        run: scripts/e2e.sh --local -v

docs/Development-Plan.md

@@ -73,11 +73,9 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
   `CLAUDE.md`. A milestone marked **needs discussion** starts with a design
   discussion first.
 - **Every change comes with tests,** through `scripts/check.sh`.
-- **The end-to-end test runs on every PR** (M1.6): the real agent, every role on a
+- **The end-to-end test follows each milestone** (M1.6): the real agent, real models,
-  local model (so it costs nothing per call), a repository of the run's own and a
+  a repository of the run's own and a real build host, driven by a scripted operator. It runs
-  real build host, driven by a scripted operator. CI runs it as its own job on pull
+  after each milestone, not on every push, since it spends model tokens. Manual
-  requests, cancelling a run a newer push supersedes. Locally only the tests being
-  written (`-k`) or diagnosed (`--lf`); `scripts/e2e.sh` reports what a run spent. Manual
   live runs are for the new loop's first runs and phase ends. Findings are filed as
   issues and either fixed before moving on or deliberately deferred.
 - **The Quint models move with the loop:** `orchestrator.qnt` while `control.py`

nisse-dev/src/nisse_dev/git.py

@@ -255,13 +255,6 @@ class GitRepo:
         self._git("commit", "-q", "-m", message)
         return self.head_sha()
 
-    def changed_paths(self, since: str) -> list[str]:
-        """Every path the work changes relative to ``since``, committed or
-        not, new files included; build artifacts aren't work (#80)."""
-        tracked = self._git("diff", "--name-only", since).splitlines()
-        new = self._git("ls-files", "--others", "--exclude-standard").splitlines()
-        return sorted({p for p in [*tracked, *new] if p})
-
     def left_out(self) -> list[str]:
         """Changes not in the last commit: build artifacts the harness left
         out, to be reported rather than silently dropped (D4)."""

nisse-dev/src/nisse_dev/loop/agent.py

@@ -150,11 +150,9 @@ LABELS = {
     "discuss": "This issue is labelled discuss: talk it through with the "
     "operator (ask_human) before restating, and restate only once you agree "
     "on what is wanted.",
-    "design": "This issue is labelled design: the design comes before any "
+    "design": "This issue is labelled design: agree the design with the "
-    "code. Your first milestone is documents only: ARCHITECTURE.md and a "
+    "operator before any code. Your restatement puts the design to them; "
-    "build plan under docs/plans/, in a PR of their own (if you restate, put "
+    "code waits for their go-ahead on it.",
-    "the design to the operator). Code starts in a later milestone, after that "
-    "PR has merged; the harness refuses a design milestone with anything else.",
 }
 
 
@@ -521,9 +519,6 @@ class LoopAgent:
             if pr.merged:
                 self._fact(item, f"PR #{loop.pr} merged")
                 loop, position = replace(loop, pr=0, pr_url="", branch=""), READY
-                if "design" in loop.labels and not loop.design_merged:
-                    self._fact(item, "the design has merged: code may follow")
-                    loop = replace(loop, design_merged=True)
             elif pr.state != "open":
                 self._fact(item, f"PR #{loop.pr} was closed without merging")
                 loop, position = replace(loop, pr=0, pr_url="", branch=""), READY
@@ -1052,8 +1047,7 @@ class LoopAgent:
                 project.repo,
                 title,
                 f"{brief}\n\nFrom a conversation with the operator, agreed there "
-                f"as rev-{loop.rev}. First: ARCHITECTURE.md and the build plan, "
+                f"as rev-{loop.rev}. First: ARCHITECTURE.md and the build plan."
-                "in a PR of documents only; code after it has merged."
                 f"\n\n<!-- nisse-item: {new_id} -->",
             )
             self._gitea.assign_issue(project.repo, number, self._account)
@@ -1217,32 +1211,6 @@ class LoopAgent:
                 )
         self._last_project = here
 
-    def _design_problem(self, item_id: str) -> str | None:
-        """D1, D2: a `design` item agrees its design before any code. Until
-        a PR of it has merged, a milestone holds documents only."""
-        item = self._store.work_item(item_id)
-        if item is None:
-            return None
-        loop = loop_of(item)
-        milestones = self._at(item).milestones
-        if "design" not in loop.labels or loop.design_merged or milestones is None:
-            return None
-        try:
-            changed = milestones.changed()
-        except GitError:
-            return None  # publishing reports what's wrong with the workspace
-        code = [
-            p for p in changed if p != "ARCHITECTURE.md" and not p.startswith("docs/")
-        ]
-        if not code:
-            return None
-        return (
-            "not published: this milestone is the design, so it holds documents "
-            "only (ARCHITECTURE.md, docs/); code starts in a later milestone, "
-            "after this one's PR has merged. Take these out of it (they can "
-            "wait in the plan): " + ", ".join(code)
-        )
-
     def _converse_run(self, item: WorkItem) -> None:
         """A conversation's turn (L1, L5): it talks, and may propose work on
         a project. No workspace, nothing to run."""
@@ -1361,10 +1329,7 @@ class LoopAgent:
                 tools=tools,
                 workspace=workspace,
                 handlers={"start_milestone": lambda args: self._start(item.id, args)},
-                checks={
+                checks={"restate": _restatement_problem},
-                    "restate": _restatement_problem,
-                    "finish_milestone": lambda args: self._design_problem(item.id),
-                },
             )
         finally:
             self._running = False

nisse-dev/src/nisse_dev/loop/items.py

@@ -55,7 +55,6 @@ class Loop:
     proposal_title: str = ""
     proposal_kind: str = ""  # "work" on a project, or a new "project"
     new_project: str = ""  # a new project being provisioned (D1)
-    design_merged: bool = False  # a `design` item's design PR has merged (D1, D2)
     # Where the operator last spoke (D5: answer where asked): "" for the
     # Matrix thread, or the PR it came from, with the inline location and a
     # little of what they said, for quoting.

nisse-dev/src/nisse_dev/loop/milestones.py

@@ -161,10 +161,6 @@ class Milestones:
         if git.remote_sha(self._remote, parked, credential=self._credential):
             git.delete_remote_branch(self._remote, parked, credential=self._credential)
 
-    def changed(self) -> list[str]:
-        """What the current milestone changes, from the default branch."""
-        return self._git.changed_paths(f"origin/{self._base}")
-
     # -- a new project's checks (D1, D4) -----------------------------------------
 
     def verify_access(self, account: str) -> list[str]:

nisse-dev/tests/test_loop_multi.py

@@ -19,7 +19,6 @@ from test_loop_milestones import (
 from test_loop_milestones import world as world  # noqa: PLC0414 -- the fixture
 
 from nisse_dev.canonical import ToolCall, ToolResult
-from nisse_dev.gitea import Issue
 from nisse_dev.loop.items import MERGE, WAITING, loop_of
 
 
@@ -175,44 +174,3 @@ def test_an_unsaid_final_closes_the_issue(world: dict[str, Any]) -> None:
     say(transport, world, "go")
     [pr] = forge.prs
     assert "Closes #14" in forge.prs[pr]["body"]
-
-
-def test_a_design_milestone_holds_documents_only(world: dict[str, Any]) -> None:
-    """D1, D2: a `design` item agrees its design before any code; the
-    harness refuses a design milestone with code in it."""
-    forge: Forge = world["forge"]
-    forge.issues[0] = Issue(14, "Add count_words", "count words", "don", ("design",))
-    model = ScriptedModel(
-        RESTATE,
-        [
-            call("start_milestone", name="design"),
-            call("write_file", path="ARCHITECTURE.md", content="# arch\n"),
-            call("write_file", path="docs/plans/words.md", content="# plan\n"),
-            call("write_file", path="words.py", content="W = 1\n"),  # too soon
-            call("finish_milestone", summary="design", final="no"),
-        ],
-        [
-            call("run_command", command="rm words.py"),
-            call("finish_milestone", summary="design", final="no"),
-        ],
-        milestone("words.py", final="yes"),  # after the design merged
-    )
-    agent, transport = loop_agent(world, model)
-    agent.intake()
-    say(transport, world, "go")
-    [design] = forge.prs
-    assert sorted(forge.files(design)) == ["ARCHITECTURE.md", "docs/plans/words.md"]
-    refused = [
-        e.event.result
-        for e in world["store"].transcript(item(world).id, "driver").read()
-        if isinstance(e.event, ToolResult)
-        and "this milestone is the design" in e.event.result
-    ]
-    assert refused and "words.py" in refused[0]
-
-    forge.merge(design)
-    agent.watch()
-    agent._schedule()
-    code = max(forge.prs)
-    assert code != design and forge.files(code) == ["words.py"]
-    assert loop_of(item(world)).design_merged

nisse-dev/tests/test_loop_properties.py

@@ -63,11 +63,6 @@ class PolicyModel:
         self.n += 1
         offered = {t["function"]["name"] for t in tools}
         facts = [getattr(e.event, "text", "") for e in events]
-        # A `design` item writes documents until its design has merged.
-        design = "labelled design" in (system or "") and not any(
-            "the design has merged" in t for t in facts
-        )
-        path = f"docs/d{self.n}.md" if design else f"f{self.n}.py"
         roll = self.rng.random()
         if roll < 0.08:
             return [call("ask_human", question=f"question {self.n}?")]
@@ -103,7 +98,7 @@ class PolicyModel:
             # Half done, then a question: work a switch must park.
             return [
                 call("start_milestone", name=f"m{self.n}"),
-                call("write_file", path=path, content=f"x = {self.n}\n"),
+                call("write_file", path=f"f{self.n}.py", content=f"x = {self.n}\n"),
                 call("ask_human", question=f"halfway {self.n}?"),
             ]
         merged = any("merged" in t for t in facts[-6:])
@@ -111,7 +106,7 @@ class PolicyModel:
             return [call("close_item", summary=f"done {self.n}")]
         return [
             call("start_milestone", name=f"m{self.n}"),
-            call("write_file", path=path, content=f"x = {self.n}\n"),
+            call("write_file", path=f"f{self.n}.py", content=f"x = {self.n}\n"),
             call(
                 "finish_milestone",
                 summary=f"work {self.n}",

scripts/e2e.sh

@@ -1,16 +1,13 @@
 #!/usr/bin/env bash
 # The end-to-end test (Development Plan M1.6): the whole agent against real
-# services, driven by a scripted operator. CI runs it on every PR (with
+# services, driven by a scripted operator. Run after each milestone; it
-# --local); locally, run only the tests you're writing (-k) or diagnosing
+# spends model tokens and takes minutes, so it is not part of check.sh or CI.
-# (--lf).
 #
 #   scripts/e2e.sh            on the bring-up build VM (configured below)
 #   scripts/e2e.sh --local    on a throwaway local sshd instead
 #
 # Uses tests/integration/nisse-test.yaml and its secrets (.env at the repo
-# root), like the integration tests. Every role runs on a local model
+# root), like the integration tests.
-# (tests/integration/e2e_support.py; NISSE_E2E_MODEL overrides it), so tell
-# the operator before a full run: it keeps the RTX 4090 busy for a while.
 set -euo pipefail
 cd "$(dirname "$0")/.."
 
@@ -22,27 +19,4 @@ else
 fi
 # Each run's state store is kept here, for nisse_dev.dump after a failure.
 export NISSE_E2E_RUNS=${NISSE_E2E_RUNS:-/root/nisse-e2e/test-runs}
-
+exec uv run pytest -m e2e -rs tests/integration/test_end_to_end.py tests/integration/test_end_to_end_loop.py "$@"
-# What the run spent on the test key, read from LiteLLM (spend only). The
-# end-to-end tests run on a local model, so anything above $0 means a call
-# went to a paid one: an override, or a proxy fallback.
-spend() {
-    (set -a; [ -f .env ] && . ./.env; set +a
-     uv run python -c '
-import os, httpx
-r = httpx.get("https://litellm.dongar.net/key/info", timeout=30,
-              headers={"Authorization": "Bearer " + os.environ["NISSE_LLM_TOKEN"]})
-i = r.json()["info"]
-print(i["spend"], i.get("max_budget"))' 2>/dev/null) || echo "? ?"
-}
-read -r before budget < <(spend)
-status=0
-uv run pytest -m e2e -rs tests/integration/test_end_to_end.py tests/integration/test_end_to_end_loop.py "$@" || status=$?
-read -r after _ < <(spend)
-python3 -c "
-b, a, cap = '$before', '$after', '$budget'
-try:
-    print(f'e2e spend: \${float(a) - float(b):.4f} this run; \${float(a):.2f} of \${float(cap):.2f} today')
-except ValueError:
-    print('e2e spend: unknown (could not read the key)')"
-exit "$status"

tests/integration/e2e_support.py

@@ -11,17 +11,6 @@ from dataclasses import dataclass
 from pathlib import Path
 
 from nisse_dev.gitea import GiteaClient, GiteaError
-from nisse_dev.llm import RoleMap
-
-# The end-to-end tests run every role on a local model, which costs nothing
-# per call; NISSE_E2E_MODEL overrides it (e.g. claude-haiku-4-5 for a quick,
-# paid run).
-E2E_MODEL = "qwen3.8-27b"
-
-
-def e2e_roles() -> RoleMap:
-    model = os.environ.get("NISSE_E2E_MODEL", "").strip() or E2E_MODEL
-    return RoleMap(executor=model, planner=model, helper=model, expert=model)
 
 
 def task(run: str) -> str:

tests/integration/test_end_to_end.py

@@ -25,7 +25,7 @@ from pathlib import Path
 from typing import Any
 
 import pytest
-from e2e_support import e2e_roles, fresh_repo, operator_gitea
+from e2e_support import fresh_repo, operator_gitea
 from e2e_support import task as make_task
 
 from nisse_dev.agent import Agent, build_agent
@@ -84,7 +84,6 @@ def agent(
             workspace=Path(f"/tmp/nisse-e2e-{run}/sandbox"),
             build_host=build_host(ssh_host),
         ),
-        roles=e2e_roles(),
         state_db=tmp_path / "state.db",
     )
     transport = InMemoryTransport()

tests/integration/test_end_to_end_loop.py

@@ -22,7 +22,7 @@ from pathlib import Path
 from typing import Any
 
 import pytest
-from e2e_support import dependent_task, e2e_roles, fresh_repo, operator_gitea, task
+from e2e_support import dependent_task, fresh_repo, operator_gitea, task
 
 from nisse_dev.config import AgentConfig, LoopTuning
 from nisse_dev.execution import SSHHost
@@ -75,7 +75,6 @@ def running(
             build_host=build_host(ssh_host),
         ),
         loop="new",
-        roles=e2e_roles(),
         state_db=keep / "state.db",
         # Short, so a bug's grace period runs out within the test.
         tuning=replace(live_config.tuning, loop=LoopTuning(bug_grace_s=BUG_GRACE_S)),