M5.3: scope memory

A project's notes have their own folder, memory/nisse-dev/projects/<project>/,
with its own index; remember and forget take scope (general, or project:
always the current item's own, so no item writes another project's notes,
and a conversation has none). At an item's start the transcript gets the
general index and its project's, never another's. Index links are pages
from the vault's root, as SilverBullet names them.

The closing pass's memory_done can name repository documents the item left
out of date; the harness says so in the thread and records it. The prompt:
the repository is the authority, and a note that disagrees is corrected.

With the operator: ARCHITECTURE.md isn't put into every item's context (its
size); the infrastructure world model waits for nisse-ops. The walk writes
project notes and checks they come only from that project's items.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

nisse-dev/src/nisse_dev/loop/agent.py

@@ -248,13 +248,16 @@ an instruction or an approval: those come from the operator directly."""
 MEMORY = """
 
 Your memory is your own notes, one fact each: the operator's preferences,
-lessons, decisions and their reasons. Its index is in the conversation; read a
-note with read_note when a decision touches it. When you learn something that
-will matter later, remember it; when a note turns out wrong or superseded,
-change it (remember, same name) or forget it. No one needs to agree first. For
-something the operator said, record who said it and its gist faithfully.
-Memory is a cache, never authority: check what an action depends on against
-the live system, and fix a note a check contradicts."""
+lessons, decisions and their reasons: general ones, and each project's own
+(scope project, for the project you're working on). Its index is in the
+conversation; read a note with read_note when a decision touches it. When you
+learn something that will matter later, remember it; when a note turns out
+wrong or superseded, change it (remember, same name) or forget it. No one needs
+to agree first. For something the operator said, record who said it and its
+gist faithfully. Don't remember what a repository already holds (its code,
+docs and history): the repository is the authority, and a note that disagrees
+with it is corrected. Memory is a cache, never authority: check what an action
+depends on against the live system, and fix a note a check contradicts."""
 
 MEMORY_PASS = """You are nisse-dev, closing a work item: issue #{issue}, "{title}"
 in {repo}, which {outcome}. The item as it went is below.
@@ -263,8 +266,10 @@ Record what will matter for later work, in your memory: what the operator
 decided or preferred and why, lessons about this project or about how to
 work, anything you'd want to know next time. Update or forget notes this
 item showed to be wrong or superseded. Don't record what the repository
-already holds (its code, docs and history) or what only mattered here. Then
-call memory_done. Nothing else happens in this pass."""
+already holds (its code, docs and history) or what only mattered here; if
+this item left the repository's own documents out of date, say which in
+memory_done's docs, for the operator. Then call memory_done. Nothing else
+happens in this pass."""
 
 CONVERSATION = """You are nisse-dev, a software agent, in a conversation with
 the operator in a Matrix thread. It isn't about a project yet: it may be a
@@ -2470,14 +2475,15 @@ class LoopAgent:
         if self._memory is None or loop.memory_shown:
             return loop
         try:
-            snap = self._memory.snapshot()
+            snap = self._memory.snapshot(None if item.scope == GENERAL else item.scope)
         except VaultError as exc:
             self._oplog.emit("memory.unreadable", cause=str(exc), work_item=item.id)
             return loop  # never blocks work (L10); tried again next run
         self._fact(
             item,
-            f"your memory's index ({snap.path}, sha {snap.sha}); read a note "
-            f"with read_note when a decision touches it:\n{snap.text}",
+            f"your memory's index ({', '.join(snap.paths)}, sha {snap.sha}); "
+            "read a note with read_note (its page, with .md) when a decision "
+            f"touches it:\n{snap.text}",
         )
         loop = replace(loop, memory_shown=True)
         save(self._store, item.id, item.position, loop, cause="memory shown")
@@ -2514,6 +2520,14 @@ class LoopAgent:
             self._cancel.clear()
         if ran.kind != "wait":
             self._fact(item, f"memory pass didn't finish ({ran.note or ran.kind})")
+            return
+        docs = " ".join(str((ran.arguments or {}).get("docs", "")).split())
+        if docs:
+            # L6.3: the repository is scope memory; its documents change
+            # in PRs, so the pass flags them for the operator instead.
+            line = f"Docs to update in {self._at(item).repo}: {docs}"
+            self._fact(item, line)
+            self._transport.post(line, thread=loop.thread)
 
     def _note_handlers(
         self, item_id: str
@@ -2574,9 +2588,22 @@ class LoopAgent:
             if item is not None:
                 self._transport.post(line, thread=loop_of(item).thread)
 
+        def scoped(args: dict[str, Any]) -> str | None:
+            """L6.3: project means this item's own project, never another."""
+            scope = str(args.get("scope", "") or "general").strip().lower()
+            if scope == "general":
+                return None
+            item = self._store.work_item(item_id)
+            if scope != "project":
+                raise VaultError(f"scope is general or project, not {scope!r}")
+            if item is None or item.scope in (None, GENERAL):
+                raise VaultError("a conversation has no project: use general")
+            return item.scope
+
         def remember(args: dict[str, Any]) -> tuple[str, bool]:
             name = str(args.get("name", "")).strip()
             about = str(args.get("description", ""))
+            project = scoped(args)
             item = self._store.work_item(item_id)
             replaced = memory.remember(
                 name,
@@ -2585,7 +2612,9 @@ class LoopAgent:
                 provenance={
                     "source": f"work item {item_id}",
                     "thread": loop_of(item).thread if item else "",
+                    **({"project": project} if project else {}),
                 },
+                project=project,
             )
             said(
                 f"{'Updated' if replaced else 'Remembered'}: {' '.join(about.split())}"
@@ -2595,7 +2624,7 @@ class LoopAgent:
         def forget(args: dict[str, Any]) -> tuple[str, bool]:
             name = str(args.get("name", "")).strip()
             why = " ".join(str(args.get("why", "")).split())
-            memory.forget(name)
+            memory.forget(name, scoped(args))
             said(f"Forgot: {name}" + (f" ({why})" if why else ""))
             return f"forgot {name}", True
 

nisse-dev/src/nisse_dev/loop/tools.py

@@ -117,18 +117,23 @@ REMEMBER = _tool(
     description="One line saying what the note is, for your memory's index.",
     fact="The fact and why it matters; for something the operator said, who "
     "said it and its gist.",
+    scope="general, or project for a note about this work item's project "
+    "(kept with that project's notes).",
 )
 FORGET = _tool(
     "forget",
     "Forget a note of yours that is wrong or superseded.",
     name="The note's name.",
     why="Why it's no longer true or needed.",
+    scope="general or project, as it was remembered.",
 )
 MEMORY_TOOLS = [REMEMBER, FORGET]
 MEMORY_DONE = _tool(
     "memory_done",
     "You've recorded what will matter later: end the pass.",
     summary="What you remembered or forgot, in a line.",
+    docs="Documents in the repository this item left out of date (like "
+    "ARCHITECTURE.md), and how; empty if none.",
 )
 
 # A conversation's tools (L1, L5): it talks, and may propose work.

nisse-dev/src/nisse_dev/memory.py

@@ -22,79 +22,132 @@ _NAME = re.compile(r"[a-z0-9][a-z0-9-]{0,63}")
 
 @dataclass(frozen=True)
 class Snapshot:
-    """The index as read, for the transcript (L6: with its provenance)."""
+    """The indexes as read, for the transcript (L6: with their provenance)."""
 
-    path: str
+    paths: tuple[str, ...]
     text: str
     sha: str
 
 
 class Memory:
-    """``folder``: the agent's read-write folder in the vault."""
+    """``folder``: the agent's read-write folder in the vault. General
+    notes live there; a project's under ``projects/<project>/`` (L6.3), each
+    folder with an index the harness keeps."""
 
     def __init__(self, vault: Vault, folder: str) -> None:
         self._vault = vault
         self.folder = folder.strip("/")
 
-    def _path(self, name: str) -> str:
+    def _where(self, project: str | None) -> str:
+        if project is None:
+            return self.folder
+        if not _NAME.fullmatch(project):
+            raise VaultError(f"{project!r}: not a project's name")
+        return f"{self.folder}/projects/{project}"
+
+    def _path(self, name: str, project: str | None) -> str:
         if not _NAME.fullmatch(name) or name == INDEX:
             raise VaultError(
                 f"{name!r}: a note's name is lowercase letters, digits and -, "
                 f"up to 64, and not {INDEX}"
             )
-        return f"{self.folder}/{name}.md"
+        return f"{self._where(project)}/{name}.md"
 
     def remember(
-        self, name: str, description: str, fact: str, provenance: Mapping[str, str]
+        self,
+        name: str,
+        description: str,
+        fact: str,
+        provenance: Mapping[str, str],
+        project: str | None = None,
     ) -> bool:
-        """Write the note, or replace it; True if it existed."""
+        """Write the note, or replace it; True if it existed. ``project``:
+        a note about that project, in its folder; None: a general one."""
         description = " ".join(description.split())
         if not description or not fact.strip():
             raise VaultError("a note needs a one-line description and the fact")
-        path = self._path(name)
+        path = self._path(name, project)
         existed = path in self._vault.notes(self.folder)
         front = yaml.safe_dump({"description": description}, allow_unicode=True)
         text = f"---\n{front}---\n{fact.strip()}\n"
         self._vault.write(path, text, provenance=provenance)
-        self._rebuild()
+        self._rebuild(project)
         return existed
 
-    def forget(self, name: str) -> None:
-        self._vault.delete(self._path(name))
-        self._rebuild()
+    def forget(self, name: str, project: str | None = None) -> None:
+        self._vault.delete(self._path(name, project))
+        self._rebuild(project)
 
-    def _entries(self) -> list[tuple[str, str]]:
+    def _entries(self, where: str) -> list[tuple[str, str]]:
+        """The notes directly in ``where``: their pages and descriptions."""
         entries = []
         for path in self._vault.notes(self.folder):
-            name = path.rsplit("/", 1)[-1].removesuffix(".md")
-            if name == INDEX or "/" in path.removeprefix(f"{self.folder}/"):
+            page = path.removesuffix(".md")
+            if page.rsplit("/", 1)[0] != where or page.endswith(f"/{INDEX}"):
                 continue
             try:
                 front, _ = split_frontmatter(self._vault.read(path))
             except VaultError:
                 front = {}
-            entries.append((name, str(front.get("description", "")).strip()))
+            entries.append((page, str(front.get("description", "")).strip()))
         return entries
 
-    def _rebuild(self) -> None:
-        lines = [f"- [[{name}]] — {about}" for name, about in self._entries()]
-        body = "# Memory\n\nOne note per fact; kept by the harness.\n\n" + (
+    def _projects(self) -> list[str]:
+        prefix = f"{self.folder}/projects/"
+        return sorted(
+            {
+                p.removeprefix(prefix).split("/", 1)[0]
+                for p in self._vault.notes(self.folder)
+                if p.startswith(prefix) and not p.endswith(f"/{INDEX}.md")
+            }
+        )
+
+    def _rebuild(self, project: str | None = None) -> None:
+        """The changed folder's index, and the top one, which links each
+        project's. Links are pages from the vault's root, as SilverBullet
+        names them."""
+        if project is not None:
+            self._write_index(
+                self._where(project), f"# Memory: {project}", self._lines(project)
+            )
+        lines = self._lines(None)
+        projects = self._projects()
+        if projects:
+            lines += ["", "Projects:"] + [
+                f"- [[{self._where(p)}/{INDEX}]] — notes about {p}" for p in projects
+            ]
+        self._write_index(self.folder, "# Memory", lines)
+
+    def _lines(self, project: str | None) -> list[str]:
+        return [
+            f"- [[{page}]] — {about}"
+            for page, about in self._entries(self._where(project))
+        ]
+
+    def _write_index(self, where: str, title: str, lines: list[str]) -> None:
+        body = f"{title}\n\nOne note per fact; kept by the harness.\n\n" + (
             "\n".join(lines) or "Nothing yet."
         )
         self._vault.write(
-            f"{self.folder}/{INDEX}.md",
+            f"{where}/{INDEX}.md",
             body + "\n",
             provenance={"source": "the harness, after a change"},
         )
 
-    def snapshot(self) -> Snapshot:
-        """The index for the transcript; rebuilt first if it's missing."""
-        path = f"{self.folder}/{INDEX}.md"
-        try:
-            text = self._vault.read(path)
-        except VaultError:
-            self._rebuild()
-            text = self._vault.read(path)
-        _, body = split_frontmatter(text)
-        sha = hashlib.sha256(body.encode()).hexdigest()[:12]
-        return Snapshot(path, body.strip(), sha)
+    def snapshot(self, project: str | None = None) -> Snapshot:
+        """The general index, and ``project``'s when it has notes, for the
+        transcript; the general one is made first if it's missing."""
+        paths = [f"{self.folder}/{INDEX}.md"]
+        if project is not None and project in self._projects():
+            paths.append(f"{self._where(project)}/{INDEX}.md")
+        bodies = []
+        for path in paths:
+            try:
+                text = self._vault.read(path)
+            except VaultError:
+                self._rebuild(None if path == paths[0] else project)
+                text = self._vault.read(path)
+            bodies.append(split_frontmatter(text)[1].strip())
+        joined = "\n\n".join(bodies)
+        sha = hashlib.sha256(joined.encode()).hexdigest()[:12]
+        return Snapshot(tuple(paths), joined, sha)

nisse-dev/tests/test_loop_memory.py

@@ -72,7 +72,7 @@ def test_the_model_remembers_and_forgets_without_asking(
     assert "Remembered: Count words with split" in said(transport)
     assert "Forgot: old-style (the operator changed it)" in said(transport)
     index = memory.snapshot().text
-    assert "[[uses-split]]" in index and "old-style" not in index
+    assert "[[memory/nisse-dev/uses-split]]" in index and "old-style" not in index
 
 
 def test_the_index_is_shown_once_per_item(
@@ -86,7 +86,7 @@ def test_the_index_is_shown_once_per_item(
     shown = [f for f in facts(world) if f.startswith("your memory's index")]
     assert len(shown) == 1
     assert f"sha {memory.snapshot().sha}" in shown[0]
-    assert "[[uses-split]] — Count words with split" in shown[0]
+    assert "[[memory/nisse-dev/uses-split]] — Count words with split" in shown[0]
 
 
 def test_a_pass_at_the_close_records_what_will_matter(
@@ -109,7 +109,7 @@ def test_a_pass_at_the_close_records_what_will_matter(
     say(transport, world, "abandon")
     assert the_item(world).status == "closed"
     assert "memory pass: recording what this item decided and learned" in facts(world)
-    assert "[[no-count-words]]" in memory.snapshot().text
+    assert "[[memory/nisse-dev/no-count-words]]" in memory.snapshot().text
     offered = model.offered[-1]
     assert offered == {
         "read_note",
@@ -156,5 +156,80 @@ def test_a_conversation_can_remember(world: dict[str, Any], memory: Memory) -> N
     )
     _, transport = agent(world, model, memory)
     transport.receive(DON, "I have a 4090 for local models.")
-    assert "[[has-4090]]" in memory.snapshot().text
+    assert "[[memory/nisse-dev/has-4090]]" in memory.snapshot().text
     assert "Remembered: Has a 4090" in said(transport)
+
+
+def test_a_project_note_goes_with_the_items_project(
+    world: dict[str, Any], memory: Memory
+) -> None:
+    memory.remember("other-fact", "Other's", "o", provenance={}, project="other")
+    memory.remember("old", "Sandbox's", "s", provenance={}, project="sandbox")
+    model = Spy(
+        [
+            call(
+                "remember",
+                name="deploys-by-tag",
+                description="Sandbox deploys by tag",
+                fact="f",
+                scope="project",
+            ),
+            call("remember", name="x", description="d", fact="f", scope="team"),
+        ],
+        RESTATE,
+    )
+    loop, _ = agent(world, model, memory)
+    loop.intake()
+    note = memory._vault.read(f"{FOLDER}/projects/sandbox/deploys-by-tag.md")
+    assert "project: sandbox" in note
+    [shown] = [f for f in facts(world) if f.startswith("your memory's index")]
+    assert "projects/sandbox/old]]" in shown  # this project's notes
+    assert "other-fact" not in shown  # never another's
+    results = [
+        e.event.result
+        for e in world["store"].transcript(the_item(world).id, "driver").read()
+        if hasattr(e.event, "result")
+    ]
+    assert "scope is general or project, not 'team'" in results
+
+
+def test_a_conversation_has_no_project_scope(
+    world: dict[str, Any], memory: Memory
+) -> None:
+    world["forge"].issues.clear()
+    model = Spy(
+        [call("remember", name="x", description="d", fact="f", scope="project")],
+        [call("reply", text="Noted.")],
+    )
+    _, transport = agent(world, model, memory)
+    transport.receive(DON, "remember this")
+    assert not (memory._vault.root / FOLDER / "projects").exists()
+    log = world["store"].all_events()
+    assert any(
+        getattr(e.event, "result", "") == "a conversation has no project: use general"
+        for e in log
+    )
+
+
+def test_the_pass_flags_documents_left_out_of_date(
+    world: dict[str, Any], memory: Memory
+) -> None:
+    model = Spy(
+        RESTATE,
+        [
+            call(
+                "memory_done",
+                summary="nothing new",
+                docs="ARCHITECTURE.md still describes the old text module",
+            )
+        ],
+    )
+    loop, transport = agent(world, model, memory)
+    loop.intake()
+    say(transport, world, "abandon")
+    line = (
+        "Docs to update in don/sandbox: ARCHITECTURE.md still describes the old "
+        "text module"
+    )
+    assert line in said(transport)
+    assert line in facts(world)

nisse-dev/tests/test_loop_properties.py

@@ -57,7 +57,7 @@ from nisse_dev.memory import Memory
 from nisse_dev.oplog import OpLog
 from nisse_dev.tools import Workspace
 from nisse_dev.transport import InboundMessage, InMemoryTransport
-from nisse_dev.vault import Vault
+from nisse_dev.vault import Vault, split_frontmatter
 from nisse_dev.worker import InlineWorker
 
 # How many walks, and from which seed: all 48 by default; CI runs fewer on a
@@ -87,6 +87,16 @@ class PolicyModel:
         self.projects = projects
         self.n = 0
 
+    def _remember(self) -> Any:
+        scope = _scope(self.rng)
+        return call(
+            "remember", name=f"n{self.n % 4}", description="d", fact="f", scope=scope
+        )
+
+    def _forget(self) -> Any:
+        scope = _scope(self.rng)
+        return call("forget", name=f"n{self.n % 4}", why="superseded", scope=scope)
+
     def complete(self, events, tools, model, system=None) -> list[Any]:
         assert_renderable(events)  # as the real API would
         self.n += 1
@@ -101,16 +111,14 @@ class PolicyModel:
             # The memory pass at a close (M5.2): a note, a forgetting, done.
             roll = self.rng.random()
             if roll < 0.3:
-                return [
-                    call("remember", name=f"n{self.n % 4}", description="d", fact="f")
-                ]
+                return [self._remember()]
             if roll < 0.55:
-                return [call("forget", name=f"n{self.n % 4}", why="superseded")]
+                return [self._forget()]
             return [call("memory_done", summary="done")]
         if "remember" in offered and self.rng.random() < 0.06:
             if self.rng.random() < 0.4:
-                return [call("forget", name=f"n{self.n % 4}", why="superseded")]
-            return [call("remember", name=f"n{self.n % 4}", description="d", fact="f")]
+                return [self._forget()]
+            return [self._remember()]
         if "toolchain_ready" in offered:
             # A preflight (M4.1): usually reported, sometimes badly or not.
             if self.rng.random() < 0.15:
@@ -213,6 +221,10 @@ class PolicyModel:
         ]
 
 
+def _scope(rng: random.Random) -> str:
+    return rng.choice(["general", "project"])
+
+
 class Watched:
     """The driver's model, watched: a run on a project item (not its
     preflight) must find the project's marker on the VM, and no model runs
@@ -715,6 +727,18 @@ class Walk:
             and p != root / "notes" / "Ops.md"
         ]
         assert not strays, f"memory_stays_home: {strays}: {where}"
+        # M5.3: a project's notes come only from that project's items.
+        projects = root / "memory" / "nisse-dev" / "projects"
+        for note in projects.rglob("*.md") if projects.exists() else []:
+            if note.name == "index.md":
+                continue
+            project = note.relative_to(projects).parts[0]
+            front, _ = split_frontmatter(note.read_text())
+            source = front["provenance"]["source"]
+            assert front["provenance"]["project"] == project, where
+            assert source.startswith(f"work item {project}-"), (
+                f"project_notes_stay_with_their_project: {note}: {source}: {where}"
+            )
         assert (root / "notes" / "Ops.md").read_text() == "the operator's\n", where
         # #137: no model works on a project whose build VM is out of reach.
         assert not self.while_down, f"no_model_on_a_lost_vm: {self.while_down}: {where}"

nisse-dev/tests/test_memory.py

@@ -40,7 +40,9 @@ def test_the_same_name_replaces_and_the_index_follows(memory: Memory) -> None:
     memory.remember("a-fact", "Ay", "a", provenance={})
     assert memory.remember("a-fact", "Ay, again", "a2", provenance={})
     index = memory.snapshot().text
-    assert index.endswith("- [[a-fact]] — Ay, again\n- [[b-fact]] — Bee")
+    assert index.endswith(
+        "- [[memory/nisse-dev/a-fact]] — Ay, again\n- [[memory/nisse-dev/b-fact]] — Bee"
+    )
     memory.forget("a-fact")
     assert "a-fact" not in memory.snapshot().text
     with pytest.raises(VaultError, match="no such note"):
@@ -60,8 +62,34 @@ def test_a_note_needs_a_description_and_a_fact(memory: Memory) -> None:
 
 def test_the_snapshot_is_hashed_and_made_if_missing(memory: Memory) -> None:
     first = memory.snapshot()
-    assert first.path == f"{FOLDER}/index.md"
+    assert first.paths == (f"{FOLDER}/index.md",)
     assert "Nothing yet." in first.text
     assert memory.snapshot().sha == first.sha  # same index, same hash
     memory.remember("x", "Ex", "f", provenance={})
     assert memory.snapshot().sha != first.sha
+
+
+def test_a_projects_notes_have_their_own_folder_and_index(memory: Memory) -> None:
+    memory.remember("general-one", "General", "g", provenance={})
+    memory.remember("deploy", "Deploys by tag", "d", provenance={}, project="sandbox")
+    memory.remember("other", "Other's", "o", provenance={}, project="other")
+    root = memory._vault.root / FOLDER
+    assert (root / "projects" / "sandbox" / "deploy.md").exists()
+    top = memory.snapshot().text
+    assert "- [[memory/nisse-dev/general-one]] — General" in top
+    assert "deploy" not in top.split("Projects:")[0]
+    assert "- [[memory/nisse-dev/projects/sandbox/index]] — notes about sandbox" in top
+    seen = memory.snapshot("sandbox")
+    assert seen.paths == (
+        f"{FOLDER}/index.md",
+        f"{FOLDER}/projects/sandbox/index.md",
+    )
+    assert (
+        "- [[memory/nisse-dev/projects/sandbox/deploy]] — Deploys by tag" in seen.text
+    )
+    assert "projects/other/other]]" not in seen.text
+    memory.forget("deploy", "sandbox")
+    assert memory.snapshot("sandbox").paths == (f"{FOLDER}/index.md",)
+    assert "sandbox/index" not in memory.snapshot().text
+    with pytest.raises(VaultError, match="not a project's name"):
+        memory.remember("x", "d", "f", provenance={}, project="../x")