From e5c18b3bf4f22c07e846bd048ffe2801e139786a Mon Sep 17 00:00:00 2001 From: Don Garrett Date: Mon, 28 Sep 2026 01:25:38 +0000 Subject: [PATCH 1/5] Spec: record sharing the local inference host as an open decision nisse-dev and nisse-ops both send local roles to the one 24GB host, where llama-swap keeps one model resident. Different models per agent would reload weights on every switch. Per-key rate limits don't help. Lists the options, simplest first, to decide after the executor benchmark and the ops evaluation. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/Architecture-Build-Spec.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/Architecture-Build-Spec.md b/docs/Architecture-Build-Spec.md index b795a53..50a2984 100644 --- a/docs/Architecture-Build-Spec.md +++ b/docs/Architecture-Build-Spec.md @@ -1414,6 +1414,24 @@ The rest resolve when reached, not now. and ticket-driven ops work already require one. Whether every gated work item needs one (for instance, an ops question that turns into a change to make) is best settled once C2's step sequences are in use. +- **Open: sharing the local inference host between agents.** Each agent keeps + one active work item (0.2), but `nisse-dev` and `nisse-ops` are separate agents, + and both send their local roles through LiteLLM to the one 24GB host. llama-swap + there keeps one model resident at a time. If the two agents' local roles map to + different models, interleaved requests reload weights on every switch (10–60 s + each), and each agent waits on the other's loads. LiteLLM's per-key rate limits + don't address this: they cap one key's throughput rather than share the host, + and know nothing about which model is loaded. (Measured 2026-09-28: they only + throttled the benchmark, and the token limiter leaked until LiteLLM restarted.) + Options, simplest first: + - *One local model for every local role.* No swaps at all. It needs one model + good enough both as the dev executor and for ops' Home Assistant tool calls. + - *Two models co-resident*, if VRAM allows (llama-swap groups): for instance a + small tool-caller for ops beside the executor model. It costs context window. + - *A scheduler in front of the host* that batches each model's requests to keep + swaps rare. The most work; only if neither of the above fits. + Decide once the executor benchmark and the ops evaluation show whether one model + can serve both roles. --- From 50a2a75342f5cedd83dc31669a5f2bc78c564398 Mon Sep 17 00:00:00 2001 From: Don Garrett Date: Mon, 28 Sep 2026 01:29:29 +0000 Subject: [PATCH 2/5] Spec: prefer one local model; add vision as an open decision The host-sharing item now records the operator's preference: one local model for every local role, with agents tagging in and out of the host as the fallback if no single model fits. A new item records vision as a requirement (OpenSCAD renders, later video files) and what the text-only agent needs before any vision work. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/Architecture-Build-Spec.md | 37 +++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/docs/Architecture-Build-Spec.md b/docs/Architecture-Build-Spec.md index 50a2984..3adf9f4 100644 --- a/docs/Architecture-Build-Spec.md +++ b/docs/Architecture-Build-Spec.md @@ -1423,15 +1423,34 @@ The rest resolve when reached, not now. don't address this: they cap one key's throughput rather than share the host, and know nothing about which model is loaded. (Measured 2026-09-28: they only throttled the benchmark, and the token limiter leaked until LiteLLM restarted.) - Options, simplest first: - - *One local model for every local role.* No swaps at all. It needs one model - good enough both as the dev executor and for ops' Home Assistant tool calls. - - *Two models co-resident*, if VRAM allows (llama-swap groups): for instance a - small tool-caller for ops beside the executor model. It costs context window. - - *A scheduler in front of the host* that batches each model's requests to keep - swaps rare. The most work; only if neither of the above fits. - Decide once the executor benchmark and the ops evaluation show whether one model - can serve both roles. + - **Preferred: one local model for every local role.** No swaps at all. It + needs one model good enough as the dev executor, for ops' Home Assistant tool + calls, and for vision (next item). The executor benchmark and the vision and + ops evaluations decide whether one exists. + - **Fallback: agents tag in and out.** If no single model fits, the agents + coordinate their use of the host rather than interleave: an agent holds the + host for a stretch of work on its model, then hands it over, so a swap happens + per hand-over rather than per request. Two models co-resident (llama-swap + groups), where VRAM allows, can reduce how often that's needed. How the + hand-over works (a lease, a queue, or priority for ops alerts) is only worth + designing if the preferred option fails. +- **Open: vision (A10, A15).** Needed for `nisse-dev`'s OpenSCAD work, where the + model must look at a rendered part to check it against the design, and for + video-file management planned later (which agent owns it is not yet + decided). The local candidates on the + 4090 host accept images, and video as sampled frames, but the agent is + text-only: canonical events carry strings, and neither renderer can send an + image. Needed before any vision work: + - a way for a tool to return an image, e.g. an OpenSCAD render or a frame from + a video file; + - image content in the canonical event model, rendered as OpenAI image parts + and Anthropic image blocks, so delegation to a frontier model keeps it; + - storage that keeps the durable log (A2) small, e.g. a reference and hash + rather than the image bytes; + - image tokens counted in the context budget; video frames multiply them. + Which local model is good enough at seeing is measured separately from + tool calling: an evaluation with rendered OpenSCAD parts carrying seeded + mistakes, and sample video frames. --- From 9239a56bd14a9a7ab47cf2af6c4f74c5ab3a8be0 Mon Sep 17 00:00:00 2001 From: Don Garrett Date: Mon, 28 Sep 2026 01:30:18 +0000 Subject: [PATCH 3/5] Spec: park vision as a need of future agents, not current ones Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/Architecture-Build-Spec.md | 33 ++++++++++++--------------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/docs/Architecture-Build-Spec.md b/docs/Architecture-Build-Spec.md index 3adf9f4..13ca947 100644 --- a/docs/Architecture-Build-Spec.md +++ b/docs/Architecture-Build-Spec.md @@ -1424,9 +1424,10 @@ The rest resolve when reached, not now. and know nothing about which model is loaded. (Measured 2026-09-28: they only throttled the benchmark, and the token limiter leaked until LiteLLM restarted.) - **Preferred: one local model for every local role.** No swaps at all. It - needs one model good enough as the dev executor, for ops' Home Assistant tool - calls, and for vision (next item). The executor benchmark and the vision and - ops evaluations decide whether one exists. + needs one model good enough as the dev executor and for ops' Home Assistant + tool calls. The executor benchmark and the ops evaluation decide whether one + exists. More agents are planned for other jobs, which makes one shared model + more valuable still, and some of them may add vision (next item). - **Fallback: agents tag in and out.** If no single model fits, the agents coordinate their use of the host rather than interleave: an agent holds the host for a stretch of work on its model, then hands it over, so a swap happens @@ -1434,24 +1435,14 @@ The rest resolve when reached, not now. groups), where VRAM allows, can reduce how often that's needed. How the hand-over works (a lease, a queue, or priority for ops alerts) is only worth designing if the preferred option fails. -- **Open: vision (A10, A15).** Needed for `nisse-dev`'s OpenSCAD work, where the - model must look at a rendered part to check it against the design, and for - video-file management planned later (which agent owns it is not yet - decided). The local candidates on the - 4090 host accept images, and video as sampled frames, but the agent is - text-only: canonical events carry strings, and neither renderer can send an - image. Needed before any vision work: - - a way for a tool to return an image, e.g. an OpenSCAD render or a frame from - a video file; - - image content in the canonical event model, rendered as OpenAI image parts - and Anthropic image blocks, so delegation to a frontier model keeps it; - - storage that keeps the durable log (A2) small, e.g. a reference and hash - rather than the image bytes; - - image tokens counted in the context budget; video frames multiply them. - Which local model is good enough at seeing is measured separately from - tool calling: an evaluation with rendered OpenSCAD parts carrying seeded - mistakes, and sample video frames. - +- **Parked: vision, for future agents.** More agents are planned for other kinds + of jobs, and some will need vision: checking OpenSCAD renders against a design, + or managing video files. Neither current agent needs it. The local candidates on + the 4090 host already accept images, and video as sampled frames, but the agent + code is text-only. Building it will need tools that return images, image content + in canonical events (A10) and both renderers, storage that keeps the durable log + (A2) small, and image tokens in the context budget. How well a model sees is + measured when that work starts, not before. --- ## Repository Layout From 75d96b4197260328e69544d9423e73745080b90d Mon Sep 17 00:00:00 2001 From: Don Garrett Date: Mon, 28 Sep 2026 01:30:36 +0000 Subject: [PATCH 4/5] Spec: restore the blank line before the section rule Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/Architecture-Build-Spec.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Architecture-Build-Spec.md b/docs/Architecture-Build-Spec.md index 13ca947..e609ecc 100644 --- a/docs/Architecture-Build-Spec.md +++ b/docs/Architecture-Build-Spec.md @@ -1443,6 +1443,7 @@ The rest resolve when reached, not now. in canonical events (A10) and both renderers, storage that keeps the durable log (A2) small, and image tokens in the context budget. How well a model sees is measured when that work starts, not before. + --- ## Repository Layout From 2ab86bf0dae7894328dae7a9787e9d32fd2c739b Mon Sep 17 00:00:00 2001 From: Don Garrett Date: Mon, 28 Sep 2026 01:32:20 +0000 Subject: [PATCH 5/5] Spec: OpenSCAD work needs vision, in nisse-dev or its own agent Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/Architecture-Build-Spec.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/Architecture-Build-Spec.md b/docs/Architecture-Build-Spec.md index e609ecc..6430cbc 100644 --- a/docs/Architecture-Build-Spec.md +++ b/docs/Architecture-Build-Spec.md @@ -1427,7 +1427,8 @@ The rest resolve when reached, not now. needs one model good enough as the dev executor and for ops' Home Assistant tool calls. The executor benchmark and the ops evaluation decide whether one exists. More agents are planned for other jobs, which makes one shared model - more valuable still, and some of them may add vision (next item). + more valuable still, and OpenSCAD and video work would add vision (next + item). - **Fallback: agents tag in and out.** If no single model fits, the agents coordinate their use of the host rather than interleave: an agent holds the host for a stretch of work on its model, then hands it over, so a swap happens @@ -1435,9 +1436,11 @@ The rest resolve when reached, not now. groups), where VRAM allows, can reduce how often that's needed. How the hand-over works (a lease, a queue, or priority for ops alerts) is only worth designing if the preferred option fails. -- **Parked: vision, for future agents.** More agents are planned for other kinds - of jobs, and some will need vision: checking OpenSCAD renders against a design, - or managing video files. Neither current agent needs it. The local candidates on +- **Parked: vision.** OpenSCAD development needs it: the language describes 3D + parts, and the only way to check a change is to render it and look. That work + may belong to `nisse-dev` or, being different enough from ordinary code, to an + agent of its own; undecided. Video-file management, planned for a future + agent, needs it too. Ordinary code work doesn't. The local candidates on the 4090 host already accept images, and video as sampled frames, but the agent code is text-only. Building it will need tools that return images, image content in canonical events (A10) and both renderers, storage that keeps the durable log