From 50a2a75342f5cedd83dc31669a5f2bc78c564398 Mon Sep 17 00:00:00 2001 From: Don Garrett Date: Mon, 28 Sep 2026 01:29:29 +0000 Subject: [PATCH] Spec: prefer one local model; add vision as an open decision The host-sharing item now records the operator's preference: one local model for every local role, with agents tagging in and out of the host as the fallback if no single model fits. A new item records vision as a requirement (OpenSCAD renders, later video files) and what the text-only agent needs before any vision work. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/Architecture-Build-Spec.md | 37 +++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/docs/Architecture-Build-Spec.md b/docs/Architecture-Build-Spec.md index 50a2984..3adf9f4 100644 --- a/docs/Architecture-Build-Spec.md +++ b/docs/Architecture-Build-Spec.md @@ -1423,15 +1423,34 @@ The rest resolve when reached, not now. don't address this: they cap one key's throughput rather than share the host, and know nothing about which model is loaded. (Measured 2026-09-28: they only throttled the benchmark, and the token limiter leaked until LiteLLM restarted.) - Options, simplest first: - - *One local model for every local role.* No swaps at all. It needs one model - good enough both as the dev executor and for ops' Home Assistant tool calls. - - *Two models co-resident*, if VRAM allows (llama-swap groups): for instance a - small tool-caller for ops beside the executor model. It costs context window. - - *A scheduler in front of the host* that batches each model's requests to keep - swaps rare. The most work; only if neither of the above fits. - Decide once the executor benchmark and the ops evaluation show whether one model - can serve both roles. + - **Preferred: one local model for every local role.** No swaps at all. It + needs one model good enough as the dev executor, for ops' Home Assistant tool + calls, and for vision (next item). The executor benchmark and the vision and + ops evaluations decide whether one exists. + - **Fallback: agents tag in and out.** If no single model fits, the agents + coordinate their use of the host rather than interleave: an agent holds the + host for a stretch of work on its model, then hands it over, so a swap happens + per hand-over rather than per request. Two models co-resident (llama-swap + groups), where VRAM allows, can reduce how often that's needed. How the + hand-over works (a lease, a queue, or priority for ops alerts) is only worth + designing if the preferred option fails. +- **Open: vision (A10, A15).** Needed for `nisse-dev`'s OpenSCAD work, where the + model must look at a rendered part to check it against the design, and for + video-file management planned later (which agent owns it is not yet + decided). The local candidates on the + 4090 host accept images, and video as sampled frames, but the agent is + text-only: canonical events carry strings, and neither renderer can send an + image. Needed before any vision work: + - a way for a tool to return an image, e.g. an OpenSCAD render or a frame from + a video file; + - image content in the canonical event model, rendered as OpenAI image parts + and Anthropic image blocks, so delegation to a frontier model keeps it; + - storage that keeps the durable log (A2) small, e.g. a reference and hash + rather than the image bytes; + - image tokens counted in the context budget; video frames multiply them. + Which local model is good enough at seeing is measured separately from + tool calling: an evaluation with rendered OpenSCAD parts carrying seeded + mistakes, and sample video frames. ---