From 9239a56bd14a9a7ab47cf2af6c4f74c5ab3a8be0 Mon Sep 17 00:00:00 2001 From: Don Garrett Date: Mon, 28 Sep 2026 01:30:18 +0000 Subject: [PATCH] Spec: park vision as a need of future agents, not current ones Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/Architecture-Build-Spec.md | 33 ++++++++++++--------------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/docs/Architecture-Build-Spec.md b/docs/Architecture-Build-Spec.md index 3adf9f4..13ca947 100644 --- a/docs/Architecture-Build-Spec.md +++ b/docs/Architecture-Build-Spec.md @@ -1424,9 +1424,10 @@ The rest resolve when reached, not now. and know nothing about which model is loaded. (Measured 2026-09-28: they only throttled the benchmark, and the token limiter leaked until LiteLLM restarted.) - **Preferred: one local model for every local role.** No swaps at all. It - needs one model good enough as the dev executor, for ops' Home Assistant tool - calls, and for vision (next item). The executor benchmark and the vision and - ops evaluations decide whether one exists. + needs one model good enough as the dev executor and for ops' Home Assistant + tool calls. The executor benchmark and the ops evaluation decide whether one + exists. More agents are planned for other jobs, which makes one shared model + more valuable still, and some of them may add vision (next item). - **Fallback: agents tag in and out.** If no single model fits, the agents coordinate their use of the host rather than interleave: an agent holds the host for a stretch of work on its model, then hands it over, so a swap happens @@ -1434,24 +1435,14 @@ The rest resolve when reached, not now. groups), where VRAM allows, can reduce how often that's needed. How the hand-over works (a lease, a queue, or priority for ops alerts) is only worth designing if the preferred option fails. -- **Open: vision (A10, A15).** Needed for `nisse-dev`'s OpenSCAD work, where the - model must look at a rendered part to check it against the design, and for - video-file management planned later (which agent owns it is not yet - decided). The local candidates on the - 4090 host accept images, and video as sampled frames, but the agent is - text-only: canonical events carry strings, and neither renderer can send an - image. Needed before any vision work: - - a way for a tool to return an image, e.g. an OpenSCAD render or a frame from - a video file; - - image content in the canonical event model, rendered as OpenAI image parts - and Anthropic image blocks, so delegation to a frontier model keeps it; - - storage that keeps the durable log (A2) small, e.g. a reference and hash - rather than the image bytes; - - image tokens counted in the context budget; video frames multiply them. - Which local model is good enough at seeing is measured separately from - tool calling: an evaluation with rendered OpenSCAD parts carrying seeded - mistakes, and sample video frames. - +- **Parked: vision, for future agents.** More agents are planned for other kinds + of jobs, and some will need vision: checking OpenSCAD renders against a design, + or managing video files. Neither current agent needs it. The local candidates on + the 4090 host already accept images, and video as sampled frames, but the agent + code is text-only. Building it will need tools that return images, image content + in canonical events (A10) and both renderers, storage that keeps the durable log + (A2) small, and image tokens in the context budget. How well a model sees is + measured when that work starts, not before. --- ## Repository Layout