Spec: prefer one local model for all roles; record host sharing, park vision #54

Merged
dongar merged 5 commits from spec-shared-inference-host into main 2026-09-28 01:47:08 +00:00
Showing only changes of commit 9239a56bd1 - Show all commits
+12 -21
View File
@@ -1424,9 +1424,10 @@ The rest resolve when reached, not now.
and know nothing about which model is loaded. (Measured 2026-09-28: they only
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
- **Preferred: one local model for every local role.** No swaps at all. It
needs one model good enough as the dev executor, for ops' Home Assistant tool
calls, and for vision (next item). The executor benchmark and the vision and
ops evaluations decide whether one exists.
needs one model good enough as the dev executor and for ops' Home Assistant
tool calls. The executor benchmark and the ops evaluation decide whether one
exists. More agents are planned for other jobs, which makes one shared model
more valuable still, and some of them may add vision (next item).
- **Fallback: agents tag in and out.** If no single model fits, the agents
coordinate their use of the host rather than interleave: an agent holds the
host for a stretch of work on its model, then hands it over, so a swap happens
@@ -1434,24 +1435,14 @@ The rest resolve when reached, not now.
groups), where VRAM allows, can reduce how often that's needed. How the
hand-over works (a lease, a queue, or priority for ops alerts) is only worth
designing if the preferred option fails.
- **Open: vision (A10, A15).** Needed for `nisse-dev`'s OpenSCAD work, where the
model must look at a rendered part to check it against the design, and for
video-file management planned later (which agent owns it is not yet
decided). The local candidates on the
4090 host accept images, and video as sampled frames, but the agent is
text-only: canonical events carry strings, and neither renderer can send an
image. Needed before any vision work:
- a way for a tool to return an image, e.g. an OpenSCAD render or a frame from
a video file;
- image content in the canonical event model, rendered as OpenAI image parts
and Anthropic image blocks, so delegation to a frontier model keeps it;
- storage that keeps the durable log (A2) small, e.g. a reference and hash
rather than the image bytes;
- image tokens counted in the context budget; video frames multiply them.
Which local model is good enough at seeing is measured separately from
tool calling: an evaluation with rendered OpenSCAD parts carrying seeded
mistakes, and sample video frames.
- **Parked: vision, for future agents.** More agents are planned for other kinds
of jobs, and some will need vision: checking OpenSCAD renders against a design,
or managing video files. Neither current agent needs it. The local candidates on
the 4090 host already accept images, and video as sampled frames, but the agent
code is text-only. Building it will need tools that return images, image content
in canonical events (A10) and both renderers, storage that keeps the durable log
(A2) small, and image tokens in the context budget. How well a model sees is
measured when that work starts, not before.
---
## Repository Layout