Spec: prefer one local model; add vision as an open decision
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m19s

The host-sharing item now records the operator's preference: one local
model for every local role, with agents tagging in and out of the host
as the fallback if no single model fits. A new item records vision as a
requirement (OpenSCAD renders, later video files) and what the text-only
agent needs before any vision work.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-09-28 01:29:29 +00:00
1 parent e5c18b3bf4
commit 50a2a75342
1 file changed
+28 -9
+28 -9
View File
@@ -1423,15 +1423,34 @@ The rest resolve when reached, not now.
don't address this: they cap one key's throughput rather than share the host,
and know nothing about which model is loaded. (Measured 2026-09-28: they only
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
Options, simplest first:
- *One local model for every local role.* No swaps at all. It needs one model
good enough both as the dev executor and for ops' Home Assistant tool calls.
- *Two models co-resident*, if VRAM allows (llama-swap groups): for instance a
small tool-caller for ops beside the executor model. It costs context window.
- *A scheduler in front of the host* that batches each model's requests to keep
swaps rare. The most work; only if neither of the above fits.
Decide once the executor benchmark and the ops evaluation show whether one model
can serve both roles.
- **Preferred: one local model for every local role.** No swaps at all. It
needs one model good enough as the dev executor, for ops' Home Assistant tool
calls, and for vision (next item). The executor benchmark and the vision and
ops evaluations decide whether one exists.
- **Fallback: agents tag in and out.** If no single model fits, the agents
coordinate their use of the host rather than interleave: an agent holds the
host for a stretch of work on its model, then hands it over, so a swap happens
per hand-over rather than per request. Two models co-resident (llama-swap
groups), where VRAM allows, can reduce how often that's needed. How the
hand-over works (a lease, a queue, or priority for ops alerts) is only worth
designing if the preferred option fails.
- **Open: vision (A10, A15).** Needed for `nisse-dev`'s OpenSCAD work, where the
model must look at a rendered part to check it against the design, and for
video-file management planned later (which agent owns it is not yet
decided). The local candidates on the
4090 host accept images, and video as sampled frames, but the agent is
text-only: canonical events carry strings, and neither renderer can send an
image. Needed before any vision work:
- a way for a tool to return an image, e.g. an OpenSCAD render or a frame from
a video file;
- image content in the canonical event model, rendered as OpenAI image parts
and Anthropic image blocks, so delegation to a frontier model keeps it;
- storage that keeps the durable log (A2) small, e.g. a reference and hash
rather than the image bytes;
- image tokens counted in the context budget; video frames multiply them.
Which local model is good enough at seeing is measured separately from
tool calling: an evaluation with rendered OpenSCAD parts carrying seeded
mistakes, and sample video frames.
---