Spec: prefer one local model for all roles; record host sharing, park vision #54
@@ -1424,9 +1424,10 @@ The rest resolve when reached, not now.
|
||||
and know nothing about which model is loaded. (Measured 2026-09-28: they only
|
||||
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
|
||||
- **Preferred: one local model for every local role.** No swaps at all. It
|
||||
needs one model good enough as the dev executor, for ops' Home Assistant tool
|
||||
calls, and for vision (next item). The executor benchmark and the vision and
|
||||
ops evaluations decide whether one exists.
|
||||
needs one model good enough as the dev executor and for ops' Home Assistant
|
||||
tool calls. The executor benchmark and the ops evaluation decide whether one
|
||||
exists. More agents are planned for other jobs, which makes one shared model
|
||||
more valuable still, and some of them may add vision (next item).
|
||||
- **Fallback: agents tag in and out.** If no single model fits, the agents
|
||||
coordinate their use of the host rather than interleave: an agent holds the
|
||||
host for a stretch of work on its model, then hands it over, so a swap happens
|
||||
@@ -1434,24 +1435,14 @@ The rest resolve when reached, not now.
|
||||
groups), where VRAM allows, can reduce how often that's needed. How the
|
||||
hand-over works (a lease, a queue, or priority for ops alerts) is only worth
|
||||
designing if the preferred option fails.
|
||||
- **Open: vision (A10, A15).** Needed for `nisse-dev`'s OpenSCAD work, where the
|
||||
model must look at a rendered part to check it against the design, and for
|
||||
video-file management planned later (which agent owns it is not yet
|
||||
decided). The local candidates on the
|
||||
4090 host accept images, and video as sampled frames, but the agent is
|
||||
text-only: canonical events carry strings, and neither renderer can send an
|
||||
image. Needed before any vision work:
|
||||
- a way for a tool to return an image, e.g. an OpenSCAD render or a frame from
|
||||
a video file;
|
||||
- image content in the canonical event model, rendered as OpenAI image parts
|
||||
and Anthropic image blocks, so delegation to a frontier model keeps it;
|
||||
- storage that keeps the durable log (A2) small, e.g. a reference and hash
|
||||
rather than the image bytes;
|
||||
- image tokens counted in the context budget; video frames multiply them.
|
||||
Which local model is good enough at seeing is measured separately from
|
||||
tool calling: an evaluation with rendered OpenSCAD parts carrying seeded
|
||||
mistakes, and sample video frames.
|
||||
|
||||
- **Parked: vision, for future agents.** More agents are planned for other kinds
|
||||
of jobs, and some will need vision: checking OpenSCAD renders against a design,
|
||||
or managing video files. Neither current agent needs it. The local candidates on
|
||||
the 4090 host already accept images, and video as sampled frames, but the agent
|
||||
code is text-only. Building it will need tools that return images, image content
|
||||
in canonical events (A10) and both renderers, storage that keeps the durable log
|
||||
(A2) small, and image tokens in the context budget. How well a model sees is
|
||||
measured when that work starts, not before.
|
||||
---
|
||||
|
||||
## Repository Layout
|
||||
|
||||
Reference in new issue
Block a user