Spec: prefer one local model for all roles; record host sharing, park vision #54
@@ -1423,15 +1423,34 @@ The rest resolve when reached, not now.
|
||||
don't address this: they cap one key's throughput rather than share the host,
|
||||
and know nothing about which model is loaded. (Measured 2026-09-28: they only
|
||||
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
|
||||
Options, simplest first:
|
||||
- *One local model for every local role.* No swaps at all. It needs one model
|
||||
good enough both as the dev executor and for ops' Home Assistant tool calls.
|
||||
- *Two models co-resident*, if VRAM allows (llama-swap groups): for instance a
|
||||
small tool-caller for ops beside the executor model. It costs context window.
|
||||
- *A scheduler in front of the host* that batches each model's requests to keep
|
||||
swaps rare. The most work; only if neither of the above fits.
|
||||
Decide once the executor benchmark and the ops evaluation show whether one model
|
||||
can serve both roles.
|
||||
- **Preferred: one local model for every local role.** No swaps at all. It
|
||||
needs one model good enough as the dev executor, for ops' Home Assistant tool
|
||||
calls, and for vision (next item). The executor benchmark and the vision and
|
||||
ops evaluations decide whether one exists.
|
||||
- **Fallback: agents tag in and out.** If no single model fits, the agents
|
||||
coordinate their use of the host rather than interleave: an agent holds the
|
||||
host for a stretch of work on its model, then hands it over, so a swap happens
|
||||
per hand-over rather than per request. Two models co-resident (llama-swap
|
||||
groups), where VRAM allows, can reduce how often that's needed. How the
|
||||
hand-over works (a lease, a queue, or priority for ops alerts) is only worth
|
||||
designing if the preferred option fails.
|
||||
- **Open: vision (A10, A15).** Needed for `nisse-dev`'s OpenSCAD work, where the
|
||||
model must look at a rendered part to check it against the design, and for
|
||||
video-file management planned later (which agent owns it is not yet
|
||||
decided). The local candidates on the
|
||||
4090 host accept images, and video as sampled frames, but the agent is
|
||||
text-only: canonical events carry strings, and neither renderer can send an
|
||||
image. Needed before any vision work:
|
||||
- a way for a tool to return an image, e.g. an OpenSCAD render or a frame from
|
||||
a video file;
|
||||
- image content in the canonical event model, rendered as OpenAI image parts
|
||||
and Anthropic image blocks, so delegation to a frontier model keeps it;
|
||||
- storage that keeps the durable log (A2) small, e.g. a reference and hash
|
||||
rather than the image bytes;
|
||||
- image tokens counted in the context budget; video frames multiply them.
|
||||
Which local model is good enough at seeing is measured separately from
|
||||
tool calling: an evaluation with rendered OpenSCAD parts carrying seeded
|
||||
mistakes, and sample video frames.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user