Spec: prefer one local model; add vision as an open decision
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m19s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m19s
The host-sharing item now records the operator's preference: one local model for every local role, with agents tagging in and out of the host as the fallback if no single model fits. A new item records vision as a requirement (OpenSCAD renders, later video files) and what the text-only agent needs before any vision work. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e5c18b3bf4
commit
50a2a75342
1 file changed
+28
-9
@@ -1423,15 +1423,34 @@ The rest resolve when reached, not now.
|
|||||||
don't address this: they cap one key's throughput rather than share the host,
|
don't address this: they cap one key's throughput rather than share the host,
|
||||||
and know nothing about which model is loaded. (Measured 2026-09-28: they only
|
and know nothing about which model is loaded. (Measured 2026-09-28: they only
|
||||||
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
|
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
|
||||||
Options, simplest first:
|
- **Preferred: one local model for every local role.** No swaps at all. It
|
||||||
- *One local model for every local role.* No swaps at all. It needs one model
|
needs one model good enough as the dev executor, for ops' Home Assistant tool
|
||||||
good enough both as the dev executor and for ops' Home Assistant tool calls.
|
calls, and for vision (next item). The executor benchmark and the vision and
|
||||||
- *Two models co-resident*, if VRAM allows (llama-swap groups): for instance a
|
ops evaluations decide whether one exists.
|
||||||
small tool-caller for ops beside the executor model. It costs context window.
|
- **Fallback: agents tag in and out.** If no single model fits, the agents
|
||||||
- *A scheduler in front of the host* that batches each model's requests to keep
|
coordinate their use of the host rather than interleave: an agent holds the
|
||||||
swaps rare. The most work; only if neither of the above fits.
|
host for a stretch of work on its model, then hands it over, so a swap happens
|
||||||
Decide once the executor benchmark and the ops evaluation show whether one model
|
per hand-over rather than per request. Two models co-resident (llama-swap
|
||||||
can serve both roles.
|
groups), where VRAM allows, can reduce how often that's needed. How the
|
||||||
|
hand-over works (a lease, a queue, or priority for ops alerts) is only worth
|
||||||
|
designing if the preferred option fails.
|
||||||
|
- **Open: vision (A10, A15).** Needed for `nisse-dev`'s OpenSCAD work, where the
|
||||||
|
model must look at a rendered part to check it against the design, and for
|
||||||
|
video-file management planned later (which agent owns it is not yet
|
||||||
|
decided). The local candidates on the
|
||||||
|
4090 host accept images, and video as sampled frames, but the agent is
|
||||||
|
text-only: canonical events carry strings, and neither renderer can send an
|
||||||
|
image. Needed before any vision work:
|
||||||
|
- a way for a tool to return an image, e.g. an OpenSCAD render or a frame from
|
||||||
|
a video file;
|
||||||
|
- image content in the canonical event model, rendered as OpenAI image parts
|
||||||
|
and Anthropic image blocks, so delegation to a frontier model keeps it;
|
||||||
|
- storage that keeps the durable log (A2) small, e.g. a reference and hash
|
||||||
|
rather than the image bytes;
|
||||||
|
- image tokens counted in the context budget; video frames multiply them.
|
||||||
|
Which local model is good enough at seeing is measured separately from
|
||||||
|
tool calling: an evaluation with rendered OpenSCAD parts carrying seeded
|
||||||
|
mistakes, and sample video frames.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user