Spec: prefer one local model for all roles; record host sharing, park vision #54
@@ -1414,6 +1414,38 @@ The rest resolve when reached, not now.
|
||||
and ticket-driven ops work already require one. Whether every gated work item
|
||||
needs one (for instance, an ops question that turns into a change to make)
|
||||
is best settled once C2's step sequences are in use.
|
||||
- **Open: sharing the local inference host between agents.** Each agent keeps
|
||||
one active work item (0.2), but `nisse-dev` and `nisse-ops` are separate agents,
|
||||
and both send their local roles through LiteLLM to the one 24GB host. llama-swap
|
||||
there keeps one model resident at a time. If the two agents' local roles map to
|
||||
different models, interleaved requests reload weights on every switch (10–60 s
|
||||
each), and each agent waits on the other's loads. LiteLLM's per-key rate limits
|
||||
don't address this: they cap one key's throughput rather than share the host,
|
||||
and know nothing about which model is loaded. (Measured 2026-09-28: they only
|
||||
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
|
||||
- **Preferred: one local model for every local role.** No swaps at all. It
|
||||
needs one model good enough as the dev executor and for ops' Home Assistant
|
||||
tool calls. The executor benchmark and the ops evaluation decide whether one
|
||||
exists. More agents are planned for other jobs, which makes one shared model
|
||||
more valuable still, and OpenSCAD and video work would add vision (next
|
||||
item).
|
||||
- **Fallback: agents tag in and out.** If no single model fits, the agents
|
||||
coordinate their use of the host rather than interleave: an agent holds the
|
||||
host for a stretch of work on its model, then hands it over, so a swap happens
|
||||
per hand-over rather than per request. Two models co-resident (llama-swap
|
||||
groups), where VRAM allows, can reduce how often that's needed. How the
|
||||
hand-over works (a lease, a queue, or priority for ops alerts) is only worth
|
||||
designing if the preferred option fails.
|
||||
- **Parked: vision.** OpenSCAD development needs it: the language describes 3D
|
||||
parts, and the only way to check a change is to render it and look. That work
|
||||
may belong to `nisse-dev` or, being different enough from ordinary code, to an
|
||||
agent of its own; undecided. Video-file management, planned for a future
|
||||
agent, needs it too. Ordinary code work doesn't. The local candidates on
|
||||
the 4090 host already accept images, and video as sampled frames, but the agent
|
||||
code is text-only. Building it will need tools that return images, image content
|
||||
in canonical events (A10) and both renderers, storage that keeps the durable log
|
||||
(A2) small, and image tokens in the context budget. How well a model sees is
|
||||
measured when that work starts, not before.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user