Merge pull request 'Spec: prefer one local model for all roles; record host sharing, park vision' (#54) from spec-shared-inference-host into main
CI / checks (push) Successful in 1m16s

Reviewed-on: dongar/nisse#54
Reviewed-by: Don Garrett <dgarrett@acm.org>
This commit was merged in pull request #54.
This commit is contained in:
dongar committed 2026-09-28 01:47:08 +00:00
commit a891932739
1 file changed
+32
+32
View File
@@ -1414,6 +1414,38 @@ The rest resolve when reached, not now.
and ticket-driven ops work already require one. Whether every gated work item
needs one (for instance, an ops question that turns into a change to make)
is best settled once C2's step sequences are in use.
- **Open: sharing the local inference host between agents.** Each agent keeps
one active work item (0.2), but `nisse-dev` and `nisse-ops` are separate agents,
and both send their local roles through LiteLLM to the one 24GB host. llama-swap
there keeps one model resident at a time. If the two agents' local roles map to
different models, interleaved requests reload weights on every switch (10–60 s
each), and each agent waits on the other's loads. LiteLLM's per-key rate limits
don't address this: they cap one key's throughput rather than share the host,
and know nothing about which model is loaded. (Measured 2026-09-28: they only
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
- **Preferred: one local model for every local role.** No swaps at all. It
needs one model good enough as the dev executor and for ops' Home Assistant
tool calls. The executor benchmark and the ops evaluation decide whether one
exists. More agents are planned for other jobs, which makes one shared model
more valuable still, and OpenSCAD and video work would add vision (next
item).
- **Fallback: agents tag in and out.** If no single model fits, the agents
coordinate their use of the host rather than interleave: an agent holds the
host for a stretch of work on its model, then hands it over, so a swap happens
per hand-over rather than per request. Two models co-resident (llama-swap
groups), where VRAM allows, can reduce how often that's needed. How the
hand-over works (a lease, a queue, or priority for ops alerts) is only worth
designing if the preferred option fails.
- **Parked: vision.** OpenSCAD development needs it: the language describes 3D
parts, and the only way to check a change is to render it and look. That work
may belong to `nisse-dev` or, being different enough from ordinary code, to an
agent of its own; undecided. Video-file management, planned for a future
agent, needs it too. Ordinary code work doesn't. The local candidates on
the 4090 host already accept images, and video as sampled frames, but the agent
code is text-only. Building it will need tools that return images, image content
in canonical events (A10) and both renderers, storage that keeps the durable log
(A2) small, and image tokens in the context budget. How well a model sees is
measured when that work starts, not before.
---