Spec: record sharing the local inference host as an open decision
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m15s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m15s
nisse-dev and nisse-ops both send local roles to the one 24GB host, where llama-swap keeps one model resident. Different models per agent would reload weights on every switch. Per-key rate limits don't help. Lists the options, simplest first, to decide after the executor benchmark and the ops evaluation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
8467b8747f
commit
e5c18b3bf4
1 file changed
+18
@@ -1414,6 +1414,24 @@ The rest resolve when reached, not now.
|
|||||||
and ticket-driven ops work already require one. Whether every gated work item
|
and ticket-driven ops work already require one. Whether every gated work item
|
||||||
needs one (for instance, an ops question that turns into a change to make)
|
needs one (for instance, an ops question that turns into a change to make)
|
||||||
is best settled once C2's step sequences are in use.
|
is best settled once C2's step sequences are in use.
|
||||||
|
- **Open: sharing the local inference host between agents.** Each agent keeps
|
||||||
|
one active work item (0.2), but `nisse-dev` and `nisse-ops` are separate agents,
|
||||||
|
and both send their local roles through LiteLLM to the one 24GB host. llama-swap
|
||||||
|
there keeps one model resident at a time. If the two agents' local roles map to
|
||||||
|
different models, interleaved requests reload weights on every switch (10–60 s
|
||||||
|
each), and each agent waits on the other's loads. LiteLLM's per-key rate limits
|
||||||
|
don't address this: they cap one key's throughput rather than share the host,
|
||||||
|
and know nothing about which model is loaded. (Measured 2026-09-28: they only
|
||||||
|
throttled the benchmark, and the token limiter leaked until LiteLLM restarted.)
|
||||||
|
Options, simplest first:
|
||||||
|
- *One local model for every local role.* No swaps at all. It needs one model
|
||||||
|
good enough both as the dev executor and for ops' Home Assistant tool calls.
|
||||||
|
- *Two models co-resident*, if VRAM allows (llama-swap groups): for instance a
|
||||||
|
small tool-caller for ops beside the executor model. It costs context window.
|
||||||
|
- *A scheduler in front of the host* that batches each model's requests to keep
|
||||||
|
swaps rare. The most work; only if neither of the above fits.
|
||||||
|
Decide once the executor benchmark and the ops evaluation show whether one model
|
||||||
|
can serve both roles.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user