From e5c18b3bf4f22c07e846bd048ffe2801e139786a Mon Sep 17 00:00:00 2001 From: Don Garrett Date: Mon, 28 Sep 2026 01:25:38 +0000 Subject: [PATCH] Spec: record sharing the local inference host as an open decision nisse-dev and nisse-ops both send local roles to the one 24GB host, where llama-swap keeps one model resident. Different models per agent would reload weights on every switch. Per-key rate limits don't help. Lists the options, simplest first, to decide after the executor benchmark and the ops evaluation. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/Architecture-Build-Spec.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/Architecture-Build-Spec.md b/docs/Architecture-Build-Spec.md index b795a53..50a2984 100644 --- a/docs/Architecture-Build-Spec.md +++ b/docs/Architecture-Build-Spec.md @@ -1414,6 +1414,24 @@ The rest resolve when reached, not now. and ticket-driven ops work already require one. Whether every gated work item needs one (for instance, an ops question that turns into a change to make) is best settled once C2's step sequences are in use. +- **Open: sharing the local inference host between agents.** Each agent keeps + one active work item (0.2), but `nisse-dev` and `nisse-ops` are separate agents, + and both send their local roles through LiteLLM to the one 24GB host. llama-swap + there keeps one model resident at a time. If the two agents' local roles map to + different models, interleaved requests reload weights on every switch (10–60 s + each), and each agent waits on the other's loads. LiteLLM's per-key rate limits + don't address this: they cap one key's throughput rather than share the host, + and know nothing about which model is loaded. (Measured 2026-09-28: they only + throttled the benchmark, and the token limiter leaked until LiteLLM restarted.) + Options, simplest first: + - *One local model for every local role.* No swaps at all. It needs one model + good enough both as the dev executor and for ops' Home Assistant tool calls. + - *Two models co-resident*, if VRAM allows (llama-swap groups): for instance a + small tool-caller for ops beside the executor model. It costs context window. + - *A scheduler in front of the host* that batches each model's requests to keep + swaps rare. The most work; only if neither of the above fits. + Decide once the executor benchmark and the ops evaluation show whether one model + can serve both roles. ---