Spec: prefer one local model for all roles; record host sharing, park vision #54

2026-09-28T01:25:39Z

dev-agent commented

2026-09-28 01:25:39 +00:00

Open: sharing the local inference host between agents

nisse-dev and nisse-ops are separate agents, and both send their local roles to the one 24GB host, where llama-swap keeps one model resident at a time. If the agents use different local models, interleaved requests reload weights on every switch, at 10–60 s each. LiteLLM's per-key rate limits don't address this: they cap one key's throughput and know nothing about which model is loaded. In practice they only throttled the benchmark.

Preferred: one local model for every local role, the dev executor and ops' Home Assistant tool calls. No model swaps. More agents are planned for other jobs, which makes one shared model more valuable still.
Fallback: agents tag in and out. Each agent holds the host for a stretch of work, then hands it over, so swaps happen per hand-over rather than per request. It's only designed if no single model fits.

Parked: vision

OpenSCAD development needs vision, because the only way to check a change to a 3D part is to render it and look. That work may belong to nisse-dev, or to an agent of its own since it's different enough from ordinary code; that's undecided. Future video-file management needs vision too. The item notes what the text-only agent code would need when that work starts, and defers measuring how well models see until then.

🤖 Generated with Claude Code

## Open: sharing the local inference host between agents `nisse-dev` and `nisse-ops` are separate agents, and both send their local roles to the one 24GB host, where llama-swap keeps one model resident at a time. If the agents use different local models, interleaved requests reload weights on every switch, at 10–60 s each. LiteLLM's per-key rate limits don't address this: they cap one key's throughput and know nothing about which model is loaded. In practice they only throttled the benchmark. - **Preferred: one local model for every local role**, the dev executor and ops' Home Assistant tool calls. No model swaps. More agents are planned for other jobs, which makes one shared model more valuable still. - **Fallback: agents tag in and out.** Each agent holds the host for a stretch of work, then hands it over, so swaps happen per hand-over rather than per request. It's only designed if no single model fits. ## Parked: vision OpenSCAD development needs vision, because the only way to check a change to a 3D part is to render it and look. That work may belong to `nisse-dev`, or to an agent of its own since it's different enough from ordinary code; that's undecided. Future video-file management needs vision too. The item notes what the text-only agent code would need when that work starts, and defers measuring how well models see until then. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-28 01:25:39 +00:00

Spec: record sharing the local inference host as an open decision

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m15s

Details

e5c18b3bf4

nisse-dev and nisse-ops both send local roles to the one 24GB host,
where llama-swap keeps one model resident. Different models per agent
would reload weights on every switch. Per-key rate limits don't help.
Lists the options, simplest first, to decide after the executor
benchmark and the ops evaluation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dev-agent added 1 commit 2026-09-28 01:29:32 +00:00

Spec: prefer one local model; add vision as an open decision

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m19s

Details

50a2a75342

The host-sharing item now records the operator's preference: one local
model for every local role, with agents tagging in and out of the host
as the fallback if no single model fits. A new item records vision as a
requirement (OpenSCAD renders, later video files) and what the text-only
agent needs before any vision work.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dev-agent changed title from ~~Spec: record sharing the local inference host as an open decision~~ to Spec: prefer one local model for all roles; add host sharing and vision as open decisions

2026-09-28 01:29:38 +00:00

dev-agent added 1 commit 2026-09-28 01:30:21 +00:00

Spec: park vision as a need of future agents, not current ones

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m20s

Details

9239a56bd1

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dev-agent changed title from ~~Spec: prefer one local model for all roles; add host sharing and vision as open decisions~~ to Spec: prefer one local model for all roles; record host sharing, park vision

2026-09-28 01:30:37 +00:00

dev-agent added 1 commit 2026-09-28 01:30:37 +00:00

Spec: restore the blank line before the section rule

CI / checks (push) Skipped

CI / checks (pull_request) Failing after 1m14s

Details

75d96b4197

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dev-agent added 1 commit 2026-09-28 01:32:23 +00:00

Spec: OpenSCAD work needs vision, in nisse-dev or its own agent

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m18s

Details

2ab86bf0da

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dongar approved these changes

2026-09-28 01:47:04 +00:00

dongar merged commit a891932739 into main

2026-09-28 01:47:08 +00:00

dongar deleted branch spec-shared-inference-host

2026-09-28 01:47:08 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#54