nisse/docs/Tuning-Guide.md

Tuning Guide

How to fit a local model onto the inference host, and how to set the agent's tuning knobs. Part 1 covers the ideas you need to reason about the hardware. Part 2 covers each knob in nisse-dev.yaml. Part 3 gives an order to tune in.

The knobs live under tuning: in the config file (see nisse-dev.example.yaml). They are read at startup, so a change takes effect on the next restart. Unknown or misspelt keys are startup errors.

---

Part 1 — How a local model spends VRAM

A model server holding a model in GPU memory spends it on three things:

1. Weights, which are fixed for a given model and quantization.
2. The KV cache, which grows with context length.
3. Working memory for activations and runtime buffers. Budget 1–2 GB of headroom.

On a 24 GB card, the whole game is trading (1) against (2).

Weights and quantization

A model's weights take roughly parameters × bits per weight ÷ 8 bytes.

	16-bit (full)	8-bit	~4.5-bit (Q4_K_M, AWQ)
24B	48 GB	24 GB	~14 GB
32B	64 GB	32 GB	~19 GB

Quantization stores each weight in fewer bits. Common 4-bit formats come out nearer 4.5–5 bits per weight once scaling factors are counted, so check the actual file size. The quality cost is small at 5–6 bits, usually acceptable at 4, and it climbs quickly below that. For agent work the cost shows up first as worse tool-call discipline, not as worse prose. The capability matrix (Part 3) measures exactly that.

Mixture-of-experts models such as Qwen3-30B-A3B are different. Memory is set by total parameters (30B), but speed is set by the parameters active for each token (3B). An MoE model of a given size therefore generates much faster than a dense model of the same size. It's often weaker than a dense model of the same total size, but not always for tool use.

The KV cache, and why context costs memory

As the model reads the prompt, every layer stores a key and a value vector for every token, so that later tokens can attend to them without recomputing. That store is the KV cache. Its size per token is:

2 (key and value) × layers × KV heads × head size × bytes per value

The numbers come from the model's config.json (num_hidden_layers, num_key_value_heads, head_dim). Some worked examples at 16-bit:

Model	Layers × KV heads × head size	Per token	32k tokens
Qwen3-32B	64 × 8 × 128	256 KB	~8 GB
Mistral/Devstral Small 24B	40 × 8 × 128	160 KB	~5 GB
Qwen3-30B-A3B (MoE)	48 × 4 × 128	96 KB	~3 GB

(Taken from the published configs; recheck against the model card before relying on them.)

Two consequences:

• Context length is a memory decision, not a free setting. The spec's estimate (Part H) was about 5 GB left for KV cache beside a 4-bit 32B model. At 16-bit, that's only about 20k tokens of context for Qwen3-32B.
• KV-cache quantization stores keys and values in 8 bits (or 4). At 8 bits it halves the cache and is close to lossless. At 4 bits it is noticeably worse. Quantizing the cache to 8-bit is usually the first lever to pull, since it buys twice the context for almost nothing.

The server allocates the cache for the context length you configure. It doesn't matter how much of it a request uses. So set the context length on the server deliberately: it is the ceiling for prompt plus response on every request.

Prompt processing vs generation

A request has two phases, and they run at very different speeds:

• Prefill reads the prompt. It is parallel and fast: thousands of tokens per second on a 4090.
• Decode writes the response one token at a time. Roughly 30–40 tokens per second for a dense 32B at 4-bit, and several times that for a 3B-active MoE. Treat these numbers as ballpark; measure your own.

An agent loop re-sends the whole transcript on every turn, which makes the prompt long and the responses short (a tool call is often under 200 tokens). So prefill matters more than the headline tokens-per-second suggests.

Prefix caching is the fix. If the start of a prompt matches the previous request, the server reuses the KV cache for it and prefills only the new part. llama.cpp does this per slot, and vLLM does it automatically. For this workload it is often the single largest speed win. It works here because the rendered transcript only grows at the end, so consecutive turns share everything but their newest events.

Tokens and characters

For English and code, a token is roughly 4 characters. That's useful for converting the character-based knobs below into context cost.

---

Part 2 — The knobs

For each knob: what it controls, what goes wrong at either extreme, and what to look at when deciding.

llm.max_output_tokens — default 16384

The ceiling on a single response. It is sent with every request, for every role.

• Why it exists: a response that hits the ceiling is truncated mid-sentence, or worse, mid-tool-call, where it can still parse. The agent detects truncation and treats it as an error, but a ceiling that's too tight still costs a failed turn. Planning with a frontier model was observed using over 4,000 output tokens.
• Too high, on a local model: the server reserves room for the whole response inside the context window. With a 32k context and a 16k ceiling, only 16k is left for the prompt. Some servers reject any request where prompt plus ceiling exceeds the context.
• Too low: truncated plans and tool calls.
• Deciding: executor turns are short, so 2–4k is plenty for a local executor. Frontier planning wants the large default. This one knob covers every role for now; a per-model setting is coming with the context budget (see Coming next).
• Environment override: NISSE_MAX_OUTPUT_TOKENS.

tuning.executor.max_turns — default 24

How many model calls one milestone may make before the executor stops with capped and asks you what to do.

• Too low: milestones that were making progress stop short. You'll see capped outcomes on work that was going well.
• Too high: a confused model keeps going. Every turn also adds its tool calls and results to the transcript, so a long run spends context as well as time.
• Deciding: look at how many turns successful milestones take, and set the cap comfortably above the high end, around 1.5–2× the turns of your longer successful runs. A stronger model usually needs fewer turns, not more.

tuning.executor.failure_limit — default 2

How many consecutive failed run_command calls trigger escalation: the sub-task is handed to a frontier model as a delegation (spec A12). A successful command resets the count. Failed file reads don't count, since reading is looking around, not doing the work.

• Why it's a count and not a judgment: small models are poorly calibrated about their own competence. Asked "are you stuck?", they say no until far too late, or yes constantly. A count of observable failures is a signal that doesn't depend on the model's opinion of itself.
• Too low: normal iteration escalates. Test-first work expects a failing test run before the fix, so a limit of 1 hands off routine work and spends frontier tokens on it.
• Too high: a weak model circles, re-running the same failing command with small variations, while the transcript fills up.
• Deciding: raise it as local models get stronger. Watch two things together. First, how often delegations fix what the local model couldn't (escalation is doing its job). Second, how often the local model recovers on its own after its second failure (the limit may be too low).

tuning.expert.consult_budget — default 3

How many ask_expert calls one milestone may make. ask_expert is a single stateless question to a stronger model, with no access to the repository (spec A14), for things like "is this API deprecated?".

• Too low: the executor can't get unstuck on a knowledge question it can't answer from the code.
• Too high: a confused executor asks variations of the same question instead of doing the work or failing. That's the circling A12 warns about.
• Deciding: count consults per milestone, and check whether the answers changed what the executor did next. If it regularly exhausts the budget without progress, the budget is fine and the model or the plan is the problem.

tuning.delegation — hard caps on one delegation

A delegation is a frontier model working a sub-task with full tool access (spec A13). When a cap is hit, the delegation fails, and its branch and work are discarded, because resuming on a half-edited tree isn't safe.

Knob	Default	Caps
max_turns	30	model calls
max_tokens	200000	input + output tokens, as reported by the provider
wall_clock_s	900	elapsed time

• max_tokens is a scope limit, not the money limit. Spending limits live on the LiteLLM key (spec A5). This cap stops one delegation from growing without bound. An unbounded one could still exhaust a daily budget and take every frontier model away until it resets.
• Too low: delegations fail just short of finishing and their work is thrown away. That is the most expensive way to be wrong, because you paid for the work.
• Too high: a stuck delegation burns tokens and time before failing.
• Deciding: every delegation records the tokens it used and how long it ran (the delegation_completed event, captured for evaluation per spec A16). Set each cap above what successful delegations actually use, with room to spare.

tuning.tools.command_timeout_s — default 120

How long one run_command may run before its whole process group is killed.

• Too low: slow test suites, dependency installs and Docker builds get killed and reported as failures. That also counts toward failure_limit.
• Too high: a command that never exits blocks the turn for the whole timeout, such as a dev server started in the foreground or a test waiting on input. stop waits for a running command to finish too, so this is also the worst-case delay before a stop takes effect.
• Deciding: set it just above the slowest legitimate command in the projects the agent works on.

tuning.tools.max_output_chars — default 4000

How much of a tool's output (command output, file contents, directory listings) the model sees. Anything past this is cut, with a note saying how much was dropped.

• This is a context knob. Every tool result stays in the transcript and is re-sent every turn. 4,000 characters is about 1,000 tokens, so 20 turns of maximal output is about 20k tokens, which is already most of a local model's context.
• Too low: the model doesn't see the part of the output that matters.
• Too high: the transcript outgrows the context window sooner.
• A known weakness: truncation keeps the start of the output, but test runners put their failure summary at the end. A long, failing pytest run can lose exactly the lines the model needed. Keeping the head and the tail would fix this, and is a candidate for the context-budget work.

---

Part 3 — An order to tune in

Upstream choices change what's right downstream, so tune in this order and change one thing at a time.

1. Model and quantization. Start with whether a candidate holds the tool-call contract at all:

   NISSE_CAPABILITY_MODELS=alias-a,alias-b \
     uv run pytest -m integration tests/integration/test_model_capability.py
   

   A model that falls back to prose instead of calling tools is disqualified, however good its benchmarks look. Before comparing aliases, check what each one actually points at in LiteLLM (/model/info, the litellm_params.model field). Two aliases believed to be the same model once weren't, and that cost a lot of debugging.

2. Context length and KV-cache precision on the server. Take the largest context that fits beside the weights with 1–2 GB of headroom. Try an 8-bit KV cache before giving up context.

3. llm.max_output_tokens, so the prompt keeps most of that context.

4. Real milestones. Run representative work and look at the evidence: turns used, escalations, delegation outcomes, and time taken.

5. The loop knobs: failure_limit, max_turns, consult_budget, the delegation caps, then the tool limits, one at a time.

Expect noise. Local model behaviour varies a lot from run to run. One early comparison measured 12 of 20 runs holding the tool contract against 20 of 20, a real difference (p ≈ 0.003). Three runs each could easily have pointed the other way. Before changing a knob because of a result, repeat the run enough times that the result would survive a coin flip.

---

Coming next

The next changes in the 4090-readiness work add knobs, and this guide gains a section for each:

• A context budget. Rendering a transcript that fits the model's context window, with what to drop or summarize when it doesn't, and per-model output ceilings.
• Enforced tool calls. An explicit finish tool, so the server can be told a tool call is required and hold the model to the contract, where today it can fall back to prose.
• A benchmark script that runs the capability matrix and representative milestones across candidate models and prints a comparison table.