nisse/docs/Tuning-Guide.md

Tuning Guide

How to fit a local model onto the inference host, and how to set the agent's tuning knobs. Part 1 covers the ideas you need to reason about the hardware. Part 2 covers each knob in nisse-dev.yaml. Part 3 gives an order to tune in.

The knobs live under tuning: in the config file (see nisse-dev.example.yaml). They are read at startup, so a change takes effect on the next restart. Unknown or misspelt keys are startup errors.

---

Part 1 — How a local model spends VRAM

A model server holding a model in GPU memory spends it on three things:

1. Weights, which are fixed for a given model and quantization.
2. The KV cache, which grows with context length.
3. Working memory for activations and runtime buffers. Budget 1–2 GB of headroom.

On a 24 GB card, the whole game is trading (1) against (2).

Weights and quantization

A model's weights take roughly parameters × bits per weight ÷ 8 bytes.

	16-bit (full)	8-bit	~4.5-bit (Q4_K_M, AWQ)
24B	48 GB	24 GB	~14 GB
32B	64 GB	32 GB	~19 GB

Quantization stores each weight in fewer bits. Common 4-bit formats come out nearer 4.5–5 bits per weight once scaling factors are counted, so check the actual file size. The quality cost is small at 5–6 bits, usually acceptable at 4, and it climbs quickly below that. For agent work the cost shows up first as worse tool-call discipline, not as worse prose. The capability matrix (Part 3) measures exactly that.

Mixture-of-experts models such as Qwen3-30B-A3B are different. Memory is set by total parameters (30B), but speed is set by the parameters active for each token (3B). An MoE model of a given size therefore generates much faster than a dense model of the same size. It's often weaker than a dense model of the same total size, but not always for tool use.

The KV cache, and why context costs memory

As the model reads the prompt, every layer stores a key and a value vector for every token, so that later tokens can attend to them without recomputing. That store is the KV cache. Its size per token is:

2 (key and value) × layers × KV heads × head size × bytes per value

The numbers come from the model's config.json (num_hidden_layers, num_key_value_heads, head_dim). Some worked examples at 16-bit:

Model	Layers × KV heads × head size	Per token	32k tokens
Qwen3-32B	64 × 8 × 128	256 KB	~8 GB
Mistral/Devstral Small 24B	40 × 8 × 128	160 KB	~5 GB
Qwen3-30B-A3B (MoE)	48 × 4 × 128	96 KB	~3 GB

(Taken from the published configs; recheck against the model card before relying on them.)

Two consequences:

• Context length is a memory decision, not a free setting. The spec's estimate (the original build spec) was about 5 GB left for KV cache beside a 4-bit 32B model. At 16-bit, that's only about 20k tokens of context for Qwen3-32B.
• KV-cache quantization stores keys and values in 8 bits (or 4). At 8 bits it halves the cache and is close to lossless. At 4 bits it is noticeably worse. Quantizing the cache to 8-bit is usually the first lever to pull, since it buys twice the context for almost nothing.

The server allocates the cache for the context length you configure. It doesn't matter how much of it a request uses. So set the context length on the server deliberately: it is the ceiling for prompt plus response on every request.

Prompt processing vs generation

A request has two phases, and they run at very different speeds:

• Prefill reads the prompt. It is parallel and fast: thousands of tokens per second on a 4090.
• Decode writes the response one token at a time. Roughly 30–40 tokens per second for a dense 32B at 4-bit, and several times that for a 3B-active MoE. Treat these numbers as ballpark; measure your own.

An agent loop re-sends the whole transcript on every turn, which makes the prompt long and the responses short (a tool call is often under 200 tokens). So prefill matters more than the headline tokens-per-second suggests.

Prefix caching is the fix. If the start of a prompt matches the previous request, the server reuses the KV cache for it and prefills only the new part. llama.cpp does this per slot, and vLLM does it automatically. For this workload it is often the single largest speed win. It works here because the rendered transcript only grows at the end, so consecutive turns share everything but their newest events.

Tokens and characters

For English and code, a token is roughly 4 characters. That's useful for converting the character-based knobs below into context cost.

---

Part 2 — The knobs

For each knob: what it controls, what goes wrong at either extreme, and what to look at when deciding.

llm.max_output_tokens — default 16384

The ceiling on a single response, for every role. The ceiling actually sent is the smaller of this and the model's own max_output_tokens in LiteLLM, so a local model can be held to less without affecting the frontier roles.

• Why it exists: a response that hits the ceiling is truncated mid-sentence, or worse, mid-tool-call, where it can still parse. The agent detects truncation and treats it as an error, but a ceiling that's too tight still costs a failed turn. Planning with a frontier model was observed using over 4,000 output tokens.
• Too high, on a local model: the server reserves room for the whole response inside the context window. With a 32k context and a 16k ceiling, only 16k is left for the prompt. Some servers reject any request where prompt plus ceiling exceeds the context.
• Too low: truncated restatements and tool calls.
• Deciding: a working turn is short, so 2–4k is plenty for a local model doing a milestone's work. A frontier driver restating and designing wants the large default. Leave this high, and set a local model's smaller ceiling as its max_output_tokens in LiteLLM.

llm.request_timeout_s — default 600

How long one model request may take before it is abandoned. It applies to every request, on both routes.

• Too low: a slow model's long answer is cut off, and the driver's run fails on work that was going well. At 120 s, the old fixed value, a dense 27B model on the RTX 4090 timed out writing one module in one turn.
• Too high: a hung connection holds the milestone for that long before anything notices.
• Deciding: this is a safety net, not a budget. max_output_tokens and max_turns bound the work; set this above the slowest single answer you expect, which is the model's output ceiling divided by its generation speed.

tuning.driver.max_turns — default 40

How many model calls one driver run may make before it stops as capped and asks you what to do. A run lasts until the driver waits (for you, a review or a merge), so a long milestone is usually one run. Before M2.8 this was tuning.executor.max_turns.

• Too low: milestones that were making progress stop short. You'll see capped outcomes on work that was going well.
• Too high: a confused model keeps going. Every turn also adds its tool calls and results to the transcript, so a long run spends context as well as time.
• Deciding: look at how many turns successful milestones take, and set the cap comfortably above the high end, around 1.5–2× the turns of your longer successful runs. A stronger model usually needs fewer turns, not more.
• Measured: the default was 24 until the harder benchmark tasks arrived. Haiku, used as the ceiling in calibration, hit 24 on both runs of refactor-rename (editing ten files one at a time) and of long-milestone, one of which had in fact finished the work. It is a guard against a model trying forever, not a budget, so it was raised to 40.

tuning.driver.failure_limit — default 2

How many consecutive failed run_command calls mean the driver is stuck: the first time in a milestone, a helper takes the problem as a delegation (architecture L9); the second time, the item waits on you. A successful command resets the count. Failed file reads don't count, since reading is looking around, not doing the work. Before M2.8 this was tuning.executor.failure_limit.

• Why it's a count and not a judgment: small models are poorly calibrated about their own competence. Asked "are you stuck?", they say no until far too late, or yes constantly. A count of observable failures is a signal that doesn't depend on the model's opinion of itself.
• Too low: normal iteration escalates. Test-first work expects a failing test run before the fix, so a limit of 1 hands off routine work and spends frontier tokens on it.
• Too high: a weak model circles, re-running the same failing command with small variations, while the transcript fills up.
• Deciding: raise it as local models get stronger. Watch two things together. First, how often delegations fix what the local model couldn't (escalation is doing its job). Second, how often the local model recovers on its own after its second failure (the limit may be too low).

tuning.preflight.max_turns — default 20

How many model calls one toolchain step may make (D3). The harness starts one before the driver plans whenever a project's build VM has no marker from a finished one (a new project, a new or wiped VM), and the driver asks for one when the work needs a toolchain change.

• Too low: a project with several things to install doesn't finish, and says so; work goes on with what's missing stated, and the next item tries again.
• Too high: a model that can't settle on a toolchain spends its time there instead of on the work.
• Deciding: a project that only needs checking takes 2–4 calls; each install adds one or two.

tuning.preflight.command_timeout_s — default 900

The limit on one command during a toolchain step, in place of tuning.tools.command_timeout_s: installing a compiler or a language runtime can take minutes. Commands in ordinary runs keep their own, shorter limit.

tuning.expert.consult_budget — default 3

How many ask_expert calls one milestone may make. ask_expert is a single stateless question to a stronger model, with no access to the repository (architecture L9), for things like "is this API deprecated?".

• Too low: the driver can't get unstuck on a knowledge question it can't answer from the code.
• Too high: a confused driver asks variations of the same question instead of doing the work or failing. That's the circling L9 warns about.
• Deciding: count consults per milestone, and check whether the answers changed what the driver did next. If it regularly exhausts the budget without progress, the budget is fine and the model or the brief is the problem.

tuning.delegation — hard caps on one delegation

A delegation is a frontier model working a sub-task with full tool access (architecture L9). When a cap is hit, the delegation fails, and its branch and work are discarded, because resuming on a half-edited tree isn't safe.

Knob	Default	Caps
max_turns	30	model calls
max_tokens	200000	input + output tokens, as reported by the provider
wall_clock_s	900	elapsed time

• max_tokens is a scope limit, not the money limit. Spending limits live on the LiteLLM key (architecture L9). This cap stops one delegation from growing without bound. An unbounded one could still exhaust a daily budget and take every frontier model away until it resets.
• Too low: delegations fail just short of finishing and their work is thrown away. That is the most expensive way to be wrong, because you paid for the work.
• Too high: a stuck delegation burns tokens and time before failing.
• Deciding: every delegation records the tokens it used and how long it ran (the delegation_completed event, captured for evaluation per spec L9). Set each cap above what successful delegations actually use, with room to spare.

tuning.tools.command_timeout_s — default 120

How long one run_command may run before its whole process group is killed.

• Too low: slow test suites, dependency installs and Docker builds get killed and reported as failures. That also counts toward failure_limit.
• Too high: a command that never exits blocks the turn for the whole timeout, such as a dev server started in the foreground or a test waiting on input. stop waits for a running command to finish too, so this is also the worst-case delay before a stop takes effect.
• Deciding: set it just above the slowest legitimate command in the projects the agent works on.

tuning.tools.max_output_chars — default 4000

How much of a tool's output (command output, file contents, directory listings) the model sees. Longer output keeps its first quarter and last three quarters, with a note in between saying how much was left out. The end gets the larger share because test runners and compilers print their summary of what failed last; the start keeps the exit code.

• This is a context knob. Every tool result stays in the transcript and is re-sent every turn. 4,000 characters is about 1,000 tokens, so 20 turns of maximal output is about 20k tokens, which is already most of a local model's context.
• Too low: the model doesn't see the part of the output that matters.
• Too high: the transcript outgrows the context window sooner. The context budget copes (see below), but by cutting older output harder.

tuning.context — fitting each request into the window

Before any request is sent, the agent checks it against the model's context window and makes it fit. A server doesn't always refuse a prompt that's too long. Ollama silently drops its start, which is where the system prompt and the plan are, so an unfitted request can fail in a way nobody sees.

The window is each model's max_input_tokens in LiteLLM (see Declaring a local model's limits below). A model in use without one stops the agent at startup. From the window come the output ceiling and a safety margin, and the rest is for the prompt. When the prompt is too big, three steps apply in order, stopping at the first that fits:

1. Older tool results are cut to about 600 characters, head and tail. The most recent turns keep their output whole.
2. The oldest turns are dropped, tool calls together with their results. A note at the start tells the model how many events were left out. The log keeps everything; only the request is shortened.
3. If the system prompt and the recent turns still don't fit, the request is not sent. The milestone stops and asks you.

Tokens are estimated from characters (pessimistically, at 3 per token) and the estimate is corrected from the prompt sizes the server reports. The correction only ever makes the estimate larger.

Each role also sends less in the first place: it sees its own work item's transcript (L6), with other roles' work reduced to what it needs (a helper's handback is one report to the driver).

Knob	Default
keep_recent_turns	6	model turns that are never cut or dropped
safety_margin	0.1	fraction of the window held back for estimation error

• keep_recent_turns too low: the model loses sight of what it just did and repeats itself. Too high: less room to cut before step 3 gives up; on a small window, overflows come sooner.
• safety_margin too low: an underestimate reaches the server, and a silently cutting server drops the start of the prompt. Too high: wasted context. Lower it only once the correction from reported usage has settled, which takes a few calls per model.
• Deciding: if milestones stop with "cannot fit", look first at tools.max_output_chars and the model's window, not at these.

Enforced tool calls — supports_tool_choice, declared in LiteLLM

The agent's loops depend on every turn being a well-formed call to a real tool. A model only writes text: a "tool call" is text in a format the model was trained to produce, which the server recognises and turns into a structured call. By default (tool_choice: auto) the server just parses whatever comes out, so prose, broken JSON and calls to invented tools are all possible.

With tool_choice: required, a server that supports it constrains decoding: at every step it rules out any token that couldn't lead to a valid call to one of the offered tools, with arguments matching its schema. The model still chooses the tool and the arguments; it can't produce anything else. That makes it a property of the model server (and sometimes of the model), not something a prompt can buy.

So the agent asks for it only where it is known to work: models whose LiteLLM entry declares supports_tool_choice: true. Measured on the 4070:

Alias	Server	required on a chat prompt	Declare
local-fast (Qwen3-8B-AWQ)	vLLM 0.29	calls finish every time	true
local-coder (Qwen3-14B)	llama.cpp	can't stop; writes until the output ceiling (a known Qwen3-14B bug)	false
claude-haiku-4-5	Anthropic	calls finish every time	true (LiteLLM already says so)

Asking a model that can't honour it can be worse than not asking: the Qwen3-14B turn above costs a full output ceiling of generation, where a plain prose reply would have cost a few hundred tokens.

Where it isn't enforced, the driver's run holds the contract itself:

• A waiting tool is the only way to end a run (finish_milestone, ask_human, and the like); the harness, not the model, then publishes or asks.
• A prose reply is a violation. The model is told once; a second ends the run, and the item waits on you.
• A call to a tool that wasn't offered is answered, not run: the result names the tools there are, so the model can put it right on its next turn.

The capability matrix checks the declaration: a model declared true that answers a required call in prose fails test_supports_tool_choice_is_declared_truthfully.

Declaring a local model's limits

LiteLLM knows commercial models' limits. For a local model, declare them in LiteLLM's config as the model's model_info:

model_list:
  - model_name: local-fast
    litellm_params: {model: openai/qwen3-8b-awq, api_base: http://inference-host:8080/v1}
    model_info:
      max_input_tokens: 32768       # the server's per-request context
      max_output_tokens: 4096       # the most one response may use
      supports_function_calling: true
      supports_tool_choice: true    # only once measured; see above

max_input_tokens is the whole window, prompt plus response. The agent takes the output ceiling out of it. That's how the commercial numbers work too: Claude's prompt and response together must fit in its 200k.

The number to use is what the server allocates per request, not the model's advertised maximum:

• The model's ceiling: max_position_embeddings in config.json, or <arch>.context_length in a GGUF file. Some models reach further with RoPE scaling; the model card says whether that needs switching on.
• llama.cpp / llama-swap: the -c / --ctx-size flag, shown as n_ctx in the load log and in GET /props. With --parallel N each request gets n_ctx / N, so run the agent's models with --parallel 1.
• vLLM: --max-model-len, reported by GET /v1/models.
• Ollama: 4096 unless OLLAMA_CONTEXT_LENGTH or a Modelfile's num_ctx says otherwise. GET /api/ps shows the loaded model's context_length; ollama show shows only the trained maximum.

Then check the declaration against the server:

uv run python scripts/probe_context.py local-fast

It sends one prompt half again as long as the declared window and reports whether the server refused it, cut it (and where), or took it all. A cut below the declaration is the case to fix: prompts between the two would be silently cut. Measured on the Jetson's llama3.2: cut at 4096, matching its declaration.

---

Benchmarking models

The capability matrix asks whether a model can hold the tool contract. The benchmark asks whether it finishes real work: eight canned milestones, each an approved plan over a small seed project, judged by a hidden check the model never sees. Success means the check passes, not that the model said it was done.

Task	What it exercises
write-function	write a small function and its tests
fix-bug	find and fix the bug a failing test exposes, without touching the test
multi-file	thread a new option through several files
long-output	find one failure in test output far longer than max_output_chars
trace-bug	follow a wrong total to its cause in a shared helper, and fix it there rather than at the symptom
refactor-rename	rename a function across a ten-module package, sparing two similar names a blind replace would break
spec-parser	implement a parser from an eight-rule written spec when the plan gives only three examples, with tests
long-milestone	carry out eight dependent steps (model, storage, sorting, CLI) over 25+ turns

The first four were the original set. By September 2026 all seven newer local models passed all of them, so the last four were added to separate models that can hold the tool contract from models that can do a milestone's real work. Their plans stay explicit, as an agreed restatement is; what makes them harder is the work itself: tracing a cause, searching a codebase, reading a spec closely, and holding a plan over many turns.

It runs the loop's own driver exactly as the agent does: the same system prompt, tools, client wiring, context budget, tool_choice handling and your config's tuning knobs. The task's plan is given as the agreed restatement, and finishing the milestone ends the run. Delegation and ask_expert are turned off: an escalation is a failure of the model under test, not a frontier model's success. Until M2.8 it ran the old loop's executor, so runs from before then are against older task versions and show as (old).

Evaluating a new model is one command, once its LiteLLM entry declares its limits:

scripts/evaluate_model.sh qwen3-32b-q4 "vLLM 0.29, AWQ, fp8 KV" 5

It runs the context probe, then the capability matrix, then five runs of every task, and stops at the first step that fails. Or run the steps yourself:

uv run python scripts/benchmark.py run --models A,B --runs 5 --note "how it's served"
uv run python scripts/benchmark.py report [--models A,B] [--tasks T]

Results are kept. Every run is one line in bench/results/, committed with the repo, recording:

• the model's backing name and declared facts;
• your note on how it was served;
• the task's content hash;
• the agent's commit and tuning;
• the outcome, turns, failures, tokens, time and cost.

The report rebuilds the table from every stored run. Benchmarking a new model means running only that model and reading it next to everything before it. Runs against an older version of a task are marked (old). Runs of one model served differently (different note) are kept apart.

Reading it:

• Compare the intervals, not the rates. 3 of 5 (23–88%) and 5 of 5 (57–100%) overlap: five runs don't separate them. Add runs until the intervals stop overlapping, or accept that the models are close.
• Look at why runs failed, not just how many (outcome_note and the counts in each result line). An early shakedown on local-fast failed mostly by escalation: the model was fixing its own mistakes and hit failure_limit: 2 mid-recovery. That says as much about the knob as about the model. Tuning a knob and re-running is a fair comparison, because the tuning is recorded with every run.
• Cost is per run, straight from LiteLLM, so a frontier model as driver can be compared on success rate and dollars.

---

Part 3 — An order to tune in

Upstream choices change what's right downstream, so tune in this order and change one thing at a time.

1. Model and quantization. Start with whether a candidate holds the tool-call contract at all, and whether its server enforces tool_choice:

   uv run pytest -m integration tests/integration/test_model_capability.py \
     --capability-models=alias-a,alias-b
   

   A model that falls back to prose instead of calling tools is disqualified, however good its benchmarks look. Before comparing aliases, check what each one actually points at in LiteLLM (/model/info, the litellm_params.model field). Two aliases believed to be the same model once weren't, and that cost a lot of debugging.

   Make sure LiteLLM has no fallback for a model you are measuring. A fallback answers a failed request with another model, under that model's name, in a response that otherwise looks normal. One made a local model that couldn't call tools at all look like it "held the tool contract 20 of 20": every answer was Claude's. The capability matrix and the probe now check which model answered and fail on a mismatch, but the fallback also quietly turns local work into frontier spending in production.

2. Context length and KV-cache precision on the server. Take the largest context that fits beside the weights with 1–2 GB of headroom. Try an 8-bit KV cache before giving up context.

3. Declare the window in LiteLLM and check it with the probe, then set the model's max_output_tokens there so the prompt keeps most of the window.

4. The benchmark. Run it and look at the evidence: success rate, turns, escalations, and time taken.

5. The loop knobs: failure_limit, max_turns, consult_budget, the delegation caps, then the tool limits, one at a time.

Expect noise. Local model behaviour varies a lot from run to run. One early comparison measured 12 of 20 runs holding the tool contract against 20 of 20, a real difference (p ≈ 0.003). Three runs each could easily have pointed the other way. Before changing a knob because of a result, repeat the run enough times that the result would survive a coin flip.

---

Coming next

Later work that will add knobs, each with its own section here:

• Compaction. A model-written summary replacing old turns, instead of dropping them. The log already has an event for it; it needs a summarizing model and a policy for what is safe to summarize.
• Benchmarking the other roles: the helper, with delegation turned back on for comparisons across frontier tiers.

tuning.loop.bug_grace_s — default 600

How long the restatement of an issue labelled bug waits for a reply before the agent goes ahead on its own (architecture D1). Any reply in the item's thread stops the countdown, and go starts at once.

• Too low: you can't read the restatement before the fix starts, so a misread report costs a wasted milestone.
• Too high: bugs wait as long as features, and the label buys nothing.
• Deciding: about the time it takes you to notice a message and read it when you're around. A wrong start costs only time, since the fix still comes as a PR you review.

tuning.loop.general_idle_s — default 86400

How long a general conversation (one not about a project, architecture L1) stays open without a message. It then closes with a note in its log; a later reply in its thread starts a new conversation linked to it.

• Too low: a conversation you come back to after lunch has to be picked up again from its summary.
• Too high: idle conversations pile up in status and in the queue.

tuning.memory.max_turns — default 10

Model calls the memory pass at a work item's close may make (architecture L6): reading notes, remembering, forgetting, then memory_done. Running out leaves what it wrote and is noted; the close never waits on it.

• Too low: a long item's lessons are cut short.
• Too high: a model that keeps rereading notes costs calls at every close.
• Deciding: a few notes per item is typical; each costs about one call.

tuning.build_vm.retry_s — default 300

How often a build VM that can't be reached is tried again (architecture D3). While it's out of reach, every project on it is blocked; each try is one SSH connection that gives up after 15 seconds.

• Too low: a VM that is down for long is tried constantly, and each try holds up the agent's other duties for up to 15 seconds.
• Too high: work waits longer than it needs to after the VM is back.
• Deciding: about how long a reboot of the VM's host takes. A VM you know is back can be cleared at once with vm ready <project>.

tuning.build_vm.settle_s — default 180

Once an unreachable VM answers again, how long it must keep answering before work goes on: SSH comes up before a VM has finished booting.

• Too low: work starts on a VM still running cloud-init or starting services, and fails in confusing ways.
• Too high: work waits after the VM is ready.
• Deciding: how long the template takes from SSH answering to fully up.

nisse-review's knobs

nisse-review (Part R) has its own configuration (nisse-review.example.yaml). llm.*, tuning.expert.consult_budget and tuning.context mean what they mean for nisse-dev, above.

tuning.review.poll_s — default 60

How often Gitea is asked for PRs that are ready, when nothing is waiting for a review. Each round lists every readable repository's open PRs: a few API calls per repository.

• Too low: load on Gitea for nothing; reviews don't start much sooner.
• Too high: a PR waits that long for its review to start.

tuning.review.max_turns — default 40

A review run's turn cap. Reading files, searching and drafting comments each take a turn; a review that hits the cap posts nothing and counts as a failed try.

• Too low: large PRs fail to be reviewed.
• Too high: a reviewer that wanders spends long on one PR.
• Deciding: look at how many turns real reviews take (the transcript) and leave headroom.

tuning.review.diff_chars — default 60000

How much of the PR's diff goes into the review's starting context; the rest it reads file by file.

• Too low: the reviewer reads files it could have seen in the diff.
• Too high: a large diff fills the window before the review starts.
• Deciding: a fraction of the reviewer's context window (about four characters a token).

tuning.review.attempts — default 2

Failed review runs on one PR, at one head, before the reviewer says on the PR that it couldn't review it, and stops trying until the next push.

tuning.review.ci_wait_s — default 3600

How long a new head's review waits while its checks are pending. A review judges the CI result (R1), and waiting keeps it off a GPU that CI is using (the nisse project's end-to-end tests). Finished checks, passed or failed, end the wait; after this long it reviews anyway and the CI result reads as pending. 0 never waits.

• Too low: reviews run beside CI, slower on a shared GPU, without its result.
• Too high: a stuck runner delays reviews that long.
• Deciding: a little longer than the repository's slowest CI run (the full end-to-end suite takes 25 minutes or more).

tuning.review.ci_grace_s — default 120

How long a new head with no checks at all waits for some to appear: Gitea Actions reports a push's checks a moment after it. Past this, a repository without CI is reviewed at once. 0 reviews such a head at once (the live review test, whose fresh repository has no CI).

• Too low: a review starts before CI has reported, and runs beside it.
• Too high: repositories without CI wait that long for every review.