nisse/docs/Tuning-Guide.md

# Tuning Guide

How to fit a local model onto the inference host, and how to set the agent's
tuning knobs. Part 1 covers the ideas you need to reason about the hardware.
Part 2 covers each knob in `nisse-dev.yaml`. Part 3 gives an order to tune
in.

The knobs live under `tuning:` in the config file (see
`nisse-dev.example.yaml`). They are read at startup, so a change takes effect
on the next restart. Unknown or misspelt keys are startup errors.

---

## Part 1 — How a local model spends VRAM

A model server holding a model in GPU memory spends it on three things:

1. **Weights**, which are fixed for a given model and quantization.
2. **The KV cache**, which grows with context length.
3. **Working memory** for activations and runtime buffers. Budget 1–2 GB of
   headroom.

On a 24 GB card, the whole game is trading (1) against (2).

### Weights and quantization

A model's weights take roughly *parameters × bits per weight ÷ 8* bytes.

| | 16-bit (full) | 8-bit | ~4.5-bit (Q4_K_M, AWQ) |
|---|---|---|---|
| 24B | 48 GB | 24 GB | ~14 GB |
| 32B | 64 GB | 32 GB | ~19 GB |

Quantization stores each weight in fewer bits. Common 4-bit formats come out
nearer 4.5–5 bits per weight once scaling factors are counted, so check the
actual file size. The quality cost is small at 5–6 bits, usually acceptable at
4, and it climbs quickly below that. For agent work the cost shows up first as
**worse tool-call discipline**, not as worse prose. The capability matrix
(Part 3) measures exactly that.

**Mixture-of-experts** models such as Qwen3-30B-A3B are different. Memory is
set by *total* parameters (30B), but speed is set by the parameters *active*
for each token (3B). An MoE model of a given size therefore generates much
faster than a dense model of the same size. It's often weaker than a dense
model of the same total size, but not always for tool use.

### The KV cache, and why context costs memory

As the model reads the prompt, every layer stores a key and a value vector for
every token, so that later tokens can attend to them without recomputing.
That store is the KV cache. Its size per token is:

    2 (key and value) × layers × KV heads × head size × bytes per value

The numbers come from the model's `config.json` (`num_hidden_layers`,
`num_key_value_heads`, `head_dim`). Some worked examples at 16-bit:

| Model | Layers × KV heads × head size | Per token | 32k tokens |
|---|---|---|---|
| Qwen3-32B | 64 × 8 × 128 | 256 KB | ~8 GB |
| Mistral/Devstral Small 24B | 40 × 8 × 128 | 160 KB | ~5 GB |
| Qwen3-30B-A3B (MoE) | 48 × 4 × 128 | 96 KB | ~3 GB |

(Taken from the published configs; recheck against the model card before
relying on them.)

Two consequences:

- **Context length is a memory decision, not a free setting.** The spec's
  estimate (Part H) was about 5 GB left for KV cache beside a 4-bit 32B model.
  At 16-bit, that's only about 20k tokens of context for Qwen3-32B.
- **KV-cache quantization** stores keys and values in 8 bits (or 4). At 8
  bits it halves the cache and is close to lossless. At 4 bits it is
  noticeably worse. Quantizing the cache to 8-bit is usually the first lever
  to pull, since it buys twice the context for almost nothing.

The server allocates the cache for the context length you configure. It
doesn't matter how much of it a request uses. So set the context length on the
server deliberately: it is the ceiling for *prompt plus response* on every
request.

### Prompt processing vs generation

A request has two phases, and they run at very different speeds:

- **Prefill** reads the prompt. It is parallel and fast: thousands of tokens
  per second on a 4090.
- **Decode** writes the response one token at a time. Roughly 30–40 tokens per
  second for a dense 32B at 4-bit, and several times that for a 3B-active MoE.
  Treat these numbers as ballpark; measure your own.

An agent loop re-sends the whole transcript on every turn, which makes the
prompt long and the responses short (a tool call is often under 200 tokens).
So prefill matters more than the headline tokens-per-second suggests.

**Prefix caching** is the fix. If the start of a prompt matches the previous
request, the server reuses the KV cache for it and prefills only the new part.
llama.cpp does this per slot, and vLLM does it automatically. For this
workload it is often the single largest speed win. It works here because the
rendered transcript only grows at the end, so consecutive turns share
everything but their newest events.

### Tokens and characters

For English and code, a token is roughly **4 characters**. That's useful for
converting the character-based knobs below into context cost.

---

## Part 2 — The knobs

For each knob: what it controls, what goes wrong at either extreme, and what
to look at when deciding.

### `llm.max_output_tokens` — default 16384

The ceiling on a single response, for every role. The ceiling actually sent
is the smaller of this and the model's own `max_output_tokens` in LiteLLM, so
a local model can be held to less without affecting the frontier roles.

- **Why it exists:** a response that hits the ceiling is truncated
  mid-sentence, or worse, mid-tool-call, where it can still parse. The agent
  detects truncation and treats it as an error, but a ceiling that's too
  tight still costs a failed turn. Planning with a frontier model was observed
  using over 4,000 output tokens.
- **Too high, on a local model:** the server reserves room for the whole
  response inside the context window. With a 32k context and a 16k ceiling,
  only 16k is left for the prompt. Some servers reject any request where
  prompt plus ceiling exceeds the context.
- **Too low:** truncated plans and tool calls.
- **Deciding:** executor turns are short, so 2–4k is plenty for a local
  executor. Frontier planning wants the large default. Leave this high, and
  set a local model's smaller ceiling as its `max_output_tokens` in LiteLLM.

### `llm.request_timeout_s` — default 600

How long one model request may take before it is abandoned. It applies to
every request, on both routes.

- **Too low:** a slow model's long answer is cut off, and the executor
  escalates work that was going well. At 120 s, the old fixed value, a dense
  27B model on the RTX 4090 timed out writing one module in one turn.
- **Too high:** a hung connection holds the milestone for that long before
  anything notices.
- **Deciding:** this is a safety net, not a budget. `max_output_tokens` and
  `max_turns` bound the work; set this above the slowest single answer you
  expect, which is the model's output ceiling divided by its generation
  speed.

### `tuning.executor.max_turns` — default 24

How many model calls one milestone may make before the executor stops with
`capped` and asks you what to do.

- **Too low:** milestones that were making progress stop short. You'll see
  `capped` outcomes on work that was going well.
- **Too high:** a confused model keeps going. Every turn also adds its tool
  calls and results to the transcript, so a long run spends context as well
  as time.
- **Deciding:** look at how many turns *successful* milestones take, and set
  the cap comfortably above the high end, around 1.5–2× the turns of your
  longer successful runs. A stronger model usually needs fewer turns, not
  more.

### `tuning.executor.failure_limit` — default 2

How many consecutive failed `run_command` calls trigger escalation: the
sub-task is handed to a frontier model as a delegation (spec A12). A
successful command resets the count. Failed file reads don't count, since
reading is looking around, not doing the work.

- **Why it's a count and not a judgment:** small models are poorly calibrated
  about their own competence. Asked "are you stuck?", they say no until far
  too late, or yes constantly. A count of observable failures is a signal
  that doesn't depend on the model's opinion of itself.
- **Too low:** normal iteration escalates. Test-first work *expects* a failing
  test run before the fix, so a limit of 1 hands off routine work and spends
  frontier tokens on it.
- **Too high:** a weak model circles, re-running the same failing command
  with small variations, while the transcript fills up.
- **Deciding:** raise it as local models get stronger. Watch two things
  together. First, how often delegations fix what the local model couldn't
  (escalation is doing its job). Second, how often the local model recovers
  on its own after its second failure (the limit may be too low).

### `tuning.expert.consult_budget` — default 3

How many `ask_expert` calls one milestone may make. `ask_expert` is a single
stateless question to a stronger model, with no access to the repository
(spec A14), for things like "is this API deprecated?".

- **Too low:** the executor can't get unstuck on a knowledge question it
  can't answer from the code.
- **Too high:** a confused executor asks variations of the same question
  instead of doing the work or failing. That's the circling A12 warns about.
- **Deciding:** count consults per milestone, and check whether the answers
  changed what the executor did next. If it regularly exhausts the budget
  without progress, the budget is fine and the model or the plan is the
  problem.

### `tuning.delegation` — hard caps on one delegation

A delegation is a frontier model working a sub-task with full tool access
(spec A13). When a cap is hit, the delegation **fails**, and its branch and
work are discarded, because resuming on a half-edited tree isn't safe.

| Knob | Default | Caps |
|---|---|---|
| `max_turns` | 30 | model calls |
| `max_tokens` | 200000 | input + output tokens, as reported by the provider |
| `wall_clock_s` | 900 | elapsed time |

- **`max_tokens` is a scope limit, not the money limit.** Spending limits live
  on the LiteLLM key (spec A5). This cap stops one delegation from growing
  without bound. An unbounded one could still exhaust a daily budget and take
  every frontier model away until it resets.
- **Too low:** delegations fail just short of finishing and their work is
  thrown away. That is the most expensive way to be wrong, because you paid
  for the work.
- **Too high:** a stuck delegation burns tokens and time before failing.
- **Deciding:** every delegation records the tokens it used and how long it
  ran (the `delegation_completed` event, captured for evaluation per spec
  A16). Set each cap above what successful delegations actually use, with
  room to spare.

### `tuning.tools.command_timeout_s` — default 120

How long one `run_command` may run before its whole process group is killed.

- **Too low:** slow test suites, dependency installs and Docker builds get
  killed and reported as failures. That also counts toward `failure_limit`.
- **Too high:** a command that never exits blocks the turn for the whole
  timeout, such as a dev server started in the foreground or a test waiting on
  input. `stop` waits for a running command to finish too, so this is also the
  worst-case delay before a stop takes effect.
- **Deciding:** set it just above the slowest legitimate command in the
  projects the agent works on.

### `tuning.tools.max_output_chars` — default 4000

How much of a tool's output (command output, file contents, directory
listings) the model sees. Longer output keeps its first quarter and last three
quarters, with a note in between saying how much was left out. The end gets
the larger share because test runners and compilers print their summary of
what failed last; the start keeps the exit code.

- **This is a context knob.** Every tool result stays in the transcript and is
  re-sent every turn. 4,000 characters is about 1,000 tokens, so 20 turns of
  maximal output is about 20k tokens, which is already most of a local
  model's context.
- **Too low:** the model doesn't see the part of the output that matters.
- **Too high:** the transcript outgrows the context window sooner. The
  context budget copes (see below), but by cutting older output harder.

### `tuning.context` — fitting each request into the window

Before any request is sent, the agent checks it against the model's context
window and makes it fit. A server doesn't always refuse a prompt that's too
long. Ollama silently drops its start, which is where the system prompt and
the plan are, so an unfitted request can fail in a way nobody sees.

The window is each model's `max_input_tokens` in LiteLLM (see *Declaring a
local model's limits* below). A model in use without one stops the agent at
startup. From the window come the output ceiling and a safety margin, and
the rest is for the prompt. When the prompt is too big, three steps apply in
order, stopping at the first that fits:

1. **Older tool results are cut** to about 600 characters, head and tail.
   The most recent turns keep their output whole.
2. **The oldest turns are dropped,** tool calls together with their results.
   A note at the start tells the model how many events were left out. The
   log keeps everything; only the request is shortened.
3. **If the system prompt and the recent turns still don't fit,** the
   request is not sent. The milestone stops and asks you.

Tokens are estimated from characters (pessimistically, at 3 per token) and
the estimate is corrected from the prompt sizes the server reports. The
correction only ever makes the estimate larger.

The executor also sends less in the first place: the approved milestone plan
goes in its system prompt, which is never cut, and it sees only what happened
since that plan. The planning conversation and earlier milestones stay in the
log.

| Knob | Default | |
|---|---|---|
| `keep_recent_turns` | 6 | model turns that are never cut or dropped |
| `safety_margin` | 0.1 | fraction of the window held back for estimation error |

- **`keep_recent_turns` too low:** the model loses sight of what it just did
  and repeats itself. **Too high:** less room to cut before step 3 gives up;
  on a small window, overflows come sooner.
- **`safety_margin` too low:** an underestimate reaches the server, and a
  silently cutting server drops the start of the prompt. **Too high:** wasted
  context. Lower it only once the correction from reported usage has settled,
  which takes a few calls per model.
- **Deciding:** if milestones stop with "cannot fit", look first at
  `tools.max_output_chars` and the model's window, not at these.

### Enforced tool calls — `supports_tool_choice`, declared in LiteLLM

The agent's loops depend on every turn being a well-formed call to a real
tool. A model only *writes text*: a "tool call" is text in a format the model
was trained to produce, which the server recognises and turns into a
structured call. By default (`tool_choice: auto`) the server just parses
whatever comes out, so prose, broken JSON and calls to invented tools are all
possible.

With `tool_choice: required`, a server that supports it **constrains
decoding**: at every step it rules out any token that couldn't lead to a
valid call to one of the offered tools, with arguments matching its schema.
The model still chooses the tool and the arguments; it can't produce anything
else. That makes it a property of the **model server** (and sometimes of the
model), not something a prompt can buy.

So the agent asks for it only where it is known to work: models whose LiteLLM
entry declares `supports_tool_choice: true`. Measured on the 4070:

| Alias | Server | `required` on a chat prompt | Declare |
|---|---|---|---|
| `local-fast` (Qwen3-8B-AWQ) | vLLM 0.29 | calls `finish` every time | `true` |
| `local-coder` (Qwen3-14B) | llama.cpp | can't stop; writes until the output ceiling (a known Qwen3-14B bug) | `false` |
| `claude-haiku-4-5` | Anthropic | calls `finish` every time | `true` (LiteLLM already says so) |

Asking a model that can't honour it can be worse than not asking: the
Qwen3-14B turn above costs a full output ceiling of generation, where a plain
prose reply would have cost a few hundred tokens.

Where it isn't enforced, the executor holds the contract itself:

- **`finish` is the only way to end a milestone.** It is refused once if
  nothing has been changed yet.
- **A prose reply is a violation.** The model is told once; a second escalates.
- **A call to a tool that wasn't offered counts as a failure** toward
  `failure_limit`, like a failed command, so an inventing model escalates in a
  couple of turns instead of using up `max_turns`.

The capability matrix checks the declaration: a model declared `true` that
answers a required call in prose fails
`test_supports_tool_choice_is_declared_truthfully`.

### Declaring a local model's limits

LiteLLM knows commercial models' limits. For a local model, declare them in
LiteLLM's config as the model's `model_info`:

```yaml
model_list:
  - model_name: local-fast
    litellm_params: {model: openai/qwen3-8b-awq, api_base: http://inference-host:8080/v1}
    model_info:
      max_input_tokens: 32768       # the server's per-request context
      max_output_tokens: 4096       # the most one response may use
      supports_function_calling: true
      supports_tool_choice: true    # only once measured; see above
```

`max_input_tokens` is the *whole* window, prompt plus response. The agent
takes the output ceiling out of it. That's how the commercial numbers work
too: Claude's prompt and response together must fit in its 200k.

The number to use is what the **server** allocates per request, not the
model's advertised maximum:

- **The model's ceiling:** `max_position_embeddings` in `config.json`, or
  `<arch>.context_length` in a GGUF file. Some models reach further with RoPE
  scaling; the model card says whether that needs switching on.
- **llama.cpp / llama-swap:** the `-c` / `--ctx-size` flag, shown as `n_ctx`
  in the load log and in `GET /props`. With `--parallel N` each request gets
  `n_ctx / N`, so run the agent's models with `--parallel 1`.
- **vLLM:** `--max-model-len`, reported by `GET /v1/models`.
- **Ollama:** 4096 unless `OLLAMA_CONTEXT_LENGTH` or a Modelfile's `num_ctx`
  says otherwise. `GET /api/ps` shows the loaded model's `context_length`;
  `ollama show` shows only the trained maximum.

Then check the declaration against the server:

    uv run python scripts/probe_context.py local-fast

It sends one prompt half again as long as the declared window and reports
whether the server refused it, cut it (and where), or took it all. A cut
below the declaration is the case to fix: prompts between the two would be
silently cut. Measured on the Jetson's `llama3.2`: cut at 4096, matching its
declaration.

---

## Benchmarking models

The capability matrix asks whether a model *can* hold the tool contract. The
benchmark asks whether it *finishes real work*: four canned milestones, each an
approved plan over a small seed project, judged by a hidden check the model
never sees. Success means the check passes, not that the model said it was
done.

| Task | What it exercises |
|---|---|
| `write-function` | write a small function and its tests |
| `fix-bug` | find and fix the bug a failing test exposes, without touching the test |
| `multi-file` | thread a new option through several files |
| `long-output` | find one failure in test output far longer than `max_output_chars` |

It runs the executor exactly as the agent does (the same client wiring,
context budget and `tool_choice` handling, and your config's tuning knobs),
with **delegation and `ask_expert` turned off**. An escalation is a failure of
the model under test, not a frontier model's success.

**Evaluating a new model** is one command, once its LiteLLM entry declares its
limits:

    scripts/evaluate_model.sh qwen3-32b-q4 "vLLM 0.29, AWQ, fp8 KV" 5

It runs the context probe, then the capability matrix, then five runs of
every task, and stops at the first step that fails. Or run the steps
yourself:

    uv run python scripts/benchmark.py run --models A,B --runs 5 --note "how it's served"
    uv run python scripts/benchmark.py report [--models A,B] [--tasks T]

**Results are kept.** Every run is one line in `bench/results/`, committed
with the repo, recording:
- the model's backing name and declared facts;
- your note on how it was served;
- the task's content hash;
- the agent's commit and tuning;
- the outcome, turns, failures, tokens, time and cost.

The report rebuilds the table from every stored run. Benchmarking a new model
means running only that model and reading it next to everything before it.
Runs against an older version of a task are marked `(old)`. Runs of one model
served differently (different note) are kept apart.

**Reading it:**
- **Compare the intervals, not the rates.** 3 of 5 (23–88%) and 5 of 5
  (57–100%) overlap: five runs don't separate them. Add runs until the
  intervals stop overlapping, or accept that the models are close.
- **Look at why runs failed**, not just how many (`outcome_note` and the
  counts in each result line). An early shakedown on `local-fast` failed
  mostly by escalation: the model was fixing its own mistakes and hit
  `failure_limit: 2` mid-recovery. That says as much about the knob as about
  the model. Tuning a knob and re-running is a fair comparison, because the
  tuning is recorded with every run.
- **Cost** is per run, straight from LiteLLM, so a frontier model as executor
  can be compared on success rate *and* dollars.

---

## Part 3 — An order to tune in

Upstream choices change what's right downstream, so tune in this order and
change one thing at a time.

1. **Model and quantization.** Start with whether a candidate holds the
   tool-call contract at all, and whether its server enforces `tool_choice`:

       uv run pytest -m integration tests/integration/test_model_capability.py \
         --capability-models=alias-a,alias-b

   A model that falls back to prose instead of calling tools is disqualified,
   however good its benchmarks look. Before comparing aliases, check what
   each one actually points at in LiteLLM (`/model/info`, the
   `litellm_params.model` field). Two aliases believed to be the same model
   once weren't, and that cost a lot of debugging.

   **Make sure LiteLLM has no fallback for a model you are measuring.** A
   fallback answers a failed request with another model, under that model's
   name, in a response that otherwise looks normal. One made a local model
   that couldn't call tools at all look like it "held the tool contract 20 of
   20": every answer was Claude's. The capability matrix and the probe now
   check which model answered and fail on a mismatch, but the fallback also
   quietly turns local work into frontier spending in production.
2. **Context length and KV-cache precision** on the server. Take the largest
   context that fits beside the weights with 1–2 GB of headroom. Try an 8-bit
   KV cache before giving up context.
3. **Declare the window** in LiteLLM and check it with the probe, then set
   the model's `max_output_tokens` there so the prompt keeps most of the
   window.
4. **The benchmark.** Run it and look at the evidence: success rate, turns,
   escalations, and time taken.
5. **The loop knobs**: `failure_limit`, `max_turns`, `consult_budget`, the
   delegation caps, then the tool limits, one at a time.

**Expect noise.** Local model behaviour varies a lot from run to run. One
early comparison measured 12 of 20 runs holding the tool contract against 20
of 20, a real difference (p ≈ 0.003). Three runs each could easily have
pointed the other way. Before changing a knob because of a result, repeat the
run enough times that the result would survive a coin flip.

---

## Coming next

Later work that will add knobs, each with its own section here:

- **Compaction.** A model-written summary replacing old turns, instead of
  dropping them. The log already has an event for it; it needs a summarizing
  model and a policy for what is safe to summarize.
- **Benchmarking the other roles:** planner and delegate, with delegation
  turned back on for comparisons across frontier tiers.