Add an executor benchmark that keeps its results

Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.

- Four canned milestones under nisse_dev/bench/tasks: write a function,
  fix a bug a failing test exposes, thread an option through several
  files, and find one failure in output far past max_output_chars. Each
  is an approved plan over a seed project, judged by a hidden check the
  model never sees; a reference solution proves each is solvable and
  that its check tells the difference
- Runs use the agent's own client wiring, now shared as
  make_llm_clients, with delegation and ask_expert off: an escalation is
  a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
  the model's backing name and declared facts, a note on how it was
  served, the task's content hash, the agent's commit and tuning, the
  outcome, turns, misbehaviour counts, context trims, tokens, time and
  LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
  95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
  a new model through the context probe, the capability matrix and the
  benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
  requests it had to trim
- The tuning guide gains "Benchmarking models"

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/Tuning-Guide.md

@@ -370,6 +370,66 @@ declaration.
 
 ---
 
+## Benchmarking models
+
+The capability matrix asks whether a model *can* hold the tool contract. The
+benchmark asks whether it *finishes real work*: four canned milestones, each an
+approved plan over a small seed project, judged by a hidden check the model
+never sees. Success means the check passes, not that the model said it was
+done.
+
+| Task | What it exercises |
+|---|---|
+| `write-function` | write a small function and its tests |
+| `fix-bug` | find and fix the bug a failing test exposes, without touching the test |
+| `multi-file` | thread a new option through several files |
+| `long-output` | find one failure in test output far longer than `max_output_chars` |
+
+It runs the executor exactly as the agent does (the same client wiring,
+context budget and `tool_choice` handling, and your config's tuning knobs),
+with **delegation and `ask_expert` turned off**. An escalation is a failure of
+the model under test, not a frontier model's success.
+
+**Evaluating a new model** is one command, once its LiteLLM entry declares its
+limits:
+
+    scripts/evaluate_model.sh qwen3-32b-q4 "vLLM 0.29, AWQ, fp8 KV" 5
+
+It runs the context probe, then the capability matrix, then five runs of
+every task, and stops at the first step that fails. Or run the steps
+yourself:
+
+    uv run python scripts/benchmark.py run --models A,B --runs 5 --note "how it's served"
+    uv run python scripts/benchmark.py report [--models A,B] [--tasks T]
+
+**Results are kept.** Every run is one line in `bench/results/`, committed
+with the repo, recording:
+- the model's backing name and declared facts;
+- your note on how it was served;
+- the task's content hash;
+- the agent's commit and tuning;
+- the outcome, turns, failures, tokens, time and cost.
+
+The report rebuilds the table from every stored run. Benchmarking a new model
+means running only that model and reading it next to everything before it.
+Runs against an older version of a task are marked `(old)`. Runs of one model
+served differently (different note) are kept apart.
+
+**Reading it:**
+- **Compare the intervals, not the rates.** 3 of 5 (23–88%) and 5 of 5
+  (57–100%) overlap: five runs don't separate them. Add runs until the
+  intervals stop overlapping, or accept that the models are close.
+- **Look at why runs failed**, not just how many (`outcome_note` and the
+  counts in each result line). An early shakedown on `local-fast` failed
+  mostly by escalation: the model was fixing its own mistakes and hit
+  `failure_limit: 2` mid-recovery. That says as much about the knob as about
+  the model. Tuning a knob and re-running is a fair comparison, because the
+  tuning is recorded with every run.
+- **Cost** is per run, straight from LiteLLM, so a frontier model as executor
+  can be compared on success rate *and* dollars.
+
+---
+
 ## Part 3 — An order to tune in
 
 Upstream choices change what's right downstream, so tune in this order and
@@ -400,8 +460,8 @@ change one thing at a time.
 3. **Declare the window** in LiteLLM and check it with the probe, then set
    the model's `max_output_tokens` there so the prompt keeps most of the
    window.
-4. **Real milestones.** Run representative work and look at the evidence:
-   turns used, escalations, delegation outcomes, and time taken.
+4. **The benchmark.** Run it and look at the evidence: success rate, turns,
+   escalations, and time taken.
 5. **The loop knobs**: `failure_limit`, `max_turns`, `consult_budget`, the
    delegation caps, then the tool limits, one at a time.
 
@@ -415,11 +475,10 @@ run enough times that the result would survive a coin flip.
 
 ## Coming next
 
-The next changes in the 4090-readiness work add knobs, and this guide gains a
-section for each:
+Later work that will add knobs, each with its own section here:
 
 - **Compaction.** A model-written summary replacing old turns, instead of
   dropping them. The log already has an event for it; it needs a summarizing
   model and a policy for what is safe to summarize.
-- **A benchmark script** that runs the capability matrix and representative
-  milestones across candidate models and prints a comparison table.
+- **Benchmarking the other roles:** planner and delegate, with delegation
+  turned back on for comparisons across frontier tiers.