Add an executor benchmark that keeps its results

Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.

- Four canned milestones under nisse_dev/bench/tasks: write a function,
  fix a bug a failing test exposes, thread an option through several
  files, and find one failure in output far past max_output_chars. Each
  is an approved plan over a seed project, judged by a hidden check the
  model never sees; a reference solution proves each is solvable and
  that its check tells the difference
- Runs use the agent's own client wiring, now shared as
  make_llm_clients, with delegation and ask_expert off: an escalation is
  a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
  the model's backing name and declared facts, a note on how it was
  served, the task's content hash, the agent's commit and tuning, the
  outcome, turns, misbehaviour counts, context trims, tokens, time and
  LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
  95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
  a new model through the context probe, the capability matrix and the
  benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
  requests it had to trim
- The tuning guide gains "Benchmarking models"

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

scripts/evaluate_model.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Evaluate a model for the executor role, from a declared LiteLLM alias to a
+# benchmark report next to every model measured before it.
+#
+#   scripts/evaluate_model.sh ALIAS "how it is served" [RUNS] [--config PATH]
+#
+# 1. The context probe: does the server's window match what LiteLLM declares?
+# 2. The capability matrix: does it hold the tool contract, and is its
+#    supports_tool_choice declaration true?
+# 3. The benchmark: canned milestones, RUNS times each (default 5).
+# Stops at the first step that fails; there is no point benchmarking a model
+# that can't hold the tool contract.
+set -euo pipefail
+cd "$(dirname "$0")/.."
+
+alias="${1:?usage: evaluate_model.sh ALIAS NOTE [RUNS] [--config PATH]}"
+note="${2:?give a note on how the model is served, e.g. \"vLLM 0.29, AWQ\"}"
+runs="${3:-5}"
+shift $(( $# < 3 ? $# : 3 ))
+config=("$@")
+
+echo "== 1/3 context probe"
+uv run python scripts/probe_context.py "$alias" "${config[@]}"
+
+echo "== 2/3 capability matrix"
+uv run pytest -m integration -q -rs tests/integration/test_model_capability.py \
+    --capability-models="$alias"
+
+echo "== 3/3 benchmark ($runs runs per task)"
+uv run python scripts/benchmark.py run --models "$alias" --runs "$runs" \
+    --note "$note" "${config[@]}"