Add an executor benchmark that keeps its results

Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.

- Four canned milestones under nisse_dev/bench/tasks: write a function,
  fix a bug a failing test exposes, thread an option through several
  files, and find one failure in output far past max_output_chars. Each
  is an approved plan over a seed project, judged by a hidden check the
  model never sees; a reference solution proves each is solvable and
  that its check tells the difference
- Runs use the agent's own client wiring, now shared as
  make_llm_clients, with delegation and ask_expert off: an escalation is
  a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
  the model's backing name and declared facts, a note on how it was
  served, the task's content hash, the agent's commit and tuning, the
  outcome, turns, misbehaviour counts, context trims, tokens, time and
  LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
  95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
  a new model through the context probe, the capability matrix and the
  benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
  requests it had to trim
- The tuning guide gains "Benchmarking models"

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

tests/integration/test_benchmark_live.py

@@ -0,0 +1,31 @@
+"""One live benchmark run: the simplest task, through the agent's own
+client wiring, with the test config's executor."""
+
+from pathlib import Path
+
+import pytest
+
+from nisse_dev.agent import make_llm_clients
+from nisse_dev.bench.runner import run_task
+from nisse_dev.bench.tasks import load_tasks
+from nisse_dev.catalog import ModelCatalog
+from nisse_dev.config import AgentConfig
+
+pytestmark = pytest.mark.integration
+
+
+def test_the_simplest_task_passes_live(
+    live_config: AgentConfig, tmp_path: Path
+) -> None:
+    catalog = ModelCatalog.discover(live_config.llm_base_url, live_config.llm_token)
+    normalized, _ = make_llm_clients(live_config, catalog)
+    facts = catalog.facts(live_config.roles.executor)
+    [task] = load_tasks(["write-function"])
+
+    record = run_task(task, facts, normalized, live_config.tuning, workdir=tmp_path)
+
+    assert record.fallbacks == []
+    assert record.success, (
+        f"{record.outcome}: {record.outcome_note}\n{record.check_output}"
+    )
+    assert record.cost >= 0