Add an executor benchmark that keeps its results

Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.

- Four canned milestones under nisse_dev/bench/tasks: write a function,
  fix a bug a failing test exposes, thread an option through several
  files, and find one failure in output far past max_output_chars. Each
  is an approved plan over a seed project, judged by a hidden check the
  model never sees; a reference solution proves each is solvable and
  that its check tells the difference
- Runs use the agent's own client wiring, now shared as
  make_llm_clients, with delegation and ask_expert off: an escalation is
  a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
  the model's backing name and declared facts, a note on how it was
  served, the task's content hash, the agent's commit and tuning, the
  outcome, turns, misbehaviour counts, context trims, tokens, time and
  LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
  95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
  a new model through the context probe, the capability matrix and the
  benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
  requests it had to trim
- The tuning guide gains "Benchmarking models"

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

nisse-dev/src/nisse_dev/bench/tasks/write-function/hidden/test_hidden.py

@@ -0,0 +1,19 @@
+import pytest
+from slugify import slugify
+
+
+@pytest.mark.parametrize(
+    ("text", "expected"),
+    [
+        ("Hello, World!", "hello-world"),
+        ("  Nisse  Agent  ", "nisse-agent"),
+        ("a--b", "a-b"),
+        ("", ""),
+        ("Python 3.12 Release", "python-3-12-release"),
+        ("--leading and trailing--", "leading-and-trailing"),
+        ("MiXeD CaSe", "mixed-case"),
+        ("!!!", ""),
+    ],
+)
+def test_slugify(text: str, expected: str) -> None:
+    assert slugify(text) == expected

nisse-dev/src/nisse_dev/bench/tasks/write-function/seed/README.md

@@ -0,0 +1 @@
+A small utilities project.

nisse-dev/src/nisse_dev/bench/tasks/write-function/solution/slugify.py

@@ -0,0 +1,5 @@
+import re
+
+
+def slugify(text: str) -> str:
+    return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")

nisse-dev/src/nisse_dev/bench/tasks/write-function/task.yaml

@@ -0,0 +1,13 @@
+title: Write a small function and its tests
+check: python -m pytest -q .bench_check
+plan: |
+  Milestone 1: add a slugify function.
+
+  1. Create `slugify.py` in the workspace root, defining `slugify(text: str) -> str`.
+  2. It lowercases the text, replaces every run of characters that are not a-z or
+     0-9 with a single hyphen, and strips hyphens from both ends. Examples:
+     "Hello, World!" -> "hello-world"; "  Nisse  Agent  " -> "nisse-agent";
+     "a--b" -> "a-b"; "" -> "".
+  3. Create `test_slugify.py` with pytest tests covering the four examples.
+  4. Run `python -m pytest -q test_slugify.py`, and fix anything until it passes.
+  5. Call finish.