Add an executor benchmark that keeps its results

Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.

- Four canned milestones under nisse_dev/bench/tasks: write a function,
  fix a bug a failing test exposes, thread an option through several
  files, and find one failure in output far past max_output_chars. Each
  is an approved plan over a seed project, judged by a hidden check the
  model never sees; a reference solution proves each is solvable and
  that its check tells the difference
- Runs use the agent's own client wiring, now shared as
  make_llm_clients, with delegation and ask_expert off: an escalation is
  a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
  the model's backing name and declared facts, a note on how it was
  served, the task's content hash, the agent's commit and tuning, the
  outcome, turns, misbehaviour counts, context trims, tokens, time and
  LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
  95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
  a new model through the context probe, the capability matrix and the
  benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
  requests it had to trim
- The tuning guide gains "Benchmarking models"

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

nisse-dev/src/nisse_dev/context.py

@@ -230,6 +230,7 @@ class BudgetedClient:
         # Per model: how much the character estimate undercounted, as learned
         # from reported usage. Starts at 1 and only grows.
         self._factors: dict[str, float] = {}
+        self.trimmed_requests = 0
 
     @property
     def last_usage(self) -> dict[str, int]:
@@ -239,6 +240,10 @@ class BudgetedClient:
     def last_model(self) -> str | None:
         return getattr(self._inner, "last_model", None)
 
+    @property
+    def last_cost(self) -> float:
+        return float(getattr(self._inner, "last_cost", 0.0) or 0.0)
+
     def budget_for(self, model: str) -> Budget:
         window, model_max_output = self._limits_for(model)
         ceiling = self._max_output_tokens
@@ -261,6 +266,10 @@ class BudgetedClient:
         budget = self.budget_for(model)
         factor = self._factors.get(model, 1.0)
         fitted = fit(events, system, tools, budget, factor)
+        # Counted so the benchmark can report how often a model's window was
+        # the constraint: any request that had to be cut down or shortened.
+        if fitted != list(events):
+            self.trimmed_requests += 1
         produced = self._inner.complete(
             fitted,
             tools,