Add an executor benchmark that keeps its results

Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.

- Four canned milestones under nisse_dev/bench/tasks: write a function,
  fix a bug a failing test exposes, thread an option through several
  files, and find one failure in output far past max_output_chars. Each
  is an approved plan over a seed project, judged by a hidden check the
  model never sees; a reference solution proves each is solvable and
  that its check tells the difference
- Runs use the agent's own client wiring, now shared as
  make_llm_clients, with delegation and ask_expert off: an escalation is
  a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
  the model's backing name and declared facts, a note on how it was
  served, the task's content hash, the agent's commit and tuning, the
  outcome, turns, misbehaviour counts, context trims, tokens, time and
  LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
  95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
  a new model through the context probe, the capability matrix and the
  benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
  requests it had to trim
- The tuning guide gains "Benchmarking models"

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

nisse-dev/src/nisse_dev/llm.py

@@ -143,6 +143,7 @@ class LiteLLMClient:
         # protocol stays the same for every implementation.
         self.last_usage: dict[str, int] = {}
         self.last_model: str | None = None
+        self.last_cost: float = 0.0
 
     def complete(
         self,
@@ -171,6 +172,8 @@ class LiteLLMClient:
             raise LLMError(
                 f"LLM request returned {response.status_code}: {response.text[:500]}"
             )
+        # LiteLLM prices every call; kept so a caller can meter spending.
+        self.last_cost = _response_cost(response)
         return self._parse(response.json(), requested_model=model)
 
     def _parse(
@@ -263,6 +266,7 @@ class AnthropicPassthroughClient:
         self._sleep = sleep
         self.last_usage: dict[str, int] = {}
         self.last_model: str | None = None
+        self.last_cost: float = 0.0
 
     def complete(
         self,
@@ -291,6 +295,8 @@ class AnthropicPassthroughClient:
             raise LLMError(
                 f"passthrough returned {response.status_code}: {response.text[:500]}"
             )
+        # LiteLLM prices every call; kept so a caller can meter spending.
+        self.last_cost = _response_cost(response)
         return self._parse(response.json(), requested_model=model)
 
     def _parse(
@@ -358,6 +364,13 @@ def _post_with_rate_limit_retry(
 # Model families we have a native renderer adapter for. A13: delegations take
 # the vendor's passthrough route because a normalising middlebox is lossy
 # exactly where a long-running delegation lives.
+def _response_cost(response: httpx.Response) -> float:
+    try:
+        return float(response.headers.get("x-litellm-response-cost") or 0.0)
+    except ValueError:
+        return 0.0
+
+
 def _raise_for_upstream(response: httpx.Response) -> None:
     """Classify a non-200 as upstream trouble where it plainly is.