Five runs per task at 320 W with max_turns 40, after the fixes in
#50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b,
devstral-small-2-24b and mimo-v2.6-9b; the harder four also for
claude-haiku-4-5 as the ceiling, and two calibration runs each for
local-fast as the floor. Runs failed by the proxy's rate limiter were
set aside, not stored.
Earlier results move to bench/results/archive/, which the report does
not read: they were made at other power caps, at max_turns 24, or on
the RTX 4070, and are kept as the record.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Five runs per task for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-31b,
gemma-4-26b-a4b, gemma-4-12b, mimo-v2.6-9b and granite-4.2-8b, all
served by llama.cpp b11206 through llama-swap. Every one passed 20 of
20. local-fast (vLLM 0.30.0) passed 3 of 20.
Runs were made under a 350 W power cap, recorded in each note, after
the uncapped card throttled for heat. granite-4.2-8b also has one
uncapped set, kept apart by its note.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The 4070's two local models with default tuning, as the baseline the
4090's candidates will be compared with. Every one of the 22 failures
is an escalation from failure_limit: 2; there were no prose turns,
invented tools, context trims, or finished-but-wrong results.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.
- Four canned milestones under nisse_dev/bench/tasks: write a function,
fix a bug a failing test exposes, thread an option through several
files, and find one failure in output far past max_output_chars. Each
is an approved plan over a seed project, judged by a hidden check the
model never sees; a reference solution proves each is solvable and
that its check tells the difference
- Runs use the agent's own client wiring, now shared as
make_llm_clients, with delegation and ask_expert off: an escalation is
a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
the model's backing name and declared facts, a note on how it was
served, the task's content hash, the agent's commit and tuning, the
outcome, turns, misbehaviour counts, context trims, tokens, time and
LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
a new model through the context probe, the capability matrix and the
benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
requests it had to trim
- The tuning guide gains "Benchmarking models"
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dongarandClaude Opus 5.5