Benchmark the finalists on all eight tasks, and archive older results #55

2026-09-28T03:15:04Z

dev-agent commented

2026-09-28 03:15:04 +00:00

Results only: no code changes. Five runs per task, on all eight tasks, for the five local finalists.

Setup: RTX 4090 at a 320 W cap, max_turns 40, after the fixes in #50–#52 (rate-limit waits recorded, one assistant message per turn, 600 s request timeout, tuning kept apart in the report). The GPU stage ran 00:56–03:14 UTC on 2026-09-28. claude-haiku-4-5 is the ceiling on the four new tasks; local-fast is the floor, with 2 calibration runs.

Results

Model	fix-bug	long-output	multi-file	write-function	trace-bug	refactor-rename	spec-parser	long-milestone	New four	All eight	Min/run (new four)
qwen3.8-27b	5/5	5/5	5/5	5/5	5/5	5/5	3/5	4/5	17/20 (64–95%)	37/40	3.9
qwen3.6-35b-a3b	5/5	5/5	5/5	5/5	5/5	4/5	1/5	2/5	12/20 (39–78%)	32/40	0.9
gemma-4-26b-a4b	5/5	5/5	5/5	5/5	4/5	4/5	0/5	2/5	10/20 (30–70%)	30/40	1.3
devstral-small-2-24b	5/5	3/5	5/5	5/5	5/5	1/5	0/5	0/5	6/20 (15–52%)	24/40	0.9
mimo-v2.6-9b	5/5	5/5	5/5	5/5	4/5	1/5	0/5	0/5	5/20 (11–47%)	25/40	1.2
claude-haiku-4-5 (ceiling)	–	–	–	–	5/5	5/5	2/5	3/5	15/20 (53–89%)	–	0.7
local-fast (floor)	–	–	–	–	0/2	0/2	0/2	0/2	0/8	–	0.3

Percentages in brackets are 95% intervals.

Reading it

qwen3.8-27b is the strongest executor, and it beats the Haiku ceiling: 17/20 against 15/20 on the new four. The catch is speed: about 4 minutes per run on the new tasks, 3–4× the MoE models, because it reasons at length before acting.
qwen3.6-35b-a3b and gemma-4-26b-a4b are close to each other. 12/20 against 10/20, and their intervals overlap almost entirely. Both are fast. qwen3.6's old failure mode, replying in prose instead of calling the completion tool, came up once in 40 runs.
Devstral and mimo fall behind on the harder tasks.
spec-parser is the hardest task. Only qwen3.8-27b (3/5) and Haiku (2/5) pass it regularly. The usual failure is an escalation after two failing test runs in a row while fixing its own mistakes, which is failure_limit: 2 at work.

What this means for one local model for every role (spec Part H)

The executor ranking is clear, but the choice isn't settled yet. The Home Assistant role cares about latency, and there the fast MoE models (qwen3.6-35b-a3b, gemma-4-26b-a4b) have the advantage over qwen3.8-27b. The ops evaluation decides which trade-off wins.

Failures caused by the setup, not the model

Rate-limited runs were set aside, not stored: 7 Haiku runs and 1 calibration run. They hit a leak in LiteLLM's token limiter, which stuck until LiteLLM was restarted. The key's token and request limits have since been removed.
One qwen3.8-27b spec-parser run is stored as a failure. It hit the 600 s request timeout in a single answer. At about 42 tokens/s, even a maximum-length answer (16,384 tokens) takes about 6.5 minutes, so an answer still running at 10 minutes was running past its output ceiling. That's a runaway, not too short a timeout. Whether the ceiling is enforced on this route is worth a separate check.

Archive

The older results move to bench/results/archive/, with a README. The report doesn't read that folder. Those runs were made at other power caps (450 W, 350 W), at max_turns 24, or on the RTX 4070. Their pass rates are still valid, but their timings don't compare with these runs.

🤖 Generated with Claude Code

Results only: no code changes. Five runs per task, on all eight tasks, for the five local finalists. **Setup:** RTX 4090 at a 320 W cap, `max_turns` 40, after the fixes in #50–#52 (rate-limit waits recorded, one assistant message per turn, 600 s request timeout, tuning kept apart in the report). The GPU stage ran 00:56–03:14 UTC on 2026-09-28. `claude-haiku-4-5` is the ceiling on the four new tasks; `local-fast` is the floor, with 2 calibration runs. ## Results | Model | fix-bug | long-output | multi-file | write-function | trace-bug | refactor-rename | spec-parser | long-milestone | New four | All eight | Min/run (new four) | |---|---|---|---|---|---|---|---|---|---|---|---| | **qwen3.8-27b** | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 3/5 | 4/5 | **17/20** (64–95%) | **37/40** | 3.9 | | **qwen3.6-35b-a3b** | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 4/5 | 1/5 | 2/5 | 12/20 (39–78%) | 32/40 | **0.9** | | **gemma-4-26b-a4b** | 5/5 | 5/5 | 5/5 | 5/5 | 4/5 | 4/5 | 0/5 | 2/5 | 10/20 (30–70%) | 30/40 | 1.3 | | devstral-small-2-24b | 5/5 | 3/5 | 5/5 | 5/5 | 5/5 | 1/5 | 0/5 | 0/5 | 6/20 (15–52%) | 24/40 | 0.9 | | mimo-v2.6-9b | 5/5 | 5/5 | 5/5 | 5/5 | 4/5 | 1/5 | 0/5 | 0/5 | 5/20 (11–47%) | 25/40 | 1.2 | | claude-haiku-4-5 (ceiling) | – | – | – | – | 5/5 | 5/5 | 2/5 | 3/5 | 15/20 (53–89%) | – | 0.7 | | local-fast (floor) | – | – | – | – | 0/2 | 0/2 | 0/2 | 0/2 | 0/8 | – | 0.3 | Percentages in brackets are 95% intervals. ## Reading it - **qwen3.8-27b is the strongest executor, and it beats the Haiku ceiling:** 17/20 against 15/20 on the new four. The catch is speed: about 4 minutes per run on the new tasks, 3–4× the MoE models, because it reasons at length before acting. - **qwen3.6-35b-a3b and gemma-4-26b-a4b are close to each other.** 12/20 against 10/20, and their intervals overlap almost entirely. Both are fast. qwen3.6's old failure mode, replying in prose instead of calling the completion tool, came up once in 40 runs. - **Devstral and mimo fall behind** on the harder tasks. - **`spec-parser` is the hardest task.** Only qwen3.8-27b (3/5) and Haiku (2/5) pass it regularly. The usual failure is an escalation after two failing test runs in a row while fixing its own mistakes, which is `failure_limit: 2` at work. ## What this means for one local model for every role (spec Part H) The executor ranking is clear, but the choice isn't settled yet. The Home Assistant role cares about **latency**, and there the fast MoE models (qwen3.6-35b-a3b, gemma-4-26b-a4b) have the advantage over qwen3.8-27b. The ops evaluation decides which trade-off wins. ## Failures caused by the setup, not the model - **Rate-limited runs were set aside, not stored:** 7 Haiku runs and 1 calibration run. They hit a leak in LiteLLM's token limiter, which stuck until LiteLLM was restarted. The key's token and request limits have since been removed. - **One qwen3.8-27b `spec-parser` run is stored as a failure.** It hit the 600 s request timeout in a single answer. At about 42 tokens/s, even a maximum-length answer (16,384 tokens) takes about 6.5 minutes, so an answer still running at 10 minutes was running past its output ceiling. That's a runaway, not too short a timeout. Whether the ceiling is enforced on this route is worth a separate check. ## Archive The older results move to `bench/results/archive/`, with a README. The report doesn't read that folder. Those runs were made at other power caps (450 W, 350 W), at `max_turns` 24, or on the RTX 4070. Their pass rates are still valid, but their timings don't compare with these runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-28 03:15:04 +00:00

Benchmark the finalists on all eight tasks, and archive older results

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m12s

Details

4b6c48cc96

Five runs per task at 320 W with max_turns 40, after the fixes in
#50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b,
devstral-small-2-24b and mimo-v2.6-9b; the harder four also for
claude-haiku-4-5 as the ceiling, and two calibration runs each for
local-fast as the floor. Runs failed by the proxy's rate limiter were
set aside, not stored.

Earlier results move to bench/results/archive/, which the report does
not read: they were made at other power caps, at max_turns 24, or on
the RTX 4070, and are kept as the record.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dongar approved these changes

2026-09-28 12:41:45 +00:00

dongar merged commit dfa08d4900 into main

2026-09-28 12:41:49 +00:00

dongar deleted branch bench-results-harder-tasks

2026-09-28 12:41:49 +00:00

dev-agent referenced this pull request

2026-09-29 14:18:50 +00:00

Use qwen3.8-27b as the example executor #56

dev-agent referenced this issue from a commit

2026-09-29 14:18:52 +00:00

Use qwen3.8-27b as the example executor