Benchmark the finalists on all eight tasks, and archive older results
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m12s

Five runs per task at 320 W with max_turns 40, after the fixes in
#50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b,
devstral-small-2-24b and mimo-v2.6-9b; the harder four also for
claude-haiku-4-5 as the ceiling, and two calibration runs each for
local-fast as the floor. Runs failed by the proxy's rate limiter were
set aside, not stored.

Earlier results move to bench/results/archive/, which the report does
not read: they were made at other power caps, at max_turns 24, or on
the RTX 4070, and are kept as the record.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-09-28 03:14:40 +00:00
1 parent a891932739
commit 4b6c48cc96
22 files changed
+305 -60

No files matched your search

+17
View File
@@ -0,0 +1,17 @@
# Archived benchmark results
Runs set aside on 2026-09-27. They are kept as the record of how the local
model choice was made, but the report no longer reads them: `benchmark.py
report` only looks at `bench/results/*.jsonl`, not this folder.
- **2026-09-25, `local-coder` and `local-fast`:** qwen3-14b and qwen3-8b-awq
on the original RTX 4070 box. Both models were later dropped.
- **2026-09-27, seven newer models and `local-fast` on the RTX 4090 (PR #49):**
every newer model passed all four original tasks 20/20, which is why four
harder tasks were added. Pass rates are still valid; timings are not
comparable with later runs, which were made under a 320 W power cap rather
than 450 W or 350 W, with `max_turns` 40 rather than 24.
Later results were made after the fixes in PRs #50–#52: rate-limit waits
recorded, one assistant message per turn, a 600 s request timeout, and tuning
kept apart in the report.