Benchmark the finalists on all eight tasks, and archive older results
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m12s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m12s
Five runs per task at 320 W with max_turns 40, after the fixes in #50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b, devstral-small-2-24b and mimo-v2.6-9b; the harder four also for claude-haiku-4-5 as the ceiling, and two calibration runs each for local-fast as the floor. Runs failed by the proxy's rate limiter were set aside, not stored. Earlier results move to bench/results/archive/, which the report does not read: they were made at other power caps, at max_turns 24, or on the RTX 4070, and are kept as the record. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
a891932739
commit
4b6c48cc96
22 files changed
+305
-60
No files matched your search
@@ -0,0 +1,17 @@
|
||||
# Archived benchmark results
|
||||
|
||||
Runs set aside on 2026-09-27. They are kept as the record of how the local
|
||||
model choice was made, but the report no longer reads them: `benchmark.py
|
||||
report` only looks at `bench/results/*.jsonl`, not this folder.
|
||||
|
||||
- **2026-09-25, `local-coder` and `local-fast`:** qwen3-14b and qwen3-8b-awq
|
||||
on the original RTX 4070 box. Both models were later dropped.
|
||||
- **2026-09-27, seven newer models and `local-fast` on the RTX 4090 (PR #49):**
|
||||
every newer model passed all four original tasks 20/20, which is why four
|
||||
harder tasks were added. Pass rates are still valid; timings are not
|
||||
comparable with later runs, which were made under a 320 W power cap rather
|
||||
than 450 W or 350 W, with `max_turns` 40 rather than 24.
|
||||
|
||||
Later results were made after the fixes in PRs #50–#52: rate-limit waits
|
||||
recorded, one assistant message per turn, a 600 s request timeout, and tuning
|
||||
kept apart in the report.
|
||||
Reference in new issue
Block a user