dongarandClaude Opus 5.5
4b6c48cc96
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m12s
Five runs per task at 320 W with max_turns 40, after the fixes in #50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b, devstral-small-2-24b and mimo-v2.6-9b; the harder four also for claude-haiku-4-5 as the ceiling, and two calibration runs each for local-fast as the floor. Runs failed by the proxy's rate limiter were set aside, not stored. Earlier results move to bench/results/archive/, which the report does not read: they were made at other power caps, at max_turns 24, or on the RTX 4070, and are kept as the record. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
18 lines
928 B
Markdown
18 lines
928 B
Markdown
# Archived benchmark results
|
||
|
||
Runs set aside on 2026-09-27. They are kept as the record of how the local
|
||
model choice was made, but the report no longer reads them: `benchmark.py
|
||
report` only looks at `bench/results/*.jsonl`, not this folder.
|
||
|
||
- **2026-09-25, `local-coder` and `local-fast`:** qwen3-14b and qwen3-8b-awq
|
||
on the original RTX 4070 box. Both models were later dropped.
|
||
- **2026-09-27, seven newer models and `local-fast` on the RTX 4090 (PR #49):**
|
||
every newer model passed all four original tasks 20/20, which is why four
|
||
harder tasks were added. Pass rates are still valid; timings are not
|
||
comparable with later runs, which were made under a 320 W power cap rather
|
||
than 450 W or 350 W, with `max_turns` 40 rather than 24.
|
||
|
||
Later results were made after the fixes in PRs #50–#52: rate-limit waits
|
||
recorded, one assistant message per turn, a 600 s request timeout, and tuning
|
||
kept apart in the report.
|