Add four harder benchmark tasks
All seven newer local models passed the original four tasks 20 of 20, so the benchmark no longer told them apart. The new tasks keep plans explicit, as the planner is told to write them, and make the work itself harder: - trace-bug: a wrong multi-currency total whose cause is a shared rate helper; discounts and tax go through it too, so a fix at the call site fails the check. - refactor-rename: rename load_config across a ten-module package, leaving load_config_defaults and reload_config alone. - spec-parser: implement parse_duration from an eight-rule SPEC.md given three examples, and write tests for every rule. - long-milestone: eight dependent steps adding due dates to a task tracker, from the model through storage versions to the CLI. Self-tests prove that a call-site fix, a blind text replace and a partial rename all fail their checks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f2c48d8b16
commit
25fd57351d
67 files changed
+2115
-2
No files matched your search
+12
-1
@@ -393,7 +393,7 @@ declaration.
|
||||
## Benchmarking models
|
||||
|
||||
The capability matrix asks whether a model *can* hold the tool contract. The
|
||||
benchmark asks whether it *finishes real work*: four canned milestones, each an
|
||||
benchmark asks whether it *finishes real work*: eight canned milestones, each an
|
||||
approved plan over a small seed project, judged by a hidden check the model
|
||||
never sees. Success means the check passes, not that the model said it was
|
||||
done.
|
||||
@@ -404,6 +404,17 @@ done.
|
||||
| `fix-bug` | find and fix the bug a failing test exposes, without touching the test |
|
||||
| `multi-file` | thread a new option through several files |
|
||||
| `long-output` | find one failure in test output far longer than `max_output_chars` |
|
||||
| `trace-bug` | follow a wrong total to its cause in a shared helper, and fix it there rather than at the symptom |
|
||||
| `refactor-rename` | rename a function across a ten-module package, sparing two similar names a blind replace would break |
|
||||
| `spec-parser` | implement a parser from an eight-rule written spec when the plan gives only three examples, with tests |
|
||||
| `long-milestone` | carry out eight dependent steps (model, storage, sorting, CLI) over 25+ turns |
|
||||
|
||||
The first four were the original set. By September 2026 all seven newer local
|
||||
models passed all of them, so the last four were added to separate models that
|
||||
can hold the tool contract from models that can do a milestone's real work.
|
||||
Their plans stay explicit, as the planner is told to write them; what makes
|
||||
them harder is the work itself: tracing a cause, searching a codebase,
|
||||
reading a spec closely, and holding a plan over many turns.
|
||||
|
||||
It runs the executor exactly as the agent does (the same client wiring,
|
||||
context budget and `tool_choice` handling, and your config's tuning knobs),
|
||||
|
||||
Reference in new issue
Block a user