Add four harder benchmark tasks #53

2026-09-28T00:08:52Z

dev-agent commented

2026-09-28 00:08:52 +00:00

Every newer local model passed the original four tasks 20/20 (#49), so the benchmark had stopped telling models apart. This adds four harder tasks. Their plans stay explicit and low-inference, as planner.py requires; the difficulty is in the work itself.

The tasks

Task	Seed	What's hard	What a shortcut fails on
`trace-bug`	6-module invoicing package	A wrong multi-currency total whose cause is a shared rate helper, two calls away	Discount and tax go through the same helper, so a fix at the call site fails the hidden tests
`refactor-rename`	10 modules, 38 uses of `load_config`	Rename it to `read_settings` everywhere, including docstrings, a log message and the README	A blind replace also renames `load_config_defaults` and `reload_config`; a partial rename misses the README
`spec-parser`	`SPEC.md` with 8 rules	Implement `parse_duration` from the spec when the plan gives only 3 examples, and write tests for every rule	49 hidden cases, plus at least 16 tests the model must write itself
`long-milestone`	Task-tracker CLI	8 dependent steps adding due dates: model, storage with versioned files, sorting, CLI, tests	20 hidden tests, including end-to-end CLI runs

The second commit clarifies trace-bug's plan: rates have up to six decimal places, and no step may round them to fewer. Calibration showed Haiku fixing the right constant but keeping only four decimal places. An explicit plan should say how much precision is needed.

Calibration

2 runs per task at 320 W, with max_turns 40, after the fixes in #50–#52:

Model	trace-bug	refactor-rename	spec-parser	long-milestone	Total
local-fast (floor)	0/2	0/2	0/2	0/2	0/8
mimo-v2.6-9b	2/2	0/2	0/2	0/2	2/8
devstral-small-2-24b	2/2	0/2	0/2	0/2	2/8
claude-haiku-4-5 (ceiling)	2/2	2/2	1/2	0/2 (one was a 429)	5/8
qwen3.8-27b	2/2	2/2	1/2	1/2	6/8

Every task is passable. Each one separates the models except trace-bug, which the three stronger models all pass and the floor fails. Nothing needed adjusting beyond the trace-bug wording. The runs will be committed with the full run's results.

Tests

The bundled-task test now lists eight tasks.
The existing per-task checks cover the new tasks: the seed fails, the reference solution passes, and the hidden check isn't in the seed.
New self-tests prove that a fix at the call site (trace-bug), a blind text replace and a partial rename (refactor-rename) all fail their checks.
The Tuning Guide's task table has the four new rows.

CI steps run locally: ruff check, ruff format --check, and pytest -m "not integration": 420 passed.

🤖 Generated with Claude Code

Every newer local model passed the original four tasks 20/20 (#49), so the benchmark had stopped telling models apart. This adds four harder tasks. Their plans stay explicit and low-inference, as `planner.py` requires; the difficulty is in the work itself. ## The tasks | Task | Seed | What's hard | What a shortcut fails on | |---|---|---|---| | `trace-bug` | 6-module invoicing package | A wrong multi-currency total whose cause is a shared rate helper, two calls away | Discount and tax go through the same helper, so a fix at the call site fails the hidden tests | | `refactor-rename` | 10 modules, 38 uses of `load_config` | Rename it to `read_settings` everywhere, including docstrings, a log message and the README | A blind replace also renames `load_config_defaults` and `reload_config`; a partial rename misses the README | | `spec-parser` | `SPEC.md` with 8 rules | Implement `parse_duration` from the spec when the plan gives only 3 examples, and write tests for every rule | 49 hidden cases, plus at least 16 tests the model must write itself | | `long-milestone` | Task-tracker CLI | 8 dependent steps adding due dates: model, storage with versioned files, sorting, CLI, tests | 20 hidden tests, including end-to-end CLI runs | The second commit clarifies `trace-bug`'s plan: rates have up to six decimal places, and no step may round them to fewer. Calibration showed Haiku fixing the right constant but keeping only four decimal places. An explicit plan should say how much precision is needed. ## Calibration 2 runs per task at 320 W, with `max_turns` 40, after the fixes in #50–#52: | Model | trace-bug | refactor-rename | spec-parser | long-milestone | Total | |---|---|---|---|---|---| | local-fast (floor) | 0/2 | 0/2 | 0/2 | 0/2 | 0/8 | | mimo-v2.6-9b | 2/2 | 0/2 | 0/2 | 0/2 | 2/8 | | devstral-small-2-24b | 2/2 | 0/2 | 0/2 | 0/2 | 2/8 | | claude-haiku-4-5 (ceiling) | 2/2 | 2/2 | 1/2 | 0/2 (one was a 429) | 5/8 | | qwen3.8-27b | 2/2 | 2/2 | 1/2 | 1/2 | 6/8 | Every task is passable. Each one separates the models except `trace-bug`, which the three stronger models all pass and the floor fails. Nothing needed adjusting beyond the `trace-bug` wording. The runs will be committed with the full run's results. ## Tests - The bundled-task test now lists eight tasks. - The existing per-task checks cover the new tasks: the seed fails, the reference solution passes, and the hidden check isn't in the seed. - New self-tests prove that a fix at the call site (`trace-bug`), a blind text replace and a partial rename (`refactor-rename`) all fail their checks. - The Tuning Guide's task table has the four new rows. CI steps run locally: `ruff check`, `ruff format --check`, and `pytest -m "not integration"`: 420 passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 2 commits 2026-09-28 00:08:53 +00:00

Add four harder benchmark tasks 25fd57351d

All seven newer local models passed the original four tasks 20 of 20,
so the benchmark no longer told them apart. The new tasks keep plans
explicit, as the planner is told to write them, and make the work
itself harder:

- trace-bug: a wrong multi-currency total whose cause is a shared
  rate helper; discounts and tax go through it too, so a fix at the
  call site fails the check.
- refactor-rename: rename load_config across a ten-module package,
  leaving load_config_defaults and reload_config alone.
- spec-parser: implement parse_duration from an eight-rule SPEC.md
  given three examples, and write tests for every rule.
- long-milestone: eight dependent steps adding due dates to a task
  tracker, from the model through storage versions to the CLI.

Self-tests prove that a call-site fix, a blind text replace and a
partial rename all fail their checks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

trace-bug: say how much precision the rates need

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m14s

Details

277bff6fcd

Calibration showed Haiku fixing the right constant but keeping only four
decimal places, which passes the visible test and still loses the JPY
rate's precision. An explicit plan should say so.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dongar approved these changes

2026-09-28 00:30:32 +00:00

dev-agent merged commit 8467b8747f into main

2026-09-28 00:31:17 +00:00

dev-agent deleted branch harder-bench-tasks

2026-09-28 00:31:17 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#53