Add four harder benchmark tasks

Conversation 1 Commits 2 Files Changed 67

+2117 -2

2 Commits

Author	SHA1	Message	Date
dongarandClaude Opus 5.5	277bff6fcd	trace-bug: say how much precision the rates need CI / checks (push) Skipped CI / checks (pull_request) Successful in 1m14s Details Calibration showed Haiku fixing the right constant but keeping only four decimal places, which passes the visible test and still loses the JPY rate's precision. An explicit plan should say so. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-27 22:43:39 +00:00
dongarandClaude Opus 5.5	25fd57351d	Add four harder benchmark tasks All seven newer local models passed the original four tasks 20 of 20, so the benchmark no longer told them apart. The new tasks keep plans explicit, as the planner is told to write them, and make the work itself harder: - trace-bug: a wrong multi-currency total whose cause is a shared rate helper; discounts and tax go through it too, so a fix at the call site fails the check. - refactor-rename: rename load_config across a ten-module package, leaving load_config_defaults and reload_config alone. - spec-parser: implement parse_duration from an eight-rule SPEC.md given three examples, and write tests for every rule. - long-milestone: eight dependent steps adding due dates to a task tracker, from the model through storage versions to the CLI. Self-tests prove that a call-site fix, a blind text replace and a partial rename all fail their checks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-27 22:43:39 +00:00

Author

SHA1

Message

Date

dongarandClaude Opus 5.5

277bff6fcd

trace-bug: say how much precision the rates need

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m14s

Details

Calibration showed Haiku fixing the right constant but keeping only four
decimal places, which passes the visible test and still loses the JPY
rate's precision. An explicit plan should say so.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-09-27 22:43:39 +00:00

dongarandClaude Opus 5.5

25fd57351d

Add four harder benchmark tasks

All seven newer local models passed the original four tasks 20 of 20,
so the benchmark no longer told them apart. The new tasks keep plans
explicit, as the planner is told to write them, and make the work
itself harder:

- trace-bug: a wrong multi-currency total whose cause is a shared
  rate helper; discounts and tax go through it too, so a fix at the
  call site fails the check.
- refactor-rename: rename load_config across a ten-module package,
  leaving load_config_defaults and reload_config alone.
- spec-parser: implement parse_duration from an eight-rule SPEC.md
  given three examples, and write tests for every rule.
- long-milestone: eight dependent steps adding due dates to a task
  tracker, from the model through storage versions to the CLI.

Self-tests prove that a call-site fix, a blind text replace and a
partial rename all fail their checks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-09-27 22:43:39 +00:00

Add four harder benchmark tasks #53