Add four harder benchmark tasks

All seven newer local models passed the original four tasks 20 of 20,
so the benchmark no longer told them apart. The new tasks keep plans
explicit, as the planner is told to write them, and make the work
itself harder:

- trace-bug: a wrong multi-currency total whose cause is a shared
  rate helper; discounts and tax go through it too, so a fix at the
  call site fails the check.
- refactor-rename: rename load_config across a ten-module package,
  leaving load_config_defaults and reload_config alone.
- spec-parser: implement parse_duration from an eight-rule SPEC.md
  given three examples, and write tests for every rule.
- long-milestone: eight dependent steps adding due dates to a task
  tracker, from the model through storage versions to the CLI.

Self-tests prove that a call-site fix, a blind text replace and a
partial rename all fail their checks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-09-27 22:43:39 +00:00
1 parent f2c48d8b16
commit 25fd57351d
67 files changed
+2115 -2

No files matched your search

@@ -0,0 +1,18 @@
title: Rename a function used across a package
check: python -m pytest -q .bench_check
plan: |
Milestone 1: rename `load_config` to `read_settings` across the project.
1. Find every use of `load_config`: run
`grep -rnw load_config --include=*.py --include=*.md .`
It is defined in `shopkeeper/config.py` and used in many other modules,
in tests, in docstrings, in comments, in a log message and in
`README.md`.
2. Rename the function to `read_settings` everywhere: its definition,
imports, calls, `__all__`, docstrings, comments, strings and
`README.md`. Afterwards the grep in step 1 must print nothing.
3. Rename only that exact name. `load_config_defaults` and `reload_config`
are different functions and keep their names. Change no behaviour.
4. Update the tests under `tests/` to use the new name, and run
`python -m pytest -q` until every test passes.
5. Call finish.