Add four harder benchmark tasks
All seven newer local models passed the original four tasks 20 of 20, so the benchmark no longer told them apart. The new tasks keep plans explicit, as the planner is told to write them, and make the work itself harder: - trace-bug: a wrong multi-currency total whose cause is a shared rate helper; discounts and tax go through it too, so a fix at the call site fails the check. - refactor-rename: rename load_config across a ten-module package, leaving load_config_defaults and reload_config alone. - spec-parser: implement parse_duration from an eight-rule SPEC.md given three examples, and write tests for every rule. - long-milestone: eight dependent steps adding due dates to a task tracker, from the model through storage versions to the CLI. Self-tests prove that a call-site fix, a blind text replace and a partial rename all fail their checks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f2c48d8b16
commit
25fd57351d
67 files changed
+2115
-2
No files matched your search
@@ -0,0 +1,19 @@
|
||||
title: Trace a wrong total to its cause in a shared helper
|
||||
check: python -m pytest -q .bench_check
|
||||
plan: |
|
||||
Milestone 1: fix the wrong totals on multi-currency invoices.
|
||||
|
||||
1. Run `python -m pytest -q`. `test_multi_currency_total` in
|
||||
`tests/test_invoice.py` fails: totals are slightly off when an invoice has
|
||||
lines in another currency. The test is correct. Do not change any file
|
||||
under `tests/`.
|
||||
2. Find the cause. Start at `Invoice.total` in `invoicing/invoice.py` and
|
||||
follow the functions it calls, reading each one, until you find where
|
||||
precision is lost. The cause is in a function that more than one module
|
||||
uses.
|
||||
3. Fix it at its source, in that shared function or the value it uses. Do
|
||||
not special-case currencies in `invoicing/invoice.py` or
|
||||
`invoicing/currency.py`: every caller of the shared function has the
|
||||
same problem, and all of them must be fixed by your change.
|
||||
4. Run `python -m pytest -q` until every test passes.
|
||||
5. Call finish.
|
||||
Reference in new issue
Block a user