dongarandClaude Opus 5.5
4b6c48cc96
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m12s
Five runs per task at 320 W with max_turns 40, after the fixes in #50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b, devstral-small-2-24b and mimo-v2.6-9b; the harder four also for claude-haiku-4-5 as the ceiling, and two calibration runs each for local-fast as the floor. Runs failed by the proxy's rate limiter were set aside, not stored. Earlier results move to bench/results/archive/, which the report does not read: they were made at other power caps, at max_turns 24, or on the RTX 4070, and are kept as the record. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
8 lines
14 KiB
JSON
8 lines
14 KiB
JSON
{"backing": "anthropic/claude-haiku-4-5", "check_output": ".......... [100%]\n10 passed in 0.02s\n", "commit": "277bff6+dirty", "cost": 0.049727, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "claude-haiku-4-5", "model_info": {"max_input_tokens": 200000, "max_output_tokens": 64000, "supports_function_calling": true, "supports_tool_choice": true}, "note": "Anthropic API via LiteLLM", "outcome": "completed", "outcome_note": "Fixed the precision loss in multi-currency invoices by increasing RATE_PLACES in invoicing/money.py from 0.01 (2 decimal places) to 0.000001 (6 decimal places). This allows exchange rates and other fractions to maintain their full precision when normalized, preventing rounding errors that were causing incorrect totals. The fix is at the source in the shared function normalise_rate, which is used by all callers including currency conversion, tax calculation, and discounts. All tests now pass.", "prose_turns": 0, "provider": "anthropic", "rate_limit_wait_s": 0.0, "success": true, "task": "trace-bug", "task_hash": "a234c8fbeceeb166", "tokens_in": 41237, "tokens_out": 1698, "trimmed_requests": 0, "ts": "2026-09-27T22:46:57+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 12, "wall_s": 20.5}
|
|
{"backing": "anthropic/claude-haiku-4-5", "check_output": ".......... [100%]\n10 passed in 0.02s\n", "commit": "277bff6+dirty", "cost": 0.054921, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "claude-haiku-4-5", "model_info": {"max_input_tokens": 200000, "max_output_tokens": 64000, "supports_function_calling": true, "supports_tool_choice": true}, "note": "Anthropic API via LiteLLM", "outcome": "completed", "outcome_note": "Fixed wrong totals on multi-currency invoices by increasing the precision of RATE_PLACES in invoicing/money.py from 0.01 (2 decimal places) to 0.000001 (6 decimal places). This prevents loss of precision when normalizing exchange rates before applying them to currency conversions. The test test_multi_currency_total now passes, correctly converting 100 USD to 91.95 EUR instead of 92.00 EUR. All tests pass.", "prose_turns": 0, "provider": "anthropic", "rate_limit_wait_s": 0.0, "success": true, "task": "trace-bug", "task_hash": "a234c8fbeceeb166", "tokens_in": 45296, "tokens_out": 1925, "trimmed_requests": 0, "ts": "2026-09-27T22:47:18+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 11, "wall_s": 20.7}
|
|
{"backing": "anthropic/claude-haiku-4-5", "check_output": "....... [100%]\n7 passed in 0.34s\n", "commit": "277bff6+dirty", "cost": 0.223718, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "claude-haiku-4-5", "model_info": {"max_input_tokens": 200000, "max_output_tokens": 64000, "supports_function_calling": true, "supports_tool_choice": true}, "note": "Anthropic API via LiteLLM", "outcome": "completed", "outcome_note": "Successfully renamed `load_config` to `read_settings` across the entire project. Changes made:\n- Renamed function definition in shopkeeper/config.py\n- Updated all imports in shopkeeper/__init__.py, catalog.py, cli.py, inventory.py, orders.py, pricing.py, reports.py, storage.py, and notifications.py\n- Updated all function calls in these modules\n- Updated docstrings and comments that referenced `load_config()`\n- Updated the log message in notifications.py\n- Updated README.md to show the new functi", "prose_turns": 0, "provider": "anthropic", "rate_limit_wait_s": 0.0, "success": true, "task": "refactor-rename", "task_hash": "696f6832ddf02216", "tokens_in": 199253, "tokens_out": 4893, "trimmed_requests": 0, "ts": "2026-09-27T22:48:12+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 30, "wall_s": 53.2}
|
|
{"backing": "anthropic/claude-haiku-4-5", "check_output": "....... [100%]\n7 passed in 0.34s\n", "commit": "277bff6+dirty", "cost": 0.07054, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "claude-haiku-4-5", "model_info": {"max_input_tokens": 200000, "max_output_tokens": 64000, "supports_function_calling": true, "supports_tool_choice": true}, "note": "Anthropic API via LiteLLM", "outcome": "completed", "outcome_note": "Renamed `load_config` to `read_settings` across the project. Updated the function definition in shopkeeper/config.py, all imports in __init__.py and all other modules (catalog.py, cli.py, reports.py, inventory.py, storage.py, pricing.py, orders.py, notifications.py), updated all function calls, docstrings, comments, log messages, and README.md. Updated tests in test_shop.py and test_config.py. Verified that all 8 tests pass.", "prose_turns": 0, "provider": "anthropic", "rate_limit_wait_s": 0.0, "success": true, "task": "refactor-rename", "task_hash": "696f6832ddf02216", "tokens_in": 44675, "tokens_out": 5173, "trimmed_requests": 0, "ts": "2026-09-27T22:48:47+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 34.3}
|
|
{"backing": "anthropic/claude-haiku-4-5", "check_output": " components:\nE raise ValueError(f\"invalid duration: {original_text!r}\")\nE \nE for component in components:\nE # Extract number and unit from component\nE # The component should be: number followed by a single letter unit\nE match = re.match(r'^([0-9]+(?:\\.[0-9]+)?)([a-zA-Z])$', component)\nE if not match:\nE > raise ValueError(f\"invalid duration: {original_text!r}\")\nE E ValueError: invalid duration: '1d2h3m4s'\nE \nE durations.py:121: ValueError\nE =========================== short test summary info ============================\nE FAILED test_durations.py::TestRule1UnitForm::test_valid_hours_and_minutes - V...\nE FAILED test_durations.py::TestRule1UnitForm::test_valid_all_units - ValueErro...\nE FAILED test_durations.py::TestRule2CaseInsensitive::test_valid_uppercase_units\nE FAILED test_durations.py::TestRule2CaseInsensitive::test_valid_mixed_case_units\nE FAILED test_durations.py::TestRule3Order::test_valid_correct_order - ValueErr...\nE 5 failed, 29 passed in 0.06s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'test_durations.py'], returncode=1, stdou... test_durations.py::TestRule3Order::test_valid_correct_order - ValueErr...\\n5 failed, 29 passed in 0.06s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:87: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_valid[1h30m-5400] - ValueError: inva...\nFAILED .bench_check/test_hidden.py::test_valid[1d2h3m4s-93784] - ValueError: ...\nFAILED .bench_check/test_hidden.py::test_valid[1H30M-5400] - ValueError: inva...\nFAILED .bench_check/test_hidden.py::test_the_plan_asked_for_tests_and_they_pass\n4 failed, 45 passed in 0.38s\n", "commit": "277bff6+dirty", "cost": 0.057812, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "claude-haiku-4-5", "model_info": {"max_input_tokens": 200000, "max_output_tokens": 64000, "supports_function_calling": true, "supports_tool_choice": true}, "note": "Anthropic API via LiteLLM", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n.FF.FF.F.......................... [100%]\n=================================== FAILURES ===================================\n________________ TestRule1UnitForm.test_valid_hours_and_minutes ________________\n\nself = <test_durations.TestRule1UnitForm object at 0x71cdc47cf980>\n\n def test_valid_hours_and_minutes(self):\n> assert parse_duration(\"1h30m\") == 5400\n ^^^^^^^^^^^^^^^^^^^^^^^\n", "prose_turns": 0, "provider": "anthropic", "rate_limit_wait_s": 0.0, "success": false, "task": "spec-parser", "task_hash": "bcead05857457203", "tokens_in": 30872, "tokens_out": 5388, "trimmed_requests": 0, "ts": "2026-09-27T22:49:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 39.3}
|
|
{"backing": "anthropic/claude-haiku-4-5", "check_output": "................................................. [100%]\n49 passed in 0.30s\n", "commit": "277bff6+dirty", "cost": 0.03952, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "claude-haiku-4-5", "model_info": {"max_input_tokens": 200000, "max_output_tokens": 64000, "supports_function_calling": true, "supports_tool_choice": true}, "note": "Anthropic API via LiteLLM", "outcome": "completed", "outcome_note": "Implemented parse_duration function according to SPEC.md with all eight rules:\n1. Unit form (d, h, m, s components)\n2. Case-insensitive units\n3. Correct order (d, h, m, s) with no duplicates\n4. Spaces allowed around and between components, not between number and unit\n5. Bare numbers treated as seconds\n6. Decimals only in first component; result must be whole seconds\n7. Clock form M:SS or H:MM:SS with proper validation\n8. Zero allowed; empty/spaces-only/negative durations rejected\n\nCreated compre", "prose_turns": 0, "provider": "anthropic", "rate_limit_wait_s": 0.0, "success": true, "task": "spec-parser", "task_hash": "bcead05857457203", "tokens_in": 19475, "tokens_out": 4009, "trimmed_requests": 0, "ts": "2026-09-27T22:49:58+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 30.8}
|
|
{"backing": "anthropic/claude-haiku-4-5", "check_output": "...................F [100%]\n=================================== FAILURES ===================================\n_____________ test_the_projects_own_tests_pass_and_cover_due_dates _____________\n\n def test_the_projects_own_tests_pass_and_cover_due_dates() -> None:\n result = subprocess.run(\n [sys.executable, \"-m\", \"pytest\", \"-q\", \"tests\"],\n capture_output=True,\n text=True,\n check=False,\n cwd=Path.cwd(),\n )\n assert result.returncode == 0, result.stdout[-1500:]\n written = \"\\n\".join(p.read_text() for p in Path(\"tests\").glob(\"test_*.py\"))\n for name in (\"due\", \"overdue\", \"version\"):\n> assert name in written, f\"no test mentions {name!r}\"\nE AssertionError: no test mentions 'due'\nE assert 'due' in 'from pathlib import Path\\n\\nfrom tracker import storage\\nfrom tracker.cli import main\\nfrom tracker.models import Tas..., 1]\\n\\n\\ndef test_completing_an_unknown_task() -> None:\\n with pytest.raises(KeyError):\\n complete([], 7)\\n'\n\n.bench_check/test_hidden.py:186: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass_and_cover_due_dates\n1 failed, 19 passed in 0.73s\n", "commit": "277bff6+dirty", "cost": 0.121394, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "claude-haiku-4-5", "model_info": {"max_input_tokens": 200000, "max_output_tokens": 64000, "supports_function_calling": true, "supports_tool_choice": true}, "note": "Anthropic API via LiteLLM", "outcome": "completed", "outcome_note": "Milestone 1 completed: Added due dates to the task tracker.\n\n1. Added `due: date | None = None` field to Task model\n2. Updated storage.py to write version 2 files with ISO date strings and read both version 1 and 2 files\n3. Updated add_task() to accept optional `due` parameter\n4. Modified sort_tasks() to sort by: not done first, then by due date (with due dates first, earliest first), then by id\n5. Added overdue() function to find tasks not done with due dates before today\n6. Updated format_task", "prose_turns": 0, "provider": "anthropic", "rate_limit_wait_s": 0.0, "success": false, "task": "long-milestone", "task_hash": "ba436defff7d212f", "tokens_in": 106769, "tokens_out": 2925, "trimmed_requests": 0, "ts": "2026-09-27T22:50:31+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 21, "wall_s": 31.9}
|