dongarandClaude Opus 5.5
4b6c48cc96
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m12s
Five runs per task at 320 W with max_turns 40, after the fixes in #50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b, devstral-small-2-24b and mimo-v2.6-9b; the harder four also for claude-haiku-4-5 as the ceiling, and two calibration runs each for local-fast as the floor. Runs failed by the proxy's rate limiter were set aside, not stored. Earlier results move to bench/results/archive/, which the report does not read: they were made at other power caps, at max_turns 24, or on the RTX 4070, and are kept as the record. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
9 lines
16 KiB
JSON
9 lines
16 KiB
JSON
{"backing": "openai/qwen3.8-27b", "check_output": ".......... [100%]\n10 passed in 0.02s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 129\nstdout:\n\nstderr:\nwarning: Not a git repository. Use --no-index to compare two paths outside a working tree\nusage: git diff --no-index [<options>] <path> <path>\n\nDiff output format options\n -p, --patch generate patch\n -s, --no-patch suppress diff output\n -u generate patch\n -U, --unified[=<n>] generate diffs with <n> lines context\n -W, --[no-]function-context\n generate", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "trace-bug", "task_hash": "a234c8fbeceeb166", "tokens_in": 10602, "tokens_out": 2590, "trimmed_requests": 0, "ts": "2026-09-27T23:18:19+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 106.3}
|
|
{"backing": "openai/qwen3.8-27b", "check_output": ".......... [100%]\n10 passed in 0.02s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed multi-currency invoice totals. Cause: the shared `apply_rate` in invoicing/money.py rounded every rate/fraction through `normalise_rate` to `RATE_PLACES = Decimal(\"0.01\")` (2 places), so the USD\u2192EUR rate 1/1.0875 = 0.919523809\u2026 became 0.92 and 100 USD converted to 92.00 EUR instead of 91.95. Fix: set `RATE_PLACES = Decimal(\"0.000001\")` (six places, matching the JPY rate precision) in the shared module \u2014 one change fixes all callers (currency.convert, discounts.discount_amount, Invoice.tax)", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "trace-bug", "task_hash": "a234c8fbeceeb166", "tokens_in": 29977, "tokens_out": 5663, "trimmed_requests": 0, "ts": "2026-09-27T23:20:30+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 130.4}
|
|
{"backing": "openai/qwen3.8-27b", "check_output": "....... [100%]\n7 passed in 0.34s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Renamed `load_config` to `read_settings` across the project: definition in shopkeeper/config.py, imports and calls in all 9 shopkeeper modules, `__all__` in __init__.py, docstrings, a comment, a log message, README.md, and both test files. `load_config_defaults` and `reload_config` were left untouched. The original grep for `load_config` now returns nothing, and `python -m pytest -q` passes (8 tests).", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "refactor-rename", "task_hash": "696f6832ddf02216", "tokens_in": 11226, "tokens_out": 2389, "trimmed_requests": 0, "ts": "2026-09-27T23:21:26+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 55.0}
|
|
{"backing": "openai/qwen3.8-27b", "check_output": "....... [100%]\n7 passed in 0.34s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Renamed `load_config` to `read_settings` project-wide (definition, imports, calls, `__all__`, docstrings, comments, log message, README.md and tests). The step-1 grep now prints nothing; `load_config_defaults` and `reload_config` were left untouched, and `python -m pytest -q` passes with 8/8 tests.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "refactor-rename", "task_hash": "696f6832ddf02216", "tokens_in": 17374, "tokens_out": 1733, "trimmed_requests": 0, "ts": "2026-09-27T23:22:08+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 42.1}
|
|
{"backing": "openai/qwen3.8-27b", "check_output": " raise _invalid(text)\nE number, unit = match.group(1), match.group(2).lower()\nE if \".\" in number and component_index != 0:\nE raise _invalid(text)\nE rank = _UNIT_RANK[unit]\nE if rank <= previous_rank:\nE raise _invalid(text)\nE previous_rank = rank\nE component_index += 1\nE total += Decimal(number) * _UNIT_SECONDS[unit]\nE position = match.end()\nE if position != len(token):\nE > raise _invalid(text)\nE E ValueError: invalid duration: '0'\nE \nE durations.py:50: ValueError\nE =========================== short test summary info ============================\nE FAILED test_durations.py::test_rule4_spaces_valid - ValueError: invalid durat...\nE FAILED test_durations.py::test_rule5_bare_number_valid - ValueError: invalid ...\nE FAILED test_durations.py::test_rule8_zero_valid - ValueError: invalid duratio...\nE 3 failed, 14 passed in 0.04s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'test_durations.py'], returncode=1, stdou... test_durations.py::test_rule8_zero_valid - ValueError: invalid duratio...\\n3 failed, 14 passed in 0.04s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:87: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_valid[90-90] - ValueError: invalid d...\nFAILED .bench_check/test_hidden.py::test_valid[ 7 -7] - ValueError: invalid...\nFAILED .bench_check/test_hidden.py::test_valid[0-0] - ValueError: invalid dur...\nFAILED .bench_check/test_hidden.py::test_the_plan_asked_for_tests_and_they_pass\n4 failed, 45 passed in 0.34s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n......F.F.....F.. [100%]\n=================================== FAILURES ===================================\n___________________________ test_rule4_spaces_valid ____________________________\n\n def test_rule4_spaces_valid():\n assert parse_duration(\" 1h 30m \") == 5400\n assert parse_duration(\"1h 30m\") == 5400\n> assert parse_duration(\" 90 \") == 90\n ^^^^^^", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "spec-parser", "task_hash": "bcead05857457203", "tokens_in": 16662, "tokens_out": 16238, "trimmed_requests": 0, "ts": "2026-09-27T23:28:17+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 367.5}
|
|
{"backing": "openai/qwen3.8-27b", "check_output": "................................................. [100%]\n49 passed in 0.30s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Milestone 1 done. Read SPEC.md (8 rules) and implemented `parse_duration` in `durations.py`: unit form (d/h/m/s, case-insensitive, strict d\u2192h\u2192m\u2192s order, each unit once), spaces only between/around components, bare-number-as-seconds (not mixable), decimals only on the first component with whole-seconds results, clock form M:SS / H:MM:SS (fields after the first exactly two digits and below 60, no decimals/spaces inside), zero allowed; every invalid input raises ValueError with message `invalid dur", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "spec-parser", "task_hash": "bcead05857457203", "tokens_in": 58572, "tokens_out": 13638, "trimmed_requests": 0, "ts": "2026-09-27T23:38:30+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 11, "wall_s": 612.0}
|
|
{"backing": "openai/qwen3.8-27b", "check_output": ".................... [100%]\n20 passed in 0.72s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Milestone 1 complete: added due dates to the task tracker. (1) Task gained `due: date | None = None`. (2) storage saves version 2 with due as ISO string or null, loads v1 (due=None) and v2, raises ValueError(\"unsupported tasks file version: N\") otherwise. (3) add_task takes `due` keyword arg. (4) sort_tasks: not-done first, then due-date tasks earliest-first, then undated, ties by id. (5) Added overdue(tasks, today). (6) format_task(task, today) appends \" (due YYYY-MM-DD)\" and shows [!] for over", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "long-milestone", "task_hash": "ba436defff7d212f", "tokens_in": 116237, "tokens_out": 9001, "trimmed_requests": 0, "ts": "2026-09-27T23:42:03+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 23, "wall_s": 212.1}
|
|
{"backing": "openai/qwen3.8-27b", "check_output": "test_the_projects_own_tests_pass_and_cover_due_dates _____________\n\n def test_the_projects_own_tests_pass_and_cover_due_dates() -> None:\n result = subprocess.run(\n [sys.executable, \"-m\", \"pytest\", \"-q\", \"tests\"],\n capture_output=True,\n text=True,\n check=False,\n cwd=Path.cwd(),\n )\n> assert result.returncode == 0, result.stdout[-1500:]\nE AssertionError: ...........................F........... [100%]\nE =================================== FAILURES ===================================\nE ______________________ test_sort_tasks_ties_broken_by_id _______________________\nE \nE def test_sort_tasks_ties_broken_by_id() -> None:\nE tasks = []\nE add_task(tasks, \"b same due\", due=date(2026, 10, 1))\nE add_task(tasks, \"a same due\", due=date(2026, 10, 1))\nE > assert [t.id for t in sort_tasks(tasks)] == [2, 1]\nE E assert [1, 2] == [2, 1]\nE E \nE E At index 0 diff: 1 != 2\nE E Use -v to get more diff\nE \nE tests/test_sort.py:37: AssertionError\nE =========================== short test summary info ============================\nE FAILED tests/test_sort.py::test_sort_tasks_ties_broken_by_id - assert [1, 2] ...\nE 1 failed, 38 passed in 0.09s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'tests'], returncode=1, stdout='............ tests/test_sort.py::test_sort_tasks_ties_broken_by_id - assert [1, 2] ...\\n1 failed, 38 passed in 0.09s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:183: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass_and_cover_due_dates\n1 failed, 19 passed in 0.79s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "qwen3.8-27b", "model_info": {"max_input_tokens": 131072, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": true}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 128k, temp 1.0 top-p 0.95 top-k 20, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n...........................F........... [100%]\n=================================== FAILURES ===================================\n______________________ test_sort_tasks_ties_broken_by_id _______________________\n\n def test_sort_tasks_ties_broken_by_id() -> None:\n tasks = []\n add_task(tasks, \"b same due\", due=date(2026, 10, 1))\n add_task(tasks, \"a same due\", due=date(2026, 10, 1))\n> ", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "long-milestone", "task_hash": "ba436defff7d212f", "tokens_in": 95804, "tokens_out": 11221, "trimmed_requests": 0, "ts": "2026-09-27T23:46:26+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 18, "wall_s": 261.7}
|