dongarandClaude Opus 5.5
4b6c48cc96
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m12s
Five runs per task at 320 W with max_turns 40, after the fixes in #50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b, devstral-small-2-24b and mimo-v2.6-9b; the harder four also for claude-haiku-4-5 as the ceiling, and two calibration runs each for local-fast as the floor. Runs failed by the proxy's rate limiter were set aside, not stored. Earlier results move to bench/results/archive/, which the report does not read: they were made at other power caps, at max_turns 24, or on the RTX 4070, and are kept as the record. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
21 lines
36 KiB
JSON
21 lines
36 KiB
JSON
{"backing": "openai/qwen3-14b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Fixed total_value to multiply unit price by quantity. All tests now pass.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5328, "tokens_out": 1492, "trimmed_requests": 0, "ts": "2026-09-25T01:03:24+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 46.4}
|
|
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\nImportError while importing test module '/tmp/tmpr1q1pe0r/work/.bench_check/test_hidden.py'.\nHint: make sure your test modules/packages have valid Python names.\nTraceback:\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n.bench_check/test_hidden.py:4: in <module>\n from inventory import Item, low_stock, total_value\nE ImportError: cannot import name 'Item' from 'inventory' (/tmp/tmpr1q1pe0r/work/inventory.py)\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.10s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n______________________ ERROR collecting test_inventory.py ______________________\nImportError while importing test module '/tmp/tmpr1q1pe0r/work/test_inventory.py'.\nHint: make sure your test modules/packages have valid Python names.\nTraceback:\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], packa", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 2560, "tokens_out": 1136, "trimmed_requests": 0, "ts": "2026-09-25T01:03:49+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 24.3}
|
|
{"backing": "openai/qwen3-14b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Fixed the total_value function to multiply unit price by quantity, passing all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5318, "tokens_out": 1623, "trimmed_requests": 0, "ts": "2026-09-25T01:04:23+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 34.5}
|
|
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n.bench_check/test_hidden.py:4: in <module>\n from inventory import Item, low_stock, total_value\ninventory.py:1: in <module>\n def total_value(items: list[Item]) -> float:\n ^^^^\nE NameError: name 'Item' is not defined\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py - NameError: name 'Item' is not defined\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.11s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n______________________ ERROR collecting test_inventory.py ______________________\ntest_inventory.py:1: in <module>\n from inventory import Item, low_stock, total_value\ninventory.py:1: in <module>\n def total_value(items: list[Item]) -> float:\n ^^^^\nE NameError: name 'Item' is not defined\n=========================== short ", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 2565, "tokens_out": 958, "trimmed_requests": 0, "ts": "2026-09-25T01:04:45+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 20.6}
|
|
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n.bench_check/test_hidden.py:4: in <module>\n from inventory import Item, low_stock, total_value\ninventory.py:1: in <module>\n def total_value(items: list[Item]) -> float:\n ^^^^\nE NameError: name 'Item' is not defined\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py - NameError: name 'Item' is not defined\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.09s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n______________________ ERROR collecting test_inventory.py ______________________\ntest_inventory.py:1: in <module>\n from inventory import Item, low_stock, total_value\ninventory.py:1: in <module>\n def total_value(items: list[Item]) -> float:\n ^^^^\nE NameError: name 'Item' is not defined\n=========================== short ", "prose_turns": 0, "provider": "openai", "success": false, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 2565, "tokens_out": 1051, "trimmed_requests": 0, "ts": "2026-09-25T01:05:07+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 3, "wall_s": 22.6}
|
|
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmpq8iz1j8z/work/rates.py\", line 56\nE [... 1445 characters omitted ...]\nE ^^^^^^^^\nE SyntaxError: invalid syntax. Perhaps you forgot a comma?\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.14s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 13616, "tokens_out": 3186, "trimmed_requests": 0, "ts": "2026-09-25T01:06:21+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 72.8}
|
|
{"backing": "openai/qwen3-14b", "check_output": ": 'R271'\nFAILED .bench_check/test_hidden.py::test_rate[272] - KeyError: 'R272'\nFAILED .bench_check/test_hidden.py::test_rate[273] - KeyError: 'R273'\nFAILED .bench_check/test_hidden.py::test_rate[274] - KeyError: 'R274'\nFAILED .bench_check/test_hidden.py::test_rate[275] - KeyError: 'R275'\nFAILED .bench_check/test_hidden.py::test_rate[276] - KeyError: 'R276'\nFAILED .bench_check/test_hidden.py::test_rate[277] - KeyError: 'R277'\nFAILED .bench_check/test_hidden.py::test_rate[278] - KeyError: 'R278'\nFAILED .bench_check/test_hidden.py::test_rate[279] - KeyError: 'R279'\nFAILED .bench_check/test_hidden.py::test_rate[280] - KeyError: 'R280'\nFAILED .bench_check/test_hidden.py::test_rate[281] - KeyError: 'R281'\nFAILED .bench_check/test_hidden.py::test_rate[282] - KeyError: 'R282'\nFAILED .bench_check/test_hidden.py::test_rate[283] - KeyError: 'R283'\nFAILED .bench_check/test_hidden.py::test_rate[284] - KeyError: 'R284'\nFAILED .bench_check/test_hidden.py::test_rate[285] - KeyError: 'R285'\nFAILED .bench_check/test_hidden.py::test_rate[286] - KeyError: 'R286'\nFAILED .bench_check/test_hidden.py::test_rate[287] - KeyError: 'R287'\nFAILED .bench_check/test_hidden.py::test_rate[288] - KeyError: 'R288'\nFAILED .bench_check/test_hidden.py::test_rate[289] - KeyError: 'R289'\nFAILED .bench_check/test_hidden.py::test_rate[290] - KeyError: 'R290'\nFAILED .bench_check/test_hidden.py::test_rate[291] - KeyError: 'R291'\nFAILED .bench_check/test_hidden.py::test_rate[292] - KeyError: 'R292'\nFAILED .bench_check/test_hidden.py::test_rate[293] - KeyError: 'R293'\nFAILED .bench_check/test_hidden.py::test_rate[294] - KeyError: 'R294'\nFAILED .bench_check/test_hidden.py::test_rate[295] - KeyError: 'R295'\nFAILED .bench_check/test_hidden.py::test_rate[296] - KeyError: 'R296'\nFAILED .bench_check/test_hidden.py::test_rate[297] - KeyError: 'R297'\nFAILED .bench_check/test_hidden.py::test_rate[298] - KeyError: 'R298'\nFAILED .bench_check/test_hidden.py::test_rate[299] - KeyError: 'R299'\n247 failed, 54 passed in 0.49s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n.....................................................FFFFFFFFFFFFFFFFFFF [ 24%]\nFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [ 48%]\nFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [ 72%]\nFFFFFFFFFFFFFFFFFFFFF.FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [ 96%]\nFFFFFFFFFFFF [100%]\n=================================== FAILU", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 12726, "tokens_out": 2366, "trimmed_requests": 0, "ts": "2026-09-25T01:07:16+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 54.4}
|
|
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmphac71897/work/rates.py\", line 56\nE [... 1445 characters omitted ...]\nE ^^^^^^^^\nE SyntaxError: invalid syntax. Perhaps you forgot a comma?\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.15s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 13614, "tokens_out": 3000, "trimmed_requests": 0, "ts": "2026-09-25T01:08:25+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 69.2}
|
|
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmp0vs0w4kr/work/rates.py\", line 56\nE [... 1445 characters omitted ...]\nE ^^^^^^^^\nE SyntaxError: invalid syntax. Perhaps you forgot a comma?\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.14s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 13607, "tokens_out": 3223, "trimmed_requests": 0, "ts": "2026-09-25T01:09:40+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 73.8}
|
|
{"backing": "openai/qwen3-14b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmp_pxm0ldr/work/rates.py\", line 56\nE [... 1445 characters omitted ...]\nE ^^^^^^^^\nE SyntaxError: invalid syntax. Perhaps you forgot a comma?\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.15s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 13609, "tokens_out": 3180, "trimmed_requests": 0, "ts": "2026-09-25T01:10:53+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 73.0}
|
|
{"backing": "openai/qwen3-14b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n..FF [100%]\n=================================== FAILURES ===================================\n_________________________ test_cli_prints_the_greeting _________________________\n\ncapsys = <_pytest.capture.CaptureFixture object at 0x777eb60e3e90>\n\n def test_cli_prints_the_greeting(capsys) -> None:\n> assert main([\"Bob\"])\nE AssertionError: assert 0\nE + wh", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 13753, "tokens_out": 5755, "trimmed_requests": 0, "ts": "2026-09-25T01:12:55+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 10, "wall_s": 121.2}
|
|
{"backing": "openai/qwen3-14b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added --shout option to greeter, updated core and cli code, added tests, and verified with pytest.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 21373, "tokens_out": 9952, "trimmed_requests": 0, "ts": "2026-09-25T01:16:25+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 13, "wall_s": 210.0}
|
|
{"backing": "openai/qwen3-14b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n..FF [100%]\n=================================== FAILURES ===================================\n_________________________ test_cli_prints_the_greeting _________________________\n\ncapsys = <_pytest.capture.CaptureFixture object at 0x7a2788fdbe30>\n\n def test_cli_prints_the_greeting(capsys) -> None:\n> assert main([\"Bob\"])\nE AssertionError: assert 0\nE + wh", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9432, "tokens_out": 6691, "trimmed_requests": 0, "ts": "2026-09-25T01:18:47+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 142.1}
|
|
{"backing": "openai/qwen3-14b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added --shout option to greeter, updated core and cli code, added tests, and verified with pytest.", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9104, "tokens_out": 3626, "trimmed_requests": 0, "ts": "2026-09-25T01:20:03+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 75.8}
|
|
{"backing": "openai/qwen3-14b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n..FF [100%]\n=================================== FAILURES ===================================\n_________________________ test_cli_prints_the_greeting _________________________\n\ncapsys = <_pytest.capture.CaptureFixture object at 0x76eac82e3da0>\n\n def test_cli_prints_the_greeting(capsys) -> None:\n> assert main([\"Bob\"])\nE AssertionError: assert 0\nE + wh", "prose_turns": 0, "provider": "openai", "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9382, "tokens_out": 7755, "trimmed_requests": 0, "ts": "2026-09-25T01:22:48+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 164.5}
|
|
{"backing": "openai/qwen3-14b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added slugify function and passed all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 1769, "tokens_out": 1300, "trimmed_requests": 0, "ts": "2026-09-25T01:23:16+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 27.7}
|
|
{"backing": "openai/qwen3-14b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added slugify function with tests; all tests passed.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 1782, "tokens_out": 1239, "trimmed_requests": 0, "ts": "2026-09-25T01:23:43+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 2, "wall_s": 26.2}
|
|
{"backing": "openai/qwen3-14b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added slugify function and passed all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3627, "tokens_out": 1332, "trimmed_requests": 0, "ts": "2026-09-25T01:24:11+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 28.1}
|
|
{"backing": "openai/qwen3-14b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added slugify function and passed all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 734, "tokens_out": 653, "trimmed_requests": 0, "ts": "2026-09-25T01:24:25+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 1, "wall_s": 13.9}
|
|
{"backing": "openai/qwen3-14b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "50b4682+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "local-coder", "model_info": {"max_input_tokens": 32768, "max_output_tokens": 8192, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11028, Qwen3-14B Q4_K_M, q8 KV, 32k", "outcome": "completed", "outcome_note": "Added slugify function and passed all tests.", "prose_turns": 0, "provider": "openai", "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 733, "tokens_out": 523, "trimmed_requests": 0, "ts": "2026-09-25T01:24:37+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 24}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 1, "wall_s": 11.3}
|