dev/nisse
2
1 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
main
nisse/bench/results/2026-09-27-devstral-small-2-24b.jsonl
T
dongarandClaude Opus 5.5 4b6c48cc96
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m12s
Details
Benchmark the finalists on all eight tasks, and archive older results 
Five runs per task at 320 W with max_turns 40, after the fixes in
#50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b,
devstral-small-2-24b and mimo-v2.6-9b; the harder four also for
claude-haiku-4-5 as the ceiling, and two calibration runs each for
local-fast as the floor. Runs failed by the proxy's rate limiter were
set aside, not stored.

Earlier results move to bench/results/archive/, which the report does
not read: they were made at other power caps, at max_turns 24, or on
the RTX 4070, and are kept as the record.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 03:14:40 +00:00

19 lines
40 KiB
JSON
Raw Permalink Blame History

{"backing": "openai/devstral-small-2-24b", "check_output": ".......... [100%]\n10 passed in 0.02s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed the wrong totals on multi-currency invoices by increasing the precision of `RATE_PLACES` in `invoicing/money.py` from `Decimal(\"0.01\")` to `Decimal(\"0.000001\")`. This ensures that exchange rates with up to six decimal places are preserved, preventing precision loss during currency conversion.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "trace-bug", "task_hash": "a234c8fbeceeb166", "tokens_in": 18783, "tokens_out": 1034, "trimmed_requests": 0, "ts": "2026-09-27T23:47:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 60.9}
{"backing": "openai/devstral-small-2-24b", "check_output": ".......... [100%]\n10 passed in 0.02s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed precision loss in multi-currency invoice totals by increasing RATE_PLACES from 2 to 6 decimal places in invoicing/money.py. This preserves the precision of currency conversion rates, ensuring correct totals for invoices with lines in different currencies.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "trace-bug", "task_hash": "a234c8fbeceeb166", "tokens_in": 41604, "tokens_out": 2103, "trimmed_requests": 0, "ts": "2026-09-27T23:48:09+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 15, "wall_s": 41.7}
{"backing": "openai/devstral-small-2-24b", "check_output": "orks(tmp_path: Path) -> None:\n (tmp_path / \"catalog.csv\").write_text(\"sku,name,unit_price\\nSKU-2,hammer,12.50\\n\")\n ini = tmp_path / \"shop.ini\"\n ini.write_text(\n f\"[shop]\\ncurrency = EUR\\ntax_percent = 20\\ncatalog_file = {tmp_path / 'catalog.csv'}\\n\"\n )\n run = subprocess.run(\n [\n sys.executable,\n \"-m\",\n \"shopkeeper\",\n \"--config\",\n str(ini),\n \"price\",\n \"SKU-2\",\n \"2\",\n ],\n cwd=ROOT,\n capture_output=True,\n text=True,\n check=False,\n )\n> assert run.stdout.strip() == \"30.00 EUR\", run.stderr[-1000:]\nE AssertionError: Traceback (most recent call last):\nE File \"<frozen runpy>\", line 189, in _run_module_as_main\nE File \"<frozen runpy>\", line 148, in _get_module_details\nE File \"<frozen runpy>\", line 112, in _get_module_details\nE File \"/tmp/tmpj0g6dpjt/work/shopkeeper/__init__.py\", line 1, in <module>\nE from shopkeeper.config import Settings, read_settings, load_config_defaults, reload_config\nE File \"/tmp/tmpj0g6dpjt/work/shopkeeper/config.py\", line 4, in <module>\nE import tomli\nE ModuleNotFoundError: No module named 'tomli'\nE \nE assert '' == '30.00 EUR'\nE \nE - 30.00 EUR\n\n.bench_check/test_hidden.py:122: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_the_new_name_is_exported - ModuleNot...\nFAILED .bench_check/test_hidden.py::test_the_similar_names_were_left_alone - ...\nFAILED .bench_check/test_hidden.py::test_behaviour_is_unchanged - ModuleNotFo...\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass - Assert...\nFAILED .bench_check/test_hidden.py::test_the_command_line_still_works - Asser...\n5 failed, 2 passed in 0.53s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n____________________ ERROR collecting tests/test_config.py _____________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "refactor-rename", "task_hash": "696f6832ddf02216", "tokens_in": 42621, "tokens_out": 1975, "trimmed_requests": 0, "ts": "2026-09-27T23:48:50+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 16, "wall_s": 38.9}
{"backing": "openai/devstral-small-2-24b", "check_output": "ll_works _______________________\n\ntmp_path = PosixPath('/tmp/pytest-of-root/pytest-227/test_the_command_line_still_wo0')\n\n def test_the_command_line_still_works(tmp_path: Path) -> None:\n (tmp_path / \"catalog.csv\").write_text(\"sku,name,unit_price\\nSKU-2,hammer,12.50\\n\")\n ini = tmp_path / \"shop.ini\"\n ini.write_text(\n f\"[shop]\\ncurrency = EUR\\ntax_percent = 20\\ncatalog_file = {tmp_path / 'catalog.csv'}\\n\"\n )\n run = subprocess.run(\n [\n sys.executable,\n \"-m\",\n \"shopkeeper\",\n \"--config\",\n str(ini),\n \"price\",\n \"SKU-2\",\n \"2\",\n ],\n cwd=ROOT,\n capture_output=True,\n text=True,\n check=False,\n )\n> assert run.stdout.strip() == \"30.00 EUR\", run.stderr[-1000:]\nE AssertionError: Traceback (most recent call last):\nE File \"<frozen runpy>\", line 198, in _run_module_as_main\nE File \"<frozen runpy>\", line 88, in _run_code\nE File \"/tmp/tmpsbzdq0de/work/shopkeeper/__main__.py\", line 3, in <module>\nE raise SystemExit(main())\nE ^^^^^^\nE File \"/tmp/tmpsbzdq0de/work/shopkeeper/cli.py\", line 5, in main\nE parser = argparse.ArgumentParser(description=\"Shopkeeper CLI\")\nE ^^^^^^^^\nE NameError: name 'argparse' is not defined. Did you forget to import 'argparse'?\nE \nE assert '' == '30.00 EUR'\nE \nE - 30.00 EUR\n\n.bench_check/test_hidden.py:122: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_the_similar_names_were_left_alone - ...\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass - Assert...\nFAILED .bench_check/test_hidden.py::test_the_command_line_still_works - Asser...\n3 failed, 4 passed in 0.36s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n.....F [100%]\n=================================== FAILURES ===================================\n______________________________ test_shop_defaults ______________________________\n\nshop = PosixPath('/tmp/pytest-of-root/pytest-226/test_shop_defaults0/shop.ini')\n\n def test_shop_defaults(shop):\n settings = read_settings(shop)\n> assert settings.tax_percent == 0.0\nE", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "refactor-rename", "task_hash": "696f6832ddf02216", "tokens_in": 48882, "tokens_out": 2027, "trimmed_requests": 0, "ts": "2026-09-27T23:49:30+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 17, "wall_s": 39.8}
{"backing": "openai/devstral-small-2-24b", "check_output": "lse,\n )\n> assert run.returncode == 0, run.stdout[-1500:]\nE AssertionError: .....F.. [100%]\nE =================================== FAILURES ===================================\nE _____________________________ test_rule_6_decimals _____________________________\nE \nE def test_rule_6_decimals():\nE \"\"\"Test decimals (Rule 6).\"\"\"\nE assert parse_duration(\"1.5h\") == 5400\nE assert parse_duration(\"1.5m\") == 90\nE \nE # Invalid: decimal in non-first component\nE > with pytest.raises(ValueError, match=r\"invalid duration: '1h1.5m'\"):\nE E Failed: DID NOT RAISE ValueError\nE \nE test_durations.py:61: Failed\nE =========================== short test summary info ============================\nE FAILED test_durations.py::test_rule_6_decimals - Failed: DID NOT RAISE ValueE...\nE 1 failed, 7 passed in 0.02s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'test_durations.py'], returncode=1, stdou...D test_durations.py::test_rule_6_decimals - Failed: DID NOT RAISE ValueE...\\n1 failed, 7 passed in 0.02s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:87: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_invalid[.5h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_invalid[1.h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_invalid[+2h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_invalid[1h1.5m] - Failed: DID NOT RA...\nFAILED .bench_check/test_hidden.py::test_invalid[-1h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_the_plan_asked_for_tests_and_they_pass\n6 failed, 43 passed in 0.32s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n.....F.. [100%]\n=================================== FAILURES ===================================\n_____________________________ test_rule_6_decimals _____________________________\n\n def test_rule_6_decimals():\n \"\"\"Test decimals (Rule 6).\"\"\"\n assert parse_duration(\"1.5h\") == 5400\n assert parse_duration(\"1.5m\") == 90\n \n # Invalid: decimal in non", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "spec-parser", "task_hash": "bcead05857457203", "tokens_in": 20066, "tokens_out": 3800, "trimmed_requests": 0, "ts": "2026-09-27T23:50:43+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 72.3}
{"backing": "openai/devstral-small-2-24b", "check_output": " \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'test_durations.py'], returncode=1, stdou... test_durations.py::test_rule_8_zero_and_errors_invalid - AssertionErro...\\n10 failed, 6 passed in 0.05s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:87: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_valid[1h30m-5400] - AssertionError: ...\nFAILED .bench_check/test_hidden.py::test_valid[1d2h3m4s-93784] - AssertionErr...\nFAILED .bench_check/test_hidden.py::test_valid[1H30M-5400] - AssertionError: ...\nFAILED .bench_check/test_hidden.py::test_valid[2D-172800] - AssertionError: a...\nFAILED .bench_check/test_hidden.py::test_valid[1.5h-5400] - ValueError: inval...\nFAILED .bench_check/test_hidden.py::test_valid[1.5m-90] - ValueError: invalid...\nFAILED .bench_check/test_hidden.py::test_valid[0.25h-900] - ValueError: inval...\nFAILED .bench_check/test_hidden.py::test_invalid[.5h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_invalid[+2h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_invalid[30m1h] - Failed: DID NOT RAI...\nFAILED .bench_check/test_hidden.py::test_invalid[1h2h] - Failed: DID NOT RAIS...\nFAILED .bench_check/test_hidden.py::test_invalid[1s1m] - Failed: DID NOT RAIS...\nFAILED .bench_check/test_hidden.py::test_invalid[1h 30] - Failed: DID NOT RAI...\nFAILED .bench_check/test_hidden.py::test_invalid[30 1h] - Failed: DID NOT RAI...\nFAILED .bench_check/test_hidden.py::test_invalid[1:2] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_invalid[ ] - assert \"' '\" in \"in...\nFAILED .bench_check/test_hidden.py::test_invalid[-1h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_the_result_is_an_int - ValueError: i...\nFAILED .bench_check/test_hidden.py::test_the_plan_asked_for_tests_and_they_pass\n19 failed, 30 passed in 0.38s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nFFFFFF...FF..F.F [100%]\n=================================== FAILURES ===================================\n____________________________ test_rule_1_unit_form _____________________________\n\n def test_rule_1_unit_form():\n \"\"\"Test unit form with valid inputs.\"\"\"\n assert parse_duration(\"2h\") == 7200\n> assert parse_duration(\"1h30m\") == 5400\nE AssertionError: as", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "spec-parser", "task_hash": "bcead05857457203", "tokens_in": 20634, "tokens_out": 3918, "trimmed_requests": 0, "ts": "2026-09-27T23:51:58+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 74.6}
{"backing": "openai/devstral-small-2-24b", "check_output": "re diff\nE \nE tests/test_core.py:50: AssertionError\nE ______________________________ test_overdue_tasks ______________________________\nE \nE def test_overdue_tasks() -> None:\nE tasks: list[Task] = []\nE add_task(tasks, \"Task 1\", due=date(2026, 10, 1))\nE add_task(tasks, \"Task 2\", due=date(2026, 9, 1))\nE add_task(tasks, \"Task 3\", due=date(2026, 11, 1))\nE add_task(tasks, \"Task 4\")\nE complete(tasks, 2)\nE today = date(2026, 10, 5)\nE overdue_tasks = overdue(tasks, today)\nE > assert [t.id for t in overdue_tasks] == [2]\nE E assert [1] == [2]\nE E \nE E At index 0 diff: 1 != 2\nE E Use -v to get more diff\nE \nE tests/test_core.py:71: AssertionError\nE =========================== short test summary info ============================\nE FAILED tests/test_core.py::test_sort_tasks_by_due_date - assert [3, 2, 1, 4] ...\nE FAILED tests/test_core.py::test_overdue_tasks - assert [1] == [2]\nE 2 failed, 10 passed in 0.04s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'tests'], returncode=1, stdout='......F.F... 4] ...\\nFAILED tests/test_core.py::test_overdue_tasks - assert [1] == [2]\\n2 failed, 10 passed in 0.04s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:183: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_sort_order - assert [1, 7, 3, 6, 2, ...\nFAILED .bench_check/test_hidden.py::test_command_line_flow - AssertionError: ...\nFAILED .bench_check/test_hidden.py::test_today_defaults_to_the_real_date - As...\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass_and_cover_due_dates\n4 failed, 16 passed in 0.75s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n......F.F... [100%]\n=================================== FAILURES ===================================\n_________________________ test_sort_tasks_by_due_date __________________________\n\n def test_sort_tasks_by_due_date() -> None:\n tasks: list[Task] = []\n add_task(tasks, \"Task 1\", due=date(2026, 10, 1))\n add_task(tasks, \"Task 2\", due=date(2026, 9, 1))\n ", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "long-milestone", "task_hash": "ba436defff7d212f", "tokens_in": 71983, "tokens_out": 3137, "trimmed_requests": 0, "ts": "2026-09-27T23:53:04+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 22, "wall_s": 63.8}
{"backing": "openai/devstral-small-2-24b", "check_output": "_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \nE tracker/cli.py:37: in main\nE parser.error(f\"invalid --due date: {args.due}\")\nE /usr/lib/python3.12/argparse.py:2677: in error\nE self.exit(2, _('%(prog)s: error: %(message)s\\n') % args)\nE _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \nE \nE self = ArgumentParser(prog='tracker', usage=None, description=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True)\nE status = 2, message = 'tracker: error: invalid --due date: invalid-date\\n'\nE \nE def exit(self, status=0, message=None):\nE if message:\nE self._print_message(message, _sys.stderr)\nE > _sys.exit(status)\nE E SystemExit: 2\nE \nE /usr/lib/python3.12/argparse.py:2664: SystemExit\nE ----------------------------- Captured stderr call -----------------------------\nE usage: tracker [-h] [--file FILE] {add,list,done} ...\nE tracker: error: invalid --due date: invalid-date\nE =========================== short test summary info ============================\nE FAILED tests/test_storage_and_cli.py::test_cli_add_with_invalid_due_date - Sy...\nE 1 failed, 16 passed in 0.09s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'tests'], returncode=1, stdout='............ tests/test_storage_and_cli.py::test_cli_add_with_invalid_due_date - Sy...\\n1 failed, 16 passed in 0.09s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:183: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_sort_order - assert [1, 7, 3, 6, 2, ...\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass_and_cover_due_dates\n2 failed, 18 passed in 0.79s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n...............F. [100%]\n=================================== FAILURES ===================================\n______________________ test_cli_add_with_invalid_due_date ______________________\n\nargv = ['--file', '/tmp/pytest-of-root/pytest-247/test_cli_add_with_invalid_due_0/t.json', 'add', 'Pay rent', '--due', 'invalid-date']\n\n def main(argv: list[str] | None = None) -> int:\n pa", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "long-milestone", "task_hash": "ba436defff7d212f", "tokens_in": 94715, "tokens_out": 4535, "trimmed_requests": 0, "ts": "2026-09-27T23:54:37+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 25, "wall_s": 91.9}
{"backing": "openai/devstral-small-2-24b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly calculate the total value by multiplying each item's unit price by its quantity. The test now passes.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5215, "tokens_out": 442, "trimmed_requests": 0, "ts": "2026-09-27T23:54:47+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 9.5}
{"backing": "openai/devstral-small-2-24b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly calculate the total value by multiplying each item's unit price by its quantity. The test now passes.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 4865, "tokens_out": 280, "trimmed_requests": 0, "ts": "2026-09-27T23:54:53+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 6.2}
{"backing": "openai/devstral-small-2-24b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly calculate the total value by multiplying each item's unit price by its quantity and summing the results. The test now passes.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 4875, "tokens_out": 284, "trimmed_requests": 0, "ts": "2026-09-27T23:55:00+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 6.2}
{"backing": "openai/devstral-small-2-24b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly calculate the total value by multiplying each item's unit price by its quantity. The test now passes.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 4875, "tokens_out": 280, "trimmed_requests": 0, "ts": "2026-09-27T23:55:06+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 6.2}
{"backing": "openai/devstral-small-2-24b", "check_output": "...... [100%]\n6 passed in 0.01s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed the `total_value` function in `inventory.py` to correctly calculate the total value by multiplying each item's unit price by its quantity. The test now passes.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "fix-bug", "task_hash": "b78248aff00ddc39", "tokens_in": 5094, "tokens_out": 396, "trimmed_requests": 0, "ts": "2026-09-27T23:55:15+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 8.3}
{"backing": "openai/devstral-small-2-24b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 in rates.py from 0.21 to 0.12, which is the correct value calculated as (237 mod 25) / 100.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 27589, "tokens_out": 4485, "trimmed_requests": 0, "ts": "2026-09-27T23:56:44+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 6, "wall_s": 88.2}
{"backing": "openai/devstral-small-2-24b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmprrmv4d6x/work/rates.py\", line 3\nE RATES = {\nE ^\nE SyntaxError: '{' was never closed\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.12s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 16100, "tokens_out": 4140, "trimmed_requests": 0, "ts": "2026-09-27T23:58:06+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 81.6}
{"backing": "openai/devstral-small-2-24b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.20s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 in rates.py. The rate was updated from 0.21 to 0.12, which is the correct value (237 mod 25 = 12, so 12 / 100 = 0.12). All tests now pass.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 25605, "tokens_out": 4496, "trimmed_requests": 0, "ts": "2026-09-27T23:59:35+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 5, "wall_s": 88.6}
{"backing": "openai/devstral-small-2-24b", "check_output": "........................................................................ [ 23%]\n........................................................................ [ 47%]\n........................................................................ [ 71%]\n........................................................................ [ 95%]\n............. [100%]\n301 passed in 0.22s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed the rate for region code R237 in rates.py from 0.21 to 0.12, which is the correct value (237 mod 25 = 12, divided by 100). All tests now pass.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 101695, "tokens_out": 17396, "trimmed_requests": 0, "ts": "2026-09-28T00:05:27+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 351.2}
{"backing": "openai/devstral-small-2-24b", "check_output": "\n==================================== ERRORS ====================================\n_________________ ERROR collecting .bench_check/test_hidden.py _________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<frozen importlib._bootstrap>:1387: in _gcd_import\n ???\n<frozen importlib._bootstrap>:1360: in _find_and_load\n ???\n<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked\n ???\n<frozen importlib._bootstrap>:935: in _load_unlocked\n ???\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:188: in exec_module\n exec(co, module.__dict__)\n.bench_check/test_hidden.py:2: in <module>\n from rates import RATES\nE File \"/tmp/tmph16v7pxj/work/rates.py\", line 3\nE RATES = {\nE ^\nE SyntaxError: '{' was never closed\n=========================== short test summary info ============================\nERROR .bench_check/test_hidden.py\n!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!\n1 error in 0.12s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n________________________ ERROR collecting test_rates.py ________________________\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/python.py:508: in importtestmodule\n mod = import_path(\n/root/nisse/.venv/lib/python3.12/site-packages/_pytest/pathlib.py:596: in import_path\n importlib.import_module(module_name)\n/usr/lib/python3.12/importlib/__init__", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "long-output", "task_hash": "8cfbf73fb6620613", "tokens_in": 16093, "tokens_out": 4125, "trimmed_requests": 0, "ts": "2026-09-28T00:06:49+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 81.8}
Reference in new issue View Git Blame Copy Permalink