dev/nisse
2
1 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
03023b18b0b3fd88f87107e524917f5df9c16d02
nisse/bench/results/2026-09-28-devstral-small-2-24b.jsonl
T
dongarandClaude Opus 5.5 4b6c48cc96
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m12s
Details
Benchmark the finalists on all eight tasks, and archive older results 
Five runs per task at 320 W with max_turns 40, after the fixes in
#50-#52, for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-26b-a4b,
devstral-small-2-24b and mimo-v2.6-9b; the harder four also for
claude-haiku-4-5 as the ceiling, and two calibration runs each for
local-fast as the floor. Runs failed by the proxy's rate limiter were
set aside, not stored.

Earlier results move to bench/results/archive/, which the report does
not read: they were made at other power caps, at max_turns 24, or on
the RTX 4070, and are kept as the record.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 03:14:40 +00:00

23 lines
46 KiB
JSON
Raw Blame History

{"backing": "openai/devstral-small-2-24b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter. Updated core.py to support shouting, modified cli.py to accept the --shout flag, added tests for the shout functionality, and verified everything works correctly.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9422, "tokens_out": 507, "trimmed_requests": 0, "ts": "2026-09-28T00:07:00+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 10.7}
{"backing": "openai/devstral-small-2-24b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter. Updated core.py to support shouting, modified cli.py to accept the --shout flag, added tests for the new functionality, and verified everything works.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9440, "tokens_out": 506, "trimmed_requests": 0, "ts": "2026-09-28T00:07:11+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 10.3}
{"backing": "openai/devstral-small-2-24b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter. Updated core.py to support shouting, modified cli.py to accept the --shout flag, added tests for the new functionality, and verified everything works.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9440, "tokens_out": 506, "trimmed_requests": 0, "ts": "2026-09-28T00:07:21+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 10.3}
{"backing": "openai/devstral-small-2-24b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter. Updated core.py to support shout functionality, modified cli.py to accept the --shout flag, and added tests to verify the behavior. All tests pass and the command works as expected.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9431, "tokens_out": 513, "trimmed_requests": 0, "ts": "2026-09-28T00:07:32+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 10.4}
{"backing": "openai/devstral-small-2-24b", "check_output": "..... [100%]\n5 passed in 0.04s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Added --shout option to the greeter. Updated core.py to support shouting, modified cli.py to accept the --shout flag, added tests for the new functionality, and verified everything works.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "multi-file", "task_hash": "76bc0dcf36a72599", "tokens_in": 9431, "tokens_out": 506, "trimmed_requests": 0, "ts": "2026-09-28T00:07:43+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 9, "wall_s": 10.3}
{"backing": "openai/devstral-small-2-24b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Created `slugify.py` with the `slugify` function and `test_slugify.py` with tests. All tests pass.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3340, "tokens_out": 297, "trimmed_requests": 0, "ts": "2026-09-28T00:07:49+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 6.1}
{"backing": "openai/devstral-small-2-24b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Created `slugify.py` with the `slugify` function and `test_slugify.py` with tests. All tests pass.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3332, "tokens_out": 297, "trimmed_requests": 0, "ts": "2026-09-28T00:07:56+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 6.1}
{"backing": "openai/devstral-small-2-24b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Created slugify.py with the slugify function and test_slugify.py with four test cases. All tests pass.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3308, "tokens_out": 284, "trimmed_requests": 0, "ts": "2026-09-28T00:08:02+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 5.8}
{"backing": "openai/devstral-small-2-24b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Created `slugify.py` with the `slugify` function and `test_slugify.py` with tests. All tests pass.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3328, "tokens_out": 297, "trimmed_requests": 0, "ts": "2026-09-28T00:08:08+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 6.1}
{"backing": "openai/devstral-small-2-24b", "check_output": "........ [100%]\n8 passed in 0.01s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 0, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Created `slugify.py` with the `slugify` function and `test_slugify.py` with tests. All tests pass.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "write-function", "task_hash": "ec722fb0f389a6f4", "tokens_in": 3336, "tokens_out": 297, "trimmed_requests": 0, "ts": "2026-09-28T00:08:14+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 4, "wall_s": 6.0}
{"backing": "openai/devstral-small-2-24b", "check_output": ".......... [100%]\n10 passed in 0.02s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed precision loss in multi-currency invoice totals by removing unnecessary rate quantization in the `apply_rate` function. This ensures that exchange rates with up to six decimal places are used accurately in calculations.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "trace-bug", "task_hash": "a234c8fbeceeb166", "tokens_in": 15664, "tokens_out": 1074, "trimmed_requests": 0, "ts": "2026-09-28T01:52:34+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 61.1}
{"backing": "openai/devstral-small-2-24b", "check_output": ".......... [100%]\n10 passed in 0.02s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed the wrong totals on multi-currency invoices by increasing the precision of `RATE_PLACES` in `invoicing/money.py` from two to six decimal places. This prevents premature rounding of exchange rates, ensuring accurate currency conversions.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "trace-bug", "task_hash": "a234c8fbeceeb166", "tokens_in": 15466, "tokens_out": 920, "trimmed_requests": 0, "ts": "2026-09-28T01:52:53+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 18.7}
{"backing": "openai/devstral-small-2-24b", "check_output": ".......... [100%]\n10 passed in 0.02s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 1, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Fixed precision loss in multi-currency invoice totals by increasing the precision of rate normalization in `invoicing/money.py`. Changed `RATE_PLACES` from `Decimal(\"0.01\")` to `Decimal(\"0.000001\")` to preserve up to six decimal places in exchange rates, ensuring accurate currency conversions. All tests now pass.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "trace-bug", "task_hash": "a234c8fbeceeb166", "tokens_in": 15450, "tokens_out": 973, "trimmed_requests": 0, "ts": "2026-09-28T01:53:13+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 19.6}
{"backing": "openai/devstral-small-2-24b", "check_output": ":\n (tmp_path / \"catalog.csv\").write_text(\"sku,name,unit_price\\nSKU-2,hammer,12.50\\n\")\n ini = tmp_path / \"shop.ini\"\n ini.write_text(\n f\"[shop]\\ncurrency = EUR\\ntax_percent = 20\\ncatalog_file = {tmp_path / 'catalog.csv'}\\n\"\n )\n run = subprocess.run(\n [\n sys.executable,\n \"-m\",\n \"shopkeeper\",\n \"--config\",\n str(ini),\n \"price\",\n \"SKU-2\",\n \"2\",\n ],\n cwd=ROOT,\n capture_output=True,\n text=True,\n check=False,\n )\n> assert run.stdout.strip() == \"30.00 EUR\", run.stderr[-1000:]\nE AssertionError: Traceback (most recent call last):\nE File \"<frozen runpy>\", line 189, in _run_module_as_main\nE File \"<frozen runpy>\", line 148, in _get_module_details\nE File \"<frozen runpy>\", line 112, in _get_module_details\nE File \"/tmp/tmpaxiu4stj/work/shopkeeper/__init__.py\", line 1, in <module>\nE from shopkeeper.config import Settings, read_settings, load_config_defaults, reload_config\nE File \"/tmp/tmpaxiu4stj/work/shopkeeper/config.py\", line 4, in <module>\nE from pydantic import BaseModel, Field\nE ModuleNotFoundError: No module named 'pydantic'\nE \nE assert '' == '30.00 EUR'\nE \nE - 30.00 EUR\n\n.bench_check/test_hidden.py:122: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_the_new_name_is_exported - ModuleNot...\nFAILED .bench_check/test_hidden.py::test_the_similar_names_were_left_alone - ...\nFAILED .bench_check/test_hidden.py::test_behaviour_is_unchanged - ModuleNotFo...\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass - Assert...\nFAILED .bench_check/test_hidden.py::test_the_command_line_still_works - Asser...\n5 failed, 2 passed in 0.41s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n____________________ ERROR collecting tests/test_config.py _____________________\nImportError while importing test module '/tmp/tmpaxiu4stj/work/tests/test_config.py'.\nHint: make sure your test modules/packages have valid Python names.\nTraceback:\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], pa", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "refactor-rename", "task_hash": "696f6832ddf02216", "tokens_in": 43879, "tokens_out": 2266, "trimmed_requests": 0, "ts": "2026-09-28T01:53:58+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 16, "wall_s": 44.0}
{"backing": "openai/devstral-small-2-24b", "check_output": "orks(tmp_path: Path) -> None:\n (tmp_path / \"catalog.csv\").write_text(\"sku,name,unit_price\\nSKU-2,hammer,12.50\\n\")\n ini = tmp_path / \"shop.ini\"\n ini.write_text(\n f\"[shop]\\ncurrency = EUR\\ntax_percent = 20\\ncatalog_file = {tmp_path / 'catalog.csv'}\\n\"\n )\n run = subprocess.run(\n [\n sys.executable,\n \"-m\",\n \"shopkeeper\",\n \"--config\",\n str(ini),\n \"price\",\n \"SKU-2\",\n \"2\",\n ],\n cwd=ROOT,\n capture_output=True,\n text=True,\n check=False,\n )\n> assert run.stdout.strip() == \"30.00 EUR\", run.stderr[-1000:]\nE AssertionError: Traceback (most recent call last):\nE File \"<frozen runpy>\", line 189, in _run_module_as_main\nE File \"<frozen runpy>\", line 148, in _get_module_details\nE File \"<frozen runpy>\", line 112, in _get_module_details\nE File \"/tmp/tmpixv87ktq/work/shopkeeper/__init__.py\", line 1, in <module>\nE from shopkeeper.config import Settings, read_settings, load_config_defaults, reload_config\nE File \"/tmp/tmpixv87ktq/work/shopkeeper/config.py\", line 4, in <module>\nE import tomli\nE ModuleNotFoundError: No module named 'tomli'\nE \nE assert '' == '30.00 EUR'\nE \nE - 30.00 EUR\n\n.bench_check/test_hidden.py:122: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_the_new_name_is_exported - ModuleNot...\nFAILED .bench_check/test_hidden.py::test_the_similar_names_were_left_alone - ...\nFAILED .bench_check/test_hidden.py::test_behaviour_is_unchanged - ModuleNotFo...\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass - Assert...\nFAILED .bench_check/test_hidden.py::test_the_command_line_still_works - Asser...\n5 failed, 2 passed in 0.40s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 2\nstdout:\n\n==================================== ERRORS ====================================\n____________________ ERROR collecting tests/test_config.py _____________________\nImportError while importing test module '/tmp/tmpixv87ktq/work/tests/test_config.py'.\nHint: make sure your test modules/packages have valid Python names.\nTraceback:\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\n return _bootstrap._gcd_import(name[level:], pa", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "refactor-rename", "task_hash": "696f6832ddf02216", "tokens_in": 43644, "tokens_out": 2313, "trimmed_requests": 0, "ts": "2026-09-28T01:54:43+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 16, "wall_s": 44.8}
{"backing": "openai/devstral-small-2-24b", "check_output": "....... [100%]\n7 passed in 0.35s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "completed", "outcome_note": "Renamed `load_config` to `read_settings` across the project, including its definition, imports, calls, `__all__`, docstrings, comments, strings, and `README.md`. All tests pass.", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": true, "task": "refactor-rename", "task_hash": "696f6832ddf02216", "tokens_in": 156436, "tokens_out": 4031, "trimmed_requests": 0, "ts": "2026-09-28T01:56:04+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 31, "wall_s": 80.8}
{"backing": "openai/devstral-small-2-24b", "check_output": "with pytest.raises(ValueError, match=r\"invalid duration: '30m1h'\"):\nE E Failed: DID NOT RAISE ValueError\nE \nE test_durations.py:38: Failed\nE ______________________________ test_rule5_invalid ______________________________\nE \nE def test_rule5_invalid():\nE \"\"\"Test invalid bare number mixed with components (Rule 5)\"\"\"\nE > with pytest.raises(ValueError, match=r\"invalid duration: '1h 30'\"):\nE E Failed: DID NOT RAISE ValueError\nE \nE test_durations.py:60: Failed\nE =========================== short test summary info ============================\nE FAILED test_durations.py::test_rule3_invalid - Failed: DID NOT RAISE ValueError\nE FAILED test_durations.py::test_rule5_invalid - Failed: DID NOT RAISE ValueError\nE 2 failed, 14 passed in 0.03s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'test_durations.py'], returncode=1, stdou...D test_durations.py::test_rule5_invalid - Failed: DID NOT RAISE ValueError\\n2 failed, 14 passed in 0.03s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:87: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_invalid[.5h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_invalid[+2h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_invalid[30m1h] - Failed: DID NOT RAI...\nFAILED .bench_check/test_hidden.py::test_invalid[1s1m] - Failed: DID NOT RAIS...\nFAILED .bench_check/test_hidden.py::test_invalid[1h 30] - Failed: DID NOT RAI...\nFAILED .bench_check/test_hidden.py::test_invalid[30 1h] - Failed: DID NOT RAI...\nFAILED .bench_check/test_hidden.py::test_invalid[-1h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_the_plan_asked_for_tests_and_they_pass\n8 failed, 41 passed in 0.33s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n.....F...F...... [100%]\n=================================== FAILURES ===================================\n______________________________ test_rule3_invalid ______________________________\n\n def test_rule3_invalid():\n \"\"\"Test invalid order of units (Rule 3)\"\"\"\n> with pytest.raises(ValueError, match=r\"invalid duration: '30m1h'\"):\nE Failed: DID NOT RAISE ValueError\n", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "spec-parser", "task_hash": "bcead05857457203", "tokens_in": 27884, "tokens_out": 4648, "trimmed_requests": 0, "ts": "2026-09-28T01:57:34+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 8, "wall_s": 88.8}
{"backing": "openai/devstral-small-2-24b", "check_output": "s-93784] - IndexError: ...\nFAILED .bench_check/test_hidden.py::test_valid[45s-45] - ValueError: '5' is n...\nFAILED .bench_check/test_hidden.py::test_valid[1d-86400] - IndexError: string...\nFAILED .bench_check/test_hidden.py::test_valid[1H30M-5400] - IndexError: stri...\nFAILED .bench_check/test_hidden.py::test_valid[2D-172800] - IndexError: strin...\nFAILED .bench_check/test_hidden.py::test_valid[ 1h 30m -5400] - IndexError: s...\nFAILED .bench_check/test_hidden.py::test_valid[1d 2h 3m 4s-93784] - IndexErro...\nFAILED .bench_check/test_hidden.py::test_valid[1.5h-5400] - ValueError: '.' i...\nFAILED .bench_check/test_hidden.py::test_valid[1.5m-90] - ValueError: '.' is ...\nFAILED .bench_check/test_hidden.py::test_valid[0.25h-900] - ValueError: '.' i...\nFAILED .bench_check/test_hidden.py::test_valid[0s-0] - IndexError: string ind...\nFAILED .bench_check/test_hidden.py::test_invalid[.5h] - IndexError: string in...\nFAILED .bench_check/test_hidden.py::test_invalid[+2h] - IndexError: string in...\nFAILED .bench_check/test_hidden.py::test_invalid[30m1h] - IndexError: string ...\nFAILED .bench_check/test_hidden.py::test_invalid[1h2h] - IndexError: string i...\nFAILED .bench_check/test_hidden.py::test_invalid[1s1m] - IndexError: string i...\nFAILED .bench_check/test_hidden.py::test_invalid[1h 30] - IndexError: string ...\nFAILED .bench_check/test_hidden.py::test_invalid[30 1h] - IndexError: string ...\nFAILED .bench_check/test_hidden.py::test_invalid[1h1.5m] - IndexError: string...\nFAILED .bench_check/test_hidden.py::test_invalid[0.5s] - assert \"'0.5s'\" in \"...\nFAILED .bench_check/test_hidden.py::test_invalid[0.01m] - assert \"'0.01m'\" in...\nFAILED .bench_check/test_hidden.py::test_invalid[ ] - assert \"' '\" in \"in...\nFAILED .bench_check/test_hidden.py::test_invalid[-1h] - IndexError: string in...\nFAILED .bench_check/test_hidden.py::test_the_result_is_an_int - ValueError: '...\nFAILED .bench_check/test_hidden.py::test_the_plan_asked_for_tests_and_they_pass\n27 failed, 22 passed in 0.46s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nFFFFFFF..FFF..FF [100%]\n=================================== FAILURES ===================================\n____________________________ test_rule_1_unit_form _____________________________\n\n def test_rule_1_unit_form():\n \"\"\"Test unit form with valid inputs.\"\"\"\n> assert parse_duration(\"2h\") == 7200\n ^^^^^^^^^^^^^^^^^^^^\n\ntest_durations.py:9: \n_ _ _ _ _ _ _ ", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "spec-parser", "task_hash": "bcead05857457203", "tokens_in": 20066, "tokens_out": 3709, "trimmed_requests": 0, "ts": "2026-09-28T01:58:46+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 70.6}
{"backing": "openai/devstral-small-2-24b", "check_output": "ation: '0'\nE \nE durations.py:68: ValueError\nE =========================== short test summary info ============================\nE FAILED test_durations.py::test_rule_1_unit_form - ValueError: invalid duratio...\nE FAILED test_durations.py::test_rule_2_case - ValueError: invalid duration: '1...\nE FAILED test_durations.py::test_rule_4_spaces - ValueError: invalid duration: ...\nE FAILED test_durations.py::test_rule_5_bare_number - ValueError: invalid durat...\nE FAILED test_durations.py::test_rule_8_zero_and_errors - ValueError: invalid d...\nE 5 failed, 3 passed in 0.04s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'test_durations.py'], returncode=1, stdou...D test_durations.py::test_rule_8_zero_and_errors - ValueError: invalid d...\\n5 failed, 3 passed in 0.04s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:87: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_valid[1h30m-5400] - ValueError: inva...\nFAILED .bench_check/test_hidden.py::test_valid[1d2h3m4s-93784] - ValueError: ...\nFAILED .bench_check/test_hidden.py::test_valid[1H30M-5400] - ValueError: inva...\nFAILED .bench_check/test_hidden.py::test_valid[ 1h 30m -5400] - ValueError: i...\nFAILED .bench_check/test_hidden.py::test_valid[1d 2h 3m 4s-93784] - ValueErro...\nFAILED .bench_check/test_hidden.py::test_valid[90-90] - ValueError: invalid d...\nFAILED .bench_check/test_hidden.py::test_valid[ 7 -7] - ValueError: invalid...\nFAILED .bench_check/test_hidden.py::test_valid[0-0] - ValueError: invalid dur...\nFAILED .bench_check/test_hidden.py::test_invalid[1.h] - Failed: DID NOT RAISE...\nFAILED .bench_check/test_hidden.py::test_invalid[ ] - assert \"' '\" in \"in...\nFAILED .bench_check/test_hidden.py::test_the_plan_asked_for_tests_and_they_pass\n11 failed, 38 passed in 0.37s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\nFF.FF..F [100%]\n=================================== FAILURES ===================================\n____________________________ test_rule_1_unit_form _____________________________\n\n def test_rule_1_unit_form():\n \"\"\"Test Rule 1: Unit form.\"\"\"\n assert parse_duration(\"2h\") == 7200\n> assert parse_duration(\"1h30m\") == 5400\n ^^^^^^^^^^^^^^^^^^^^^", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "spec-parser", "task_hash": "bcead05857457203", "tokens_in": 19967, "tokens_out": 3247, "trimmed_requests": 0, "ts": "2026-09-28T01:59:48+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 7, "wall_s": 62.1}
{"backing": "openai/devstral-small-2-24b", "check_output": "class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True)}, required=True, help=None, metavar=None)\nE name = 'list', kwargs = {'help': 'list tasks', 'prog': 'tracker list'}\nE \nE def add_parser(self, name, **kwargs):\nE # set prog from the existing prefix\nE if kwargs.get('prog') is None:\nE kwargs['prog'] = '%s %s' % (self._prog_prefix, name)\nE \nE aliases = kwargs.pop('aliases', ())\nE \nE if name in self._name_parser_map:\nE > raise ArgumentError(self, _('conflicting subparser: %s') % name)\nE E argparse.ArgumentError: argument command: conflicting subparser: list\nE \nE /usr/lib/python3.12/argparse.py:1231: ArgumentError\nE =========================== short test summary info ============================\nE FAILED tests/test_storage_and_cli.py::test_cli_add_done_list - argparse.Argum...\nE 1 failed, 6 passed in 0.07s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'tests'], returncode=1, stdout='......F ...D tests/test_storage_and_cli.py::test_cli_add_done_list - argparse.Argum...\\n1 failed, 6 passed in 0.07s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:183: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_sort_order - assert [1, 7, 3, 6, 2, ...\nFAILED .bench_check/test_hidden.py::test_overdue - assert [2, 5] == [5, 2]\nFAILED .bench_check/test_hidden.py::test_command_line_flow - AssertionError: ...\nFAILED .bench_check/test_hidden.py::test_a_bad_due_date_is_a_usage_error - As...\nFAILED .bench_check/test_hidden.py::test_today_defaults_to_the_real_date - su...\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass_and_cover_due_dates\n6 failed, 14 passed in 0.68s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n......F [100%]\n=================================== FAILURES ===================================\n____________________________ test_cli_add_done_list ____________________________\n\ntmp_path = PosixPath('/tmp/pytest-of-root/pytest-353/test_cli_add_done_list0')\ncapsys = <_pytest.capture.CaptureFixture object at 0x792d46bff710>\n\n def test_cli_add_done_list(tmp_path: Path, caps", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "long-milestone", "task_hash": "ba436defff7d212f", "tokens_in": 38777, "tokens_out": 1883, "trimmed_requests": 0, "ts": "2026-09-28T02:00:29+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 16, "wall_s": 39.3}
{"backing": "openai/devstral-small-2-24b", "check_output": "class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True)}, required=True, help=None, metavar=None)\nE name = 'list', kwargs = {'help': 'list tasks', 'prog': 'tracker list'}\nE \nE def add_parser(self, name, **kwargs):\nE # set prog from the existing prefix\nE if kwargs.get('prog') is None:\nE kwargs['prog'] = '%s %s' % (self._prog_prefix, name)\nE \nE aliases = kwargs.pop('aliases', ())\nE \nE if name in self._name_parser_map:\nE > raise ArgumentError(self, _('conflicting subparser: %s') % name)\nE E argparse.ArgumentError: argument command: conflicting subparser: list\nE \nE /usr/lib/python3.12/argparse.py:1231: ArgumentError\nE =========================== short test summary info ============================\nE FAILED tests/test_storage_and_cli.py::test_cli_add_done_list - argparse.Argum...\nE 1 failed, 6 passed in 0.08s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'tests'], returncode=1, stdout='......F ...D tests/test_storage_and_cli.py::test_cli_add_done_list - argparse.Argum...\\n1 failed, 6 passed in 0.08s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:183: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_sort_order - assert [1, 7, 3, 6, 2, ...\nFAILED .bench_check/test_hidden.py::test_overdue - assert [2, 5] == [5, 2]\nFAILED .bench_check/test_hidden.py::test_command_line_flow - AssertionError: ...\nFAILED .bench_check/test_hidden.py::test_a_bad_due_date_is_a_usage_error - As...\nFAILED .bench_check/test_hidden.py::test_today_defaults_to_the_real_date - su...\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass_and_cover_due_dates\n6 failed, 14 passed in 0.74s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 2, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n......F [100%]\n=================================== FAILURES ===================================\n____________________________ test_cli_add_done_list ____________________________\n\ntmp_path = PosixPath('/tmp/pytest-of-root/pytest-361/test_cli_add_done_list0')\ncapsys = <_pytest.capture.CaptureFixture object at 0x71a1d30db6e0>\n\n def test_cli_add_done_list(tmp_path: Path, caps", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "long-milestone", "task_hash": "ba436defff7d212f", "tokens_in": 38701, "tokens_out": 1871, "trimmed_requests": 0, "ts": "2026-09-28T02:01:10+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 16, "wall_s": 39.3}
{"backing": "openai/devstral-small-2-24b", "check_output": " __________________________ test_sort_tasks_done_last ___________________________\nE \nE def test_sort_tasks_done_last() -> None:\nE tasks: list[Task] = []\nE add_task(tasks, \"Task 1\", due=date(2026, 10, 1))\nE add_task(tasks, \"Task 2\", due=date(2026, 9, 1))\nE add_task(tasks, \"Task 3\")\nE add_task(tasks, \"Task 4\", due=date(2026, 11, 1))\nE complete(tasks, 2)\nE sorted_tasks = sort_tasks(tasks)\nE > assert [t.id for t in sorted_tasks] == [1, 4, 3, 2]\nE E assert [3, 1, 4, 2] == [1, 4, 3, 2]\nE E \nE E At index 0 diff: 3 != 1\nE E Use -v to get more diff\nE \nE tests/test_core.py:61: AssertionError\nE =========================== short test summary info ============================\nE FAILED tests/test_core.py::test_sort_tasks_by_due_date - assert [3, 2, 1, 4] ...\nE FAILED tests/test_core.py::test_sort_tasks_done_last - assert [3, 1, 4, 2] ==...\nE 2 failed, 10 passed in 0.04s\nE \nE assert 1 == 0\nE + where 1 = CompletedProcess(args=['/root/nisse/.venv/bin/python', '-m', 'pytest', '-q', 'tests'], returncode=1, stdout='......FF.... tests/test_core.py::test_sort_tasks_done_last - assert [3, 1, 4, 2] ==...\\n2 failed, 10 passed in 0.04s\\n', stderr='').returncode\n\n.bench_check/test_hidden.py:183: AssertionError\n=========================== short test summary info ============================\nFAILED .bench_check/test_hidden.py::test_sort_order - assert [1, 7, 3, 6, 2, ...\nFAILED .bench_check/test_hidden.py::test_overdue - assert [2, 5] == [5, 2]\nFAILED .bench_check/test_hidden.py::test_command_line_flow - AssertionError: ...\nFAILED .bench_check/test_hidden.py::test_today_defaults_to_the_real_date - As...\nFAILED .bench_check/test_hidden.py::test_the_projects_own_tests_pass_and_cover_due_dates\n5 failed, 15 passed in 0.94s\n", "commit": "277bff6+dirty", "cost": 0.0, "failed_commands": 3, "fallbacks": [], "invented_tools": 0, "model": "devstral-small-2-24b", "model_info": {"max_input_tokens": 98304, "max_output_tokens": 32768, "supports_function_calling": true, "supports_tool_choice": false}, "note": "llama.cpp b11206, Q4_K_XL, q8_0 KV, 96k, temp 0.15, RTX 4090, 320 W cap", "outcome": "escalate", "outcome_note": "2 consecutive command failures; last: exit code: 1\nstdout:\n......FF.... [100%]\n=================================== FAILURES ===================================\n_________________________ test_sort_tasks_by_due_date __________________________\n\n def test_sort_tasks_by_due_date() -> None:\n tasks: list[Task] = []\n add_task(tasks, \"Task 1\", due=date(2026, 10, 1))\n add_task(tasks, \"Task 2\", due=date(2026, 9, 1))\n ", "prose_turns": 0, "provider": "openai", "rate_limit_wait_s": 0.0, "success": false, "task": "long-milestone", "task_hash": "ba436defff7d212f", "tokens_in": 71811, "tokens_out": 3151, "trimmed_requests": 0, "ts": "2026-09-28T02:02:16+00:00", "tuning": {"context": {"keep_recent_turns": 6, "safety_margin": 0.1}, "delegation": {"max_tokens": 200000, "max_turns": 30, "wall_clock_s": 900.0}, "executor": {"failure_limit": 2, "max_turns": 40}, "expert": {"consult_budget": 3}, "tools": {"command_timeout_s": 120, "max_output_chars": 4000}}, "turns": 22, "wall_s": 65.0}
Reference in new issue View Git Blame Copy Permalink