Render one assistant message per turn, and make the request timeout a setting #51

2026-09-27T22:15:31Z

dev-agent commented

2026-09-27 22:15:31 +00:00

Two agent bugs found while calibrating the new benchmark tasks. Both invalidated calibration runs, and both would fail real milestones too.

1. Strict chat templates rejected turns that start with text

Symptom: 10 of Devstral Small 2's runs failed with a 500 from llama.cpp: raise_exception('After the optional system message, conversation roles must alternate user/assistant…').

Cause: the OpenAI renderer made one message per event. When a model writes some text and then calls a tool in the same turn, that turn was sent as a text message followed by a tool-call message, two assistant messages in a row. Mistral's template rejects that. I captured the failing request from a real run; its tail was tool → assistant (text) → assistant (tool call) → tool.

Fix: render.py folds consecutive user or assistant messages into one: text is joined, and tool_calls are combined. That's the proper OpenAI shape for a turn; the Anthropic renderer already merges this way.

Verified live: with the fix, 6 real Devstral runs of fix-bug and long-output gave no 500s. Before it, 3 of 5 fix-bug runs and 5 of 5 long-output runs hit it.

2. A fixed 120 s request timeout

Symptom: 4 calibration runs "escalated" with The read operation timed out. Three were qwen3.8-27b, including both of its spec-parser runs, and one was mimo-v2.6-9b.

Cause: LiteLLMClient hardcoded timeout_s=120 (the passthrough client used 300). A dense 27B on the 4090 writing a whole module in one turn takes longer.

Fix: a new setting, llm.request_timeout_s, for both clients, default 600 s. It's a safety net for hung connections, not a work budget: max_output_tokens and max_turns bound the work. It's added to nisse-dev.example.yaml and the Tuning Guide.

Tests

test_render.py: text plus parallel calls from one turn give one assistant message. No transcript renders two user or two assistant messages in a row. The compaction test's expectation is updated, since a summary next to a user message now shares it.
test_config.py: the setting and its default, rejection of bad values, and that it reaches both clients.

CI steps run locally: ruff check, ruff format --check, and pytest -m "not integration" (407 passed).

Deferred

The benchmark still counts setup failures (500s, timeouts, 429s) against the model. Reporting them separately is a follow-up.

🤖 Generated with Claude Code

Two agent bugs found while calibrating the new benchmark tasks. Both invalidated calibration runs, and both would fail real milestones too. ## 1. Strict chat templates rejected turns that start with text **Symptom:** 10 of Devstral Small 2's runs failed with a 500 from llama.cpp: `raise_exception('After the optional system message, conversation roles must alternate user/assistant…')`. **Cause:** the OpenAI renderer made one message per event. When a model writes some text and then calls a tool in the same turn, that turn was sent as a text message followed by a tool-call message, two assistant messages in a row. Mistral's template rejects that. I captured the failing request from a real run; its tail was `tool → assistant (text) → assistant (tool call) → tool`. **Fix:** `render.py` folds consecutive user or assistant messages into one: text is joined, and `tool_calls` are combined. That's the proper OpenAI shape for a turn; the Anthropic renderer already merges this way. **Verified live:** with the fix, 6 real Devstral runs of `fix-bug` and `long-output` gave no 500s. Before it, 3 of 5 `fix-bug` runs and 5 of 5 `long-output` runs hit it. ## 2. A fixed 120 s request timeout **Symptom:** 4 calibration runs "escalated" with `The read operation timed out`. Three were qwen3.8-27b, including both of its `spec-parser` runs, and one was mimo-v2.6-9b. **Cause:** `LiteLLMClient` hardcoded `timeout_s=120` (the passthrough client used 300). A dense 27B on the 4090 writing a whole module in one turn takes longer. **Fix:** a new setting, `llm.request_timeout_s`, for both clients, **default 600 s**. It's a safety net for hung connections, not a work budget: `max_output_tokens` and `max_turns` bound the work. It's added to `nisse-dev.example.yaml` and the Tuning Guide. ## Tests - **`test_render.py`:** text plus parallel calls from one turn give one assistant message. No transcript renders two user or two assistant messages in a row. The compaction test's expectation is updated, since a summary next to a user message now shares it. - **`test_config.py`:** the setting and its default, rejection of bad values, and that it reaches both clients. CI steps run locally: `ruff check`, `ruff format --check`, and `pytest -m "not integration"` (407 passed). ## Deferred The benchmark still counts setup failures (500s, timeouts, 429s) against the model. Reporting them separately is a follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-27 22:15:31 +00:00

Render one assistant message per turn, and make the request timeout a setting

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m16s

Details

d49435d4e5

Two setup failures found while calibrating the new benchmark tasks,
both of which would also fail real milestones.

Devstral's chat template (Mistral) rejected any turn in which the model
wrote some text before a tool call: the OpenAI renderer made one message
per event, so the turn became a text message and a tool-call message in
a row, and llama.cpp answered 500. The renderer now folds consecutive
user or assistant messages into one, which is the OpenAI format for a
turn anyway. Checked against the live server: the captured failing
request no longer occurs, over six real runs.

LiteLLMClient gave up on any request after a fixed 120 s. A dense 27B
model writing a whole module in one turn takes longer, so the executor
escalated work that was going well. llm.request_timeout_s now sets it
for both clients, default 600.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dongar approved these changes

2026-09-27 22:17:53 +00:00

Dismissed

dongar approved these changes

2026-09-27 22:24:16 +00:00

dev-agent merged commit a77c84c128 into main

2026-09-27 22:24:50 +00:00

dev-agent deleted branch fix-transcript-and-timeout

2026-09-27 22:24:50 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#51