Part of #174: a slow model that is still writing no longer fails the run.
What changed
The chat route (every local model) asks for a streamed reply and puts the pieces back together into the response the server would have sent unstreamed: text, reasoning, tool calls joined from their pieces, finish reason, usage, and cost (LiteLLM sends it in the last piece's usage). Parsing and the call record are unchanged.
llm.request_timeout_s now bounds silence, not the whole reply: the wait for the next piece, including the first while the model reads its prompt.
Mid-stream: an error piece is upstream trouble (a budget one is BudgetExhausted); a stream that stops without finishing is upstream trouble; a lost connection is retried with the outage backoff, as before; a model gone quiet times out once, not retried.
The passthrough route (hosted Anthropic models only) stays unstreamed: when streamed, LiteLLM gives no cost for it.
A server that answers a streamed request with plain JSON is still read, so the existing fakes and any odd proxy keep working.
Why no retry (the plan's first idea): measured on the 4090 through LiteLLM, an abandoned plain request keeps the server busy until it finishes (a tiny request after it took 51 s, against 0.2 s idle). An abandoned stream frees it (0.3 s). A retry would have queued behind the abandoned call.
Checked live with the real client, a 10 s timeout: a 31 s qwen3.8-27b reply came through; a streamed tool call parsed; reasoning kept in the record; Haiku's cost still reported.
Tests: nisse-core/tests/test_llm.py, 11 new, two of them against a real local HTTP server (a slow steady reply outlasts the timeout; a quiet one times out). 8 mutations, all caught after dropping one unneeded check.
Docs: Tuning Guide llm.request_timeout_s, nisse-dev example YAML.
Not here: resuming a failed run from its transcript, and per-role timeouts. #174 stays open for those. The reviewer's bring-up config has request_timeout_s: 1200 from #174; it can go back to 600.
Part of #174: a slow model that is still writing no longer fails the run.
**What changed**
- The chat route (every local model) asks for a streamed reply and puts the pieces back together into the response the server would have sent unstreamed: text, reasoning, tool calls joined from their pieces, finish reason, usage, and cost (LiteLLM sends it in the last piece's usage). Parsing and the call record are unchanged.
- `llm.request_timeout_s` now bounds silence, not the whole reply: the wait for the next piece, including the first while the model reads its prompt.
- Mid-stream: an error piece is upstream trouble (a budget one is `BudgetExhausted`); a stream that stops without finishing is upstream trouble; a lost connection is retried with the outage backoff, as before; a model gone quiet times out once, not retried.
- The passthrough route (hosted Anthropic models only) stays unstreamed: when streamed, LiteLLM gives no cost for it.
- A server that answers a streamed request with plain JSON is still read, so the existing fakes and any odd proxy keep working.
**Why no retry** (the plan's first idea): measured on the 4090 through LiteLLM, an abandoned plain request keeps the server busy until it finishes (a tiny request after it took 51 s, against 0.2 s idle). An abandoned stream frees it (0.3 s). A retry would have queued behind the abandoned call.
**Checked live** with the real client, a 10 s timeout: a 31 s qwen3.8-27b reply came through; a streamed tool call parsed; reasoning kept in the record; Haiku's cost still reported.
**Tests**: `nisse-core/tests/test_llm.py`, 11 new, two of them against a real local HTTP server (a slow steady reply outlasts the timeout; a quiet one times out). 8 mutations, all caught after dropping one unneeded check.
**Docs**: Tuning Guide `llm.request_timeout_s`, nisse-dev example YAML.
**Not here**: resuming a failed run from its transcript, and per-role timeouts. #174 stays open for those. The reviewer's bring-up config has `request_timeout_s: 1200` from #174; it can go back to 600.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
A model call that passed request_timeout_s failed the whole run, though the
model was still writing. The chat route, which every local model takes, now
streams its replies and puts them back together into the response the
server would have sent, so the call record and parsing are unchanged.
request_timeout_s becomes the longest the model may go silent.
Measured on the 4090 through LiteLLM: an abandoned plain request kept the
server busy (a tiny request after it waited 51 s); an abandoned stream freed
it (0.3 s). A plain retry would have waited behind the abandoned call, so
there is none.
The passthrough route, hosted Anthropic models only, stays unstreamed: its
cost comes only in a header that streaming loses.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
dongar
Part of #174: a slow model that is still writing no longer fails the run.
What changed
llm.request_timeout_snow bounds silence, not the whole reply: the wait for the next piece, including the first while the model reads its prompt.BudgetExhausted); a stream that stops without finishing is upstream trouble; a lost connection is retried with the outage backoff, as before; a model gone quiet times out once, not retried.Why no retry (the plan's first idea): measured on the 4090 through LiteLLM, an abandoned plain request keeps the server busy until it finishes (a tiny request after it took 51 s, against 0.2 s idle). An abandoned stream frees it (0.3 s). A retry would have queued behind the abandoned call.
Checked live with the real client, a 10 s timeout: a 31 s qwen3.8-27b reply came through; a streamed tool call parsed; reasoning kept in the record; Haiku's cost still reported.
Tests:
nisse-core/tests/test_llm.py, 11 new, two of them against a real local HTTP server (a slow steady reply outlasts the timeout; a quiet one times out). 8 mutations, all caught after dropping one unneeded check.Docs: Tuning Guide
llm.request_timeout_s, nisse-dev example YAML.Not here: resuming a failed run from its transcript, and per-role timeouts. #174 stays open for those. The reviewer's bring-up config has
request_timeout_s: 1200from #174; it can go back to 600.🤖 Generated with Claude Code
https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk