A model call that passed request_timeout_s failed the whole run, though the
model was still writing. The chat route, which every local model takes, now
streams its replies and puts them back together into the response the
server would have sent, so the call record and parsing are unchanged.
request_timeout_s becomes the longest the model may go silent.
Measured on the 4090 through LiteLLM: an abandoned plain request kept the
server busy (a tiny request after it waited 51 s); an abandoned stream freed
it (0.3 s). A plain retry would have waited behind the abandoned call, so
there is none.
The passthrough route, hosted Anthropic models only, stays unstreamed: its
cost comes only in a header that streaming loses.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
dongarandClaude Opus 5.5