Stream the chat route, so the timeout is on silence (#174)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 2m44s
CI / e2e (pull_request) Successful in 4m9s

A model call that passed request_timeout_s failed the whole run, though the
model was still writing. The chat route, which every local model takes, now
streams its replies and puts them back together into the response the
server would have sent, so the call record and parsing are unchanged.
request_timeout_s becomes the longest the model may go silent.

Measured on the 4090 through LiteLLM: an abandoned plain request kept the
server busy (a tiny request after it waited 51 s); an abandoned stream freed
it (0.3 s). A plain retry would have waited behind the abandoned call, so
there is none.

The passthrough route, hosted Anthropic models only, stays unstreamed: its
cost comes only in a header that streaming loses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-10-26 22:33:34 +00:00
1 parent 00ad9a4f92
commit 4904fa0b63
5 files changed
+374 -30

No files matched your search

+3 -2
View File
@@ -40,8 +40,9 @@ llm: # L9: LiteLLM is the only LLM destination
# Ceiling on each response. The model's own max_output_tokens in LiteLLM
# caps it further, which is where to hold a local model to less.
max_output_tokens: 16384
# How long one request may take before it is abandoned. A safety net for a
# hung connection; slow local models can take minutes over a long answer.
# How long the model may go silent before its request is abandoned. Local
# models' replies are streamed, so a slow one that keeps writing is never
# cut off; this bounds reading a long prompt, or a hung model.
request_timeout_s: 600
roles: # L9: role -> LiteLLM model alias