Stream the chat route, so the timeout is on silence (#174)
A model call that passed request_timeout_s failed the whole run, though the model was still writing. The chat route, which every local model takes, now streams its replies and puts them back together into the response the server would have sent, so the call record and parsing are unchanged. request_timeout_s becomes the longest the model may go silent. Measured on the 4090 through LiteLLM: an abandoned plain request kept the server busy (a tiny request after it waited 51 s); an abandoned stream freed it (0.3 s). A plain retry would have waited behind the abandoned call, so there is none. The passthrough route, hosted Anthropic models only, stays unstreamed: its cost comes only in a header that streaming loses. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
This commit is contained in:
1 parent
00ad9a4f92
commit
4904fa0b63
5 files changed
+374
-30
No files matched your search
@@ -40,8 +40,9 @@ llm: # L9: LiteLLM is the only LLM destination
|
||||
# Ceiling on each response. The model's own max_output_tokens in LiteLLM
|
||||
# caps it further, which is where to hold a local model to less.
|
||||
max_output_tokens: 16384
|
||||
# How long one request may take before it is abandoned. A safety net for a
|
||||
# hung connection; slow local models can take minutes over a long answer.
|
||||
# How long the model may go silent before its request is abandoned. Local
|
||||
# models' replies are streamed, so a slow one that keeps writing is never
|
||||
# cut off; this bounds reading a long prompt, or a hung model.
|
||||
request_timeout_s: 600
|
||||
|
||||
roles: # L9: role -> LiteLLM model alias
|
||||
|
||||
Reference in new issue
Block a user