Stream the chat route, so the timeout is on silence (#174)
A model call that passed request_timeout_s failed the whole run, though the model was still writing. The chat route, which every local model takes, now streams its replies and puts them back together into the response the server would have sent, so the call record and parsing are unchanged. request_timeout_s becomes the longest the model may go silent. Measured on the 4090 through LiteLLM: an abandoned plain request kept the server busy (a tiny request after it waited 51 s); an abandoned stream freed it (0.3 s). A plain retry would have waited behind the abandoned call, so there is none. The passthrough route, hosted Anthropic models only, stays unstreamed: its cost comes only in a header that streaming loses. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
This commit is contained in:
1 parent
00ad9a4f92
commit
4904fa0b63
5 files changed
+374
-30
No files matched your search
+15
-10
@@ -135,18 +135,23 @@ a local model can be held to less without affecting the frontier roles.
|
||||
|
||||
### `llm.request_timeout_s` — default 600
|
||||
|
||||
How long one model request may take before it is abandoned. It applies to
|
||||
every request, on both routes.
|
||||
How long a model may go silent before its request is abandoned. On the chat
|
||||
route, which every local model takes, replies are streamed (#174): this is the
|
||||
longest wait for the next piece of a reply, including the wait for the first
|
||||
one while the model reads its prompt. A slow model that keeps writing is never
|
||||
cut off. On the passthrough route, which only hosted Anthropic models take,
|
||||
it is still the whole reply.
|
||||
|
||||
- **Too low:** a slow model's long answer is cut off, and the driver's run
|
||||
fails on work that was going well. At 120 s, the old fixed value, a dense
|
||||
27B model on the RTX 4090 timed out writing one module in one turn.
|
||||
- **Too high:** a hung connection holds the milestone for that long before
|
||||
anything notices.
|
||||
- **Too low:** a model reading a long prompt is cut off before it answers, and
|
||||
the run fails on work that was going well.
|
||||
- **Too high:** a hung model holds the run for that long before anything
|
||||
notices.
|
||||
- **Deciding:** this is a safety net, not a budget. `max_output_tokens` and
|
||||
`max_turns` bound the work; set this above the slowest single answer you
|
||||
expect, which is the model's output ceiling divided by its generation
|
||||
speed.
|
||||
`max_turns` bound the work. Set it above the longest the model takes to
|
||||
read its largest prompt (its context window divided by its prompt speed),
|
||||
and, for hosted Anthropic models, above their slowest whole answer.
|
||||
Abandoning a streamed request also frees the inference server; abandoning
|
||||
a plain one, measured on the 4090, left it finishing the reply anyway.
|
||||
|
||||
### `tuning.driver.max_turns` — default 40
|
||||
|
||||
|
||||
Reference in new issue
Block a user