Stream the chat route, so the timeout is on silence (#174)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 2m44s
CI / e2e (pull_request) Successful in 4m9s

A model call that passed request_timeout_s failed the whole run, though the
model was still writing. The chat route, which every local model takes, now
streams its replies and puts them back together into the response the
server would have sent, so the call record and parsing are unchanged.
request_timeout_s becomes the longest the model may go silent.

Measured on the 4090 through LiteLLM: an abandoned plain request kept the
server busy (a tiny request after it waited 51 s); an abandoned stream freed
it (0.3 s). A plain retry would have waited behind the abandoned call, so
there is none.

The passthrough route, hosted Anthropic models only, stays unstreamed: its
cost comes only in a header that streaming loses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-10-26 22:33:34 +00:00
1 parent 00ad9a4f92
commit 4904fa0b63
5 files changed
+374 -30

No files matched your search

+15 -10
View File
@@ -135,18 +135,23 @@ a local model can be held to less without affecting the frontier roles.
### `llm.request_timeout_s` — default 600
How long one model request may take before it is abandoned. It applies to
every request, on both routes.
How long a model may go silent before its request is abandoned. On the chat
route, which every local model takes, replies are streamed (#174): this is the
longest wait for the next piece of a reply, including the wait for the first
one while the model reads its prompt. A slow model that keeps writing is never
cut off. On the passthrough route, which only hosted Anthropic models take,
it is still the whole reply.
- **Too low:** a slow model's long answer is cut off, and the driver's run
fails on work that was going well. At 120 s, the old fixed value, a dense
27B model on the RTX 4090 timed out writing one module in one turn.
- **Too high:** a hung connection holds the milestone for that long before
anything notices.
- **Too low:** a model reading a long prompt is cut off before it answers, and
the run fails on work that was going well.
- **Too high:** a hung model holds the run for that long before anything
notices.
- **Deciding:** this is a safety net, not a budget. `max_output_tokens` and
`max_turns` bound the work; set this above the slowest single answer you
expect, which is the model's output ceiling divided by its generation
speed.
`max_turns` bound the work. Set it above the longest the model takes to
read its largest prompt (its context window divided by its prompt speed),
and, for hosted Anthropic models, above their slowest whole answer.
Abandoning a streamed request also frees the inference server; abandoning
a plain one, measured on the 4090, left it finishing the reply anyway.
### `tuning.driver.max_turns` — default 40