Wait out LLM outages with backoff (L5) #116

2026-10-07T21:46:58Z

dev-agent commented

2026-10-07 21:46:58 +00:00

L5 says outages delay, never fail. Twice on 2026-10-07 a short burst of 502s from the LiteLLM proxy stopped work outright instead:

the integration tier at 16:39;
an e2e conversation at 20:43, where one refused call left the item waiting on the operator.

What changes

Retried with backoff: a 502, 503 or 504, or a dropped connection, waits 5, 15 and 45 seconds before the call fails (about a minute in all). This applies to both LLM clients, so the driver, reader, helpers and the old loop all get it.
Not retried:
- a 500, which may be the request's fault;
- a timeout, which already had its full request_timeout_s;
- a budget refusal, which won't clear by waiting.
Rate limits keep their single 20-second retry.
Waiting time is reported in last_rate_limit_wait_s, together with the rate-limit wait, so it isn't counted as the model's time.

Tests

New in test_llm.py: an outage waited out; one that outlasts the backoff and is reported; a dropped connection retried; a 500 and a timeout not retried; a budget refusal not waited on.
The offline agent in test_agent_build.py skips the backoff, since its proxy doesn't exist. Without that, one test took 130 seconds.
Unit (734) and integration (21) tiers pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

L5 says outages delay, never fail. Twice on 2026-10-07 a short burst of 502s from the LiteLLM proxy stopped work outright instead: - the integration tier at 16:39; - an e2e conversation at 20:43, where one refused call left the item waiting on the operator. ## What changes - **Retried with backoff:** a **502, 503 or 504**, or a **dropped connection**, waits 5, 15 and 45 seconds before the call fails (about a minute in all). This applies to both LLM clients, so the driver, reader, helpers and the old loop all get it. - **Not retried:** - a 500, which may be the request's fault; - a timeout, which already had its full `request_timeout_s`; - a budget refusal, which won't clear by waiting. - **Rate limits** keep their single 20-second retry. - **Waiting time** is reported in `last_rate_limit_wait_s`, together with the rate-limit wait, so it isn't counted as the model's time. ## Tests - New in `test_llm.py`: an outage waited out; one that outlasts the backoff and is reported; a dropped connection retried; a 500 and a timeout not retried; a budget refusal not waited on. - The offline agent in `test_agent_build.py` skips the backoff, since its proxy doesn't exist. Without that, one test took 130 seconds. - Unit (734) and integration (21) tiers pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dev-agent added 1 commit 2026-10-07 21:46:58 +00:00

Wait out LLM outages with backoff (L5: outages delay, never fail)

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 6m14s

Details

CI / e2e (pull_request) Successful in 30m8s

Details

ad8a07fcb7

A 502, 503 or 504 from the LiteLLM proxy, or a dropped connection, is
retried after 5, 15 and 45 seconds before the call fails. Twice on
2026-10-07 a short burst of 502s stopped work outright: the integration
tier at 16:39 and an e2e conversation at 20:43, where the item waited on
the operator after one refused call.

Not retried: a 500 (it may be the request's fault), a timeout (it already
had its request_timeout_s), or a budget refusal. Rate limits keep their
single 20-second retry. The time spent waiting is reported with the rate
limit's (last_rate_limit_wait_s), so it isn't counted as the model's.

Tests: an outage waited out, one that outlasts the backoff, a dropped
connection, a 500 and a timeout not retried, a budget refusal not waited
on. The offline agent in test_agent_build skips the backoff (its proxy
doesn't exist).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dongar approved these changes

2026-10-07 21:47:59 +00:00

dev-agent merged commit 07a2eb9d10 into main

2026-10-07 22:29:41 +00:00

dev-agent deleted branch llm-retry

2026-10-07 22:29:41 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#116