Wait out LLM outages with backoff (L5: outages delay, never fail)
A 502, 503 or 504 from the LiteLLM proxy, or a dropped connection, is retried after 5, 15 and 45 seconds before the call fails. Twice on 2026-10-07 a short burst of 502s stopped work outright: the integration tier at 16:39 and an e2e conversation at 20:43, where the item waited on the operator after one refused call. Not retried: a 500 (it may be the request's fault), a timeout (it already had its request_timeout_s), or a budget refusal. Rate limits keep their single 20-second retry. The time spent waiting is reported with the rate limit's (last_rate_limit_wait_s), so it isn't counted as the model's. Tests: an outage waited out, one that outlasts the backoff, a dropped connection, a 500 and a timeout not retried, a budget refusal not waited on. The offline agent in test_agent_build skips the backoff (its proxy doesn't exist). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
1 parent
2d6aa98f87
commit
ad8a07fcb7
3 files changed
+116
-8
No files matched your search
@@ -143,6 +143,9 @@ class FakeTransport:
|
||||
def build_offline(tmp_path: Path, origin: Path, monkeypatch: Any) -> Any:
|
||||
"""Assemble the real graph with only the network boundaries faked."""
|
||||
config = config_for(tmp_path)
|
||||
# The proxy here doesn't exist: a call to it fails at once rather than
|
||||
# waiting out an outage's backoff (L5).
|
||||
monkeypatch.setattr("nisse_dev.llm.OUTAGE_BACKOFF_S", ())
|
||||
|
||||
monkeypatch.setattr(
|
||||
"nisse_dev.agent.GiteaClient.whoami", lambda self: "nisse-dev-testing"
|
||||
|
||||
Reference in new issue
Block a user