Wait out LLM outages with backoff (L5: outages delay, never fail)

A 502, 503 or 504 from the LiteLLM proxy, or a dropped connection, is
retried after 5, 15 and 45 seconds before the call fails. Twice on
2026-10-07 a short burst of 502s stopped work outright: the integration
tier at 16:39 and an e2e conversation at 20:43, where the item waited on
the operator after one refused call.

Not retried: a 500 (it may be the request's fault), a timeout (it already
had its request_timeout_s), or a budget refusal. Rate limits keep their
single 20-second retry. The time spent waiting is reported with the rate
limit's (last_rate_limit_wait_s), so it isn't counted as the model's.

Tests: an outage waited out, one that outlasts the backoff, a dropped
connection, a 500 and a timeout not retried, a budget refusal not waited
on. The offline agent in test_agent_build skips the backoff (its proxy
doesn't exist).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

nisse-dev/src/nisse_dev/llm.py

@@ -39,6 +39,13 @@ class LLMError(RuntimeError):
 RATE_LIMIT_STATUS = 429
 RATE_LIMIT_RETRY_S = 20.0
 
+# The proxy, or what's behind it, briefly gone: a 502/503/504 or a dropped
+# connection. Outages delay, never fail (L5): waited out with backoff before
+# the work is stopped. Not a 500 (it may be the request) and not a timeout
+# (a hung model has had its request_timeout_s already).
+OUTAGE_STATUSES = frozenset({502, 503, 504})
+OUTAGE_BACKOFF_S = (5.0, 15.0, 45.0)
+
 # Generous by default. Output is billed by what is produced, not by the
 # ceiling, and a ceiling too low does not fail loudly — it truncates a tool
 # call mid-argument and hands back something that parses. Measured: Opus
@@ -381,20 +388,39 @@ class AnthropicPassthroughClient:
 def _post_with_rate_limit_retry(
     http: httpx.Client, path: str, body: dict[str, Any], sleep: Any
 ) -> tuple[httpx.Response, float]:
-    """The response, and how long was spent waiting out a rate limit."""
+    """The response, and how long was spent waiting: out a rate limit (once)
+    or an outage (with backoff), not on the model."""
     waited = 0.0
-    for attempt in (1, 2):
+    rate_limited = False
+    outages = iter(OUTAGE_BACKOFF_S)
+    while True:
         try:
             response = http.post(path, json=body)
+        except httpx.TimeoutException as exc:
+            raise UpstreamUnavailable(f"LLM request failed: {exc}") from exc
         except httpx.HTTPError as exc:
             # Could not reach the proxy at all: nothing about the request was
             # wrong, so callers holding work in progress should keep it.
-            raise UpstreamUnavailable(f"LLM request failed: {exc}") from exc
-        if response.status_code != RATE_LIMIT_STATUS or attempt == 2:
-            return response, waited
-        sleep(RATE_LIMIT_RETRY_S)
-        waited += RATE_LIMIT_RETRY_S
-    raise AssertionError("unreachable")
+            pause = next(outages, None)
+            if pause is None:
+                raise UpstreamUnavailable(f"LLM request failed: {exc}") from exc
+            sleep(pause)
+            waited += pause
+            continue
+        if response.status_code == RATE_LIMIT_STATUS and not rate_limited:
+            rate_limited = True
+            sleep(RATE_LIMIT_RETRY_S)
+            waited += RATE_LIMIT_RETRY_S
+            continue
+        if response.status_code in OUTAGE_STATUSES and not _looks_like_budget_refusal(
+            response.text
+        ):
+            pause = next(outages, None)
+            if pause is not None:
+                sleep(pause)
+                waited += pause
+                continue
+        return response, waited
 
 
 # Model families we have a native renderer adapter for. L9: delegations take