Record rate-limit waits in benchmark runs, and leave them out of time
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m17s

The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently.
A benchmark run that waited looked slower than the model is: on
2026-09-27, fast models under the unit-test key's 200k tokens/minute
limit had 20-45 s runs that were probably waits.

Both clients now report last_rate_limit_wait_s after every call, even
one that then fails. The benchmark sums it per run into
rate_limit_wait_s, the report's time column leaves it out, and a new
429 column counts the runs that waited. Results stored before this
load unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-09-27 20:57:15 +00:00
1 parent a716d5ec8d
commit 4e45dfe06a
6 files changed
+116 -9

No files matched your search

+12 -1
View File
@@ -40,6 +40,7 @@ class MeteringClient:
self.tokens_out = 0
self.cost = 0.0
self.fallbacks: list[str] = []
self.rate_limit_wait_s = 0.0
def complete(
self,
@@ -49,7 +50,14 @@ class MeteringClient:
system: str | None = None,
) -> list[CanonicalEvent]:
self.calls += 1
produced = self._inner.complete(events, tools, model, system=system)
try:
produced = self._inner.complete(events, tools, model, system=system)
finally:
# Counted even when the call fails: a rate limit that never
# cleared still cost the run its wait.
self.rate_limit_wait_s += float(
getattr(self._inner, "last_rate_limit_wait_s", 0.0) or 0.0
)
usage = getattr(self._inner, "last_usage", {}) or {}
self.tokens_in += int(usage.get("input", 0))
self.tokens_out += int(usage.get("output", 0))
@@ -90,6 +98,8 @@ class RunRecord:
cost: float
wall_s: float
fallbacks: list[str] = field(default_factory=list)
# Part of wall_s spent waiting out the proxy's rate limit, not working.
rate_limit_wait_s: float = 0.0
check_output: str = ""
def to_json(self) -> dict[str, Any]:
@@ -214,5 +224,6 @@ def run_task(
cost=round(meter.cost, 6),
wall_s=round(wall, 1),
fallbacks=meter.fallbacks,
rate_limit_wait_s=meter.rate_limit_wait_s,
check_output=check_output,
)