Record rate-limit waits in benchmark runs, and leave them out of time
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m17s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m17s
The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently. A benchmark run that waited looked slower than the model is: on 2026-09-27, fast models under the unit-test key's 200k tokens/minute limit had 20-45 s runs that were probably waits. Both clients now report last_rate_limit_wait_s after every call, even one that then fails. The benchmark sums it per run into rate_limit_wait_s, the report's time column leaves it out, and a new 429 column counts the runs that waited. Results stored before this load unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
a716d5ec8d
commit
4e45dfe06a
6 files changed
+116
-9
No files matched your search
@@ -40,6 +40,7 @@ class MeteringClient:
|
||||
self.tokens_out = 0
|
||||
self.cost = 0.0
|
||||
self.fallbacks: list[str] = []
|
||||
self.rate_limit_wait_s = 0.0
|
||||
|
||||
def complete(
|
||||
self,
|
||||
@@ -49,7 +50,14 @@ class MeteringClient:
|
||||
system: str | None = None,
|
||||
) -> list[CanonicalEvent]:
|
||||
self.calls += 1
|
||||
produced = self._inner.complete(events, tools, model, system=system)
|
||||
try:
|
||||
produced = self._inner.complete(events, tools, model, system=system)
|
||||
finally:
|
||||
# Counted even when the call fails: a rate limit that never
|
||||
# cleared still cost the run its wait.
|
||||
self.rate_limit_wait_s += float(
|
||||
getattr(self._inner, "last_rate_limit_wait_s", 0.0) or 0.0
|
||||
)
|
||||
usage = getattr(self._inner, "last_usage", {}) or {}
|
||||
self.tokens_in += int(usage.get("input", 0))
|
||||
self.tokens_out += int(usage.get("output", 0))
|
||||
@@ -90,6 +98,8 @@ class RunRecord:
|
||||
cost: float
|
||||
wall_s: float
|
||||
fallbacks: list[str] = field(default_factory=list)
|
||||
# Part of wall_s spent waiting out the proxy's rate limit, not working.
|
||||
rate_limit_wait_s: float = 0.0
|
||||
check_output: str = ""
|
||||
|
||||
def to_json(self) -> dict[str, Any]:
|
||||
@@ -214,5 +224,6 @@ def run_task(
|
||||
cost=round(meter.cost, 6),
|
||||
wall_s=round(wall, 1),
|
||||
fallbacks=meter.fallbacks,
|
||||
rate_limit_wait_s=meter.rate_limit_wait_s,
|
||||
check_output=check_output,
|
||||
)
|
||||
Reference in new issue
Block a user