Record rate-limit waits in benchmark runs, and leave them out of time
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m17s

The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently.
A benchmark run that waited looked slower than the model is: on
2026-09-27, fast models under the unit-test key's 200k tokens/minute
limit had 20-45 s runs that were probably waits.

Both clients now report last_rate_limit_wait_s after every call, even
one that then fails. The benchmark sums it per run into
rate_limit_wait_s, the report's time column leaves it out, and a new
429 column counts the runs that waited. Results stored before this
load unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-09-27 20:57:15 +00:00
1 parent a716d5ec8d
commit 4e45dfe06a
6 files changed
+116 -9

No files matched your search

+4
View File
@@ -244,6 +244,10 @@ class BudgetedClient:
def last_cost(self) -> float:
return float(getattr(self._inner, "last_cost", 0.0) or 0.0)
@property
def last_rate_limit_wait_s(self) -> float:
return float(getattr(self._inner, "last_rate_limit_wait_s", 0.0) or 0.0)
def budget_for(self, model: str) -> Budget:
window, model_max_output = self._limits_for(model)
ceiling = self._max_output_tokens