Record rate-limit waits in benchmark runs, and leave them out of time
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m17s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m17s
The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently. A benchmark run that waited looked slower than the model is: on 2026-09-27, fast models under the unit-test key's 200k tokens/minute limit had 20-45 s runs that were probably waits. Both clients now report last_rate_limit_wait_s after every call, even one that then fails. The benchmark sums it per run into rate_limit_wait_s, the report's time column leaves it out, and a new 429 column counts the runs that waited. Results stored before this load unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
a716d5ec8d
commit
4e45dfe06a
6 files changed
+116
-9
No files matched your search
@@ -244,6 +244,10 @@ class BudgetedClient:
|
||||
def last_cost(self) -> float:
|
||||
return float(getattr(self._inner, "last_cost", 0.0) or 0.0)
|
||||
|
||||
@property
|
||||
def last_rate_limit_wait_s(self) -> float:
|
||||
return float(getattr(self._inner, "last_rate_limit_wait_s", 0.0) or 0.0)
|
||||
|
||||
def budget_for(self, model: str) -> Budget:
|
||||
window, model_max_output = self._limits_for(model)
|
||||
ceiling = self._max_output_tokens
|
||||
|
||||
Reference in new issue
Block a user