Record rate-limit waits in benchmark runs, and leave them out of time #50

Merged

dev-agent merged 1 commits from record-rate-limit-retries into main

2026-09-27 21:23:57 +00:00

Author	SHA1	Message	Date
dongarandClaude Opus 5.5	4e45dfe06a	Record rate-limit waits in benchmark runs, and leave them out of time CI / checks (push) Skipped CI / checks (pull_request) Successful in 1m17s Details The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently. A benchmark run that waited looked slower than the model is: on 2026-09-27, fast models under the unit-test key's 200k tokens/minute limit had 20-45 s runs that were probably waits. Both clients now report last_rate_limit_wait_s after every call, even one that then fails. The benchmark sums it per run into rate_limit_wait_s, the report's time column leaves it out, and a new 429 column counts the runs that waited. Results stored before this load unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-27 20:57:15 +00:00

Author

SHA1

Message

Date

dongarandClaude Opus 5.5

4e45dfe06a

Record rate-limit waits in benchmark runs, and leave them out of time

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m17s

Details

The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently.
A benchmark run that waited looked slower than the model is: on
2026-09-27, fast models under the unit-test key's 200k tokens/minute
limit had 20-45 s runs that were probably waits.

Both clients now report last_rate_limit_wait_s after every call, even
one that then fails. The benchmark sums it per run into
rate_limit_wait_s, the report's time column leaves it out, and a new
429 column counts the runs that waited. Results stored before this
load unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-09-27 20:57:15 +00:00