Record rate-limit waits in benchmark runs, and leave them out of time #50

2026-09-27T20:57:16Z

dev-agent commented

2026-09-27 20:57:16 +00:00

The fix promised in #49's follow-ups: rate-limit waits no longer hide in benchmark timings.

Why

On a 429, the client waits 20 s and retries once, and nothing records the wait. On 2026-09-27, fast models under the unit-test key's 200k tokens/minute limit had 20–45 s runs that were probably waits, not work. So the timing comparison in #49 is uncertain for those runs. Success rates weren't affected.

What changes

llm.py: both clients set last_rate_limit_wait_s after every call. It's set even when the call then fails, since a limit that never cleared still cost the run its wait, and reset per call.
context.py: BudgetedClient passes it through, like last_cost.
bench/runner.py: MeteringClient sums the waits, counting them even when the call raises. RunRecord gains rate_limit_wait_s, defaulting to 0.
bench/results.py: the report's time column now shows working time, leaving waits out. A new 429 column counts the runs in each row that waited, with a footnote explaining it.

Results stored before this change load unchanged: a missing field reads as 0.

Tests

test_llm.py: the wait is reported for a limit that clears and for one that doesn't, and it doesn't carry over to the next call.
test_bench.py: metering counts waits even when the call fails, and a run records its waits. The report leaves waits out of time and counts the runs that waited. Old results load unchanged.

CI steps run locally: ruff check, ruff format --check, and pytest -m "not integration" (403 passed).

🤖 Generated with Claude Code

The fix promised in #49's follow-ups: rate-limit waits no longer hide in benchmark timings. ## Why On a 429, the client waits 20 s and retries once, and nothing records the wait. On 2026-09-27, fast models under the unit-test key's 200k tokens/minute limit had 20–45 s runs that were probably waits, not work. So the timing comparison in #49 is uncertain for those runs. Success rates weren't affected. ## What changes - **`llm.py`:** both clients set `last_rate_limit_wait_s` after every call. It's set even when the call then fails, since a limit that never cleared still cost the run its wait, and reset per call. - **`context.py`:** `BudgetedClient` passes it through, like `last_cost`. - **`bench/runner.py`:** `MeteringClient` sums the waits, counting them even when the call raises. `RunRecord` gains `rate_limit_wait_s`, defaulting to 0. - **`bench/results.py`:** the report's **time** column now shows working time, leaving waits out. A new **429** column counts the runs in each row that waited, with a footnote explaining it. Results stored before this change load unchanged: a missing field reads as 0. ## Tests - **`test_llm.py`:** the wait is reported for a limit that clears and for one that doesn't, and it doesn't carry over to the next call. - **`test_bench.py`:** metering counts waits even when the call fails, and a run records its waits. The report leaves waits out of time and counts the runs that waited. Old results load unchanged. CI steps run locally: `ruff check`, `ruff format --check`, and `pytest -m "not integration"` (403 passed). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-27 20:57:16 +00:00

Record rate-limit waits in benchmark runs, and leave them out of time

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m17s

Details

4e45dfe06a

The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently.
A benchmark run that waited looked slower than the model is: on
2026-09-27, fast models under the unit-test key's 200k tokens/minute
limit had 20-45 s runs that were probably waits.

Both clients now report last_rate_limit_wait_s after every call, even
one that then fails. The benchmark sums it per run into
rate_limit_wait_s, the report's time column leaves it out, and a new
429 column counts the runs that waited. Results stored before this
load unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dongar approved these changes

2026-09-27 21:23:35 +00:00

dev-agent merged commit f41cbb00cf into main

2026-09-27 21:23:57 +00:00

dev-agent deleted branch record-rate-limit-retries

2026-09-27 21:23:57 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#50