dongarandClaude Opus 5.5 4e45dfe06a Record rate-limit waits in benchmark runs, and leave them out of time ...

The client waits RATE_LIMIT_RETRY_S and retries once on a 429, silently.
A benchmark run that waited looked slower than the model is: on
2026-09-27, fast models under the unit-test key's 200k tokens/minute
limit had 20-45 s runs that were probably waits.

Both clients now report last_rate_limit_wait_s after every call, even
one that then fails. The benchmark sums it per run into
rate_limit_wait_s, the report's time column leaves it out, and a new
429 column counts the runs that waited. Results stored before this
load unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-09-27 20:57:15 +00:00

..

test_agent_build.py

Enforce the tool-call contract, asking servers only where it works

2026-09-24 22:54:49 +00:00

test_approvals.py

Externalise plans to Gitea and accept approvals there (A4, Part H)

2026-09-07 22:15:08 +00:00

test_bench.py

Record rate-limit waits in benchmark runs, and leave them out of time

2026-09-27 20:57:15 +00:00

test_canonical.py

Add ask_expert and eval capture (A14/A16) — step 7c

2026-09-08 19:48:13 +00:00

test_catalog.py

Discover a model's vendor from LiteLLM instead of its name

2026-09-23 01:47:58 +00:00

test_config.py

Fit every model request to its context window

2026-09-24 00:26:30 +00:00

test_context.py

Enforce the tool-call contract, asking servers only where it works

2026-09-24 22:54:49 +00:00

test_control.py

Add control loop, gating primitives, and transport interface (A4/B1)

2026-09-05 21:28:07 +00:00

test_delegation.py

Add the operator kill switch, stop (A4)

2026-09-23 22:47:29 +00:00

test_dev_main.py

Make the config file the only home for settings

2026-09-23 23:25:58 +00:00

test_evals.py

Add ask_expert and eval capture (A14/A16) — step 7c

2026-09-08 19:48:13 +00:00

test_executor.py

Enforce the tool-call contract, asking servers only where it works

2026-09-24 22:54:49 +00:00

test_expert.py

Add ask_expert and eval capture (A14/A16) — step 7c

2026-09-08 19:48:13 +00:00

test_gates.py

Add control loop, gating primitives, and transport interface (A4/B1)

2026-09-05 21:28:07 +00:00

test_git.py

Add Gitea client and git operations (B2) — step 6a, part 1

2026-09-07 20:27:34 +00:00

test_gitea.py

Make the config file the only home for settings

2026-09-23 23:25:58 +00:00

test_llm.py

Record rate-limit waits in benchmark runs, and leave them out of time

2026-09-27 20:57:15 +00:00

test_matrix.py

Make the config file the only home for settings

2026-09-23 23:25:58 +00:00

test_oplog.py

Add literal-format test showing exact log lines

2026-09-05 21:20:41 +00:00

test_planner.py

Enforce the token cap, and treat a spent budget as an interruption

2026-09-22 23:54:57 +00:00

test_poller.py

Add PR comment intake and the review loop (B2) — step 6b

2026-09-07 21:59:50 +00:00

test_pr_open.py

Publish finished milestones as PRs (B2/A3) — step 6a, part 2

2026-09-07 20:43:20 +00:00

test_probe.py

Enforce the tool-call contract, asking servers only where it works

2026-09-24 22:54:49 +00:00

test_publish.py

Publish finished milestones as PRs (B2/A3) — step 6a, part 2

2026-09-07 20:43:20 +00:00

test_reconcile.py

Implement A9 state-change logging in the state store

2026-09-05 21:13:36 +00:00

test_render_anthropic.py

Add the Anthropic passthrough renderer and client (A10/A13) — 7a

2026-09-07 23:13:22 +00:00

test_render.py

Add executor tool loop (A12/A15) — step 5b

2026-09-07 16:35:39 +00:00

test_review_loop.py

Add PR comment intake and the review loop (B2) — step 6b

2026-09-07 21:59:50 +00:00

test_state.py

Implement A9 state-change logging in the state store

2026-09-05 21:13:36 +00:00

test_stop.py

Add the operator kill switch, stop (A4)

2026-09-23 22:47:29 +00:00

test_tools.py

Fit every model request to its context window

2026-09-24 00:26:30 +00:00

test_worker.py

Add the operator kill switch, stop (A4)

2026-09-23 22:47:29 +00:00