Benchmark seven new RTX 4090 models, and local-fast on the 4090 #49

2026-09-27T18:44:15Z

dev-agent commented

2026-09-27 18:44:15 +00:00

Results only: no code changes. These are executor benchmark runs of the seven models newly served on the RTX 4090, plus local-fast on the same card.

Results

All runs were at 5 runs per task, on llama.cpp b11206 through llama-swap, under a 350 W power cap. "Clean" counts runs that passed the hidden check and ended completed rather than escalating.

Model	Pass	Clean	Median s: fix / long / multi / write	Total time	Median output tokens
qwen3.6-35b-a3b (Q3_K_XL)	20/20	17	5 / 7 / 7 / 5	2.4 min	640
mimo-v2.6-9b (Q6_K)	20/20	20	4 / 45 / 9 / 4	4.0 min	483
gemma-4-26b-a4b (Q4_K_XL)	20/20	20	4 / 37 / 11 / 8	5.0 min	1028
qwen3.8-27b (Q4_K_XL)	20/20	20	14 / 16 / 36 / 13	6.9 min	628
gemma-4-31b (q4_0, 48k)	20/20	20	18 / 25 / 20 / 12	9.1 min	604
gemma-4-12b (q4_0)	20/20	20	11 / 110 / 22 / 8	11.4 min	842
granite-4.2-8b (Q6_K)	20/20	20	7 / 44 / 20 / 33	12.0 min	1913
local-fast, qwen3-8b-awq (vLLM 0.30.0)	3/20	3	1 / 3 / 5 / 3	2.1 min	216

Previous baseline (Sep 25): local-coder 12/20, local-fast 6/20.

Every new model passed 20 of 20. At 5 runs per task that only puts a floor of 57% on each task's true rate. The benchmark no longer separates good models; harder tasks are the next step.
qwen3.6-35b-a3b's three escalations were all in multi-file. Each time it finished the work, then replied in prose twice instead of calling the tool that reports completion. It isn't declared to support tool_choice, so the agent can't require a tool call.
Rare rambling runs are the main difference between models. granite wrote 24.5k tokens in one write-function run, and gemma-4-31b and qwen3.6 each had one run 5–10× longer than usual.
qwen3.8-27b wrote the least and never slowed down on long-output.
Every run was checked for fallbacks. Every run was answered by its own backing model, with no LiteLLM fallbacks.

How the runs were made

Power: at the default 450 W the card throttled for heat (84–88 °C, fan at 100%). All runs except one granite set are under a 350 W cap, at 76–82 °C with no throttling. That granite set, uncapped, is also 20/20 and is kept apart by its note.
Context window: each model's window was checked against llama.cpp's own n_ctx (from /props) instead of scripts/probe_context.py. The unittest key's 200k tokens/minute limit rejects the probe's over-long prompt before it reaches the server. All seven match their LiteLLM declarations.
Capability matrix: passed by all seven. The tool_choice test was skipped for qwen3.6-35b-a3b, mimo-v2.6-9b and granite-4.2-8b, which aren't declared to support it.
local-coder wasn't re-run. It failed the planner-contract test (answered in prose), and the operator dropped it along with the other old local model.

Caveats

Timings may include hidden rate-limit waits. When the client hits the key's 200k tokens/minute limit, it waits 20 s and retries without recording it. So some runs from the fast models, around 20–45 s, may include a wait. Success rates are unaffected.
One local-fast multi-file failure is a 429 that also failed on retry, not the model.

Follow-ups (not in this PR)

Probe bug: probe_context.py reports a LiteLLM 429 as the server refusing an over-long prompt. It should report that the result was inconclusive.
Record rate-limit retries in each benchmark result, so waits don't hide in the timings.
LiteLLM declarations: test tool_choice directly on qwen3.6-35b-a3b, mimo-v2.6-9b and granite-4.2-8b, and declare it where it works. Consider lowering gemma-4-31b's max_output_tokens.
Raise the unittest key's rate limit to about 400k/min, so the context probe can check windows of 128k and up.
Harder benchmark tasks, to separate models that all pass today's four.

🤖 Generated with Claude Code

Results only: no code changes. These are executor benchmark runs of the seven models newly served on the RTX 4090, plus `local-fast` on the same card. ## Results All runs were at 5 runs per task, on llama.cpp b11206 through llama-swap, under a 350 W power cap. "Clean" counts runs that passed the hidden check and ended `completed` rather than escalating. | Model | Pass | Clean | Median s: fix / long / multi / write | Total time | Median output tokens | |---|---|---|---|---|---| | qwen3.6-35b-a3b (Q3_K_XL) | 20/20 | 17 | 5 / 7 / 7 / 5 | 2.4 min | 640 | | mimo-v2.6-9b (Q6_K) | 20/20 | 20 | 4 / 45 / 9 / 4 | 4.0 min | 483 | | gemma-4-26b-a4b (Q4_K_XL) | 20/20 | 20 | 4 / 37 / 11 / 8 | 5.0 min | 1028 | | qwen3.8-27b (Q4_K_XL) | 20/20 | 20 | 14 / 16 / 36 / 13 | 6.9 min | 628 | | gemma-4-31b (q4_0, 48k) | 20/20 | 20 | 18 / 25 / 20 / 12 | 9.1 min | 604 | | gemma-4-12b (q4_0) | 20/20 | 20 | 11 / 110 / 22 / 8 | 11.4 min | 842 | | granite-4.2-8b (Q6_K) | 20/20 | 20 | 7 / 44 / 20 / 33 | 12.0 min | 1913 | | local-fast, qwen3-8b-awq (vLLM 0.30.0) | 3/20 | 3 | 1 / 3 / 5 / 3 | 2.1 min | 216 | Previous baseline (Sep 25): `local-coder` 12/20, `local-fast` 6/20. - **Every new model passed 20 of 20.** At 5 runs per task that only puts a floor of 57% on each task's true rate. The benchmark no longer separates good models; harder tasks are the next step. - **qwen3.6-35b-a3b's three escalations were all in multi-file.** Each time it finished the work, then replied in prose twice instead of calling the tool that reports completion. It isn't declared to support `tool_choice`, so the agent can't require a tool call. - **Rare rambling runs are the main difference between models.** granite wrote 24.5k tokens in one write-function run, and gemma-4-31b and qwen3.6 each had one run 5–10× longer than usual. - **qwen3.8-27b** wrote the least and never slowed down on long-output. - **Every run was checked for fallbacks.** Every run was answered by its own backing model, with no LiteLLM fallbacks. ## How the runs were made - **Power:** at the default 450 W the card throttled for heat (84–88 °C, fan at 100%). All runs except one granite set are under a 350 W cap, at 76–82 °C with no throttling. That granite set, uncapped, is also 20/20 and is kept apart by its note. - **Context window:** each model's window was checked against llama.cpp's own `n_ctx` (from `/props`) instead of `scripts/probe_context.py`. The unittest key's 200k tokens/minute limit rejects the probe's over-long prompt before it reaches the server. All seven match their LiteLLM declarations. - **Capability matrix:** passed by all seven. The `tool_choice` test was skipped for qwen3.6-35b-a3b, mimo-v2.6-9b and granite-4.2-8b, which aren't declared to support it. - **`local-coder` wasn't re-run.** It failed the planner-contract test (answered in prose), and the operator dropped it along with the other old local model. ## Caveats - **Timings may include hidden rate-limit waits.** When the client hits the key's 200k tokens/minute limit, it waits 20 s and retries without recording it. So some runs from the fast models, around 20–45 s, may include a wait. Success rates are unaffected. - **One `local-fast` multi-file failure** is a 429 that also failed on retry, not the model. ## Follow-ups (not in this PR) 1. **Probe bug:** `probe_context.py` reports a LiteLLM 429 as the server refusing an over-long prompt. It should report that the result was inconclusive. 2. **Record rate-limit retries** in each benchmark result, so waits don't hide in the timings. 3. **LiteLLM declarations:** test `tool_choice` directly on qwen3.6-35b-a3b, mimo-v2.6-9b and granite-4.2-8b, and declare it where it works. Consider lowering gemma-4-31b's `max_output_tokens`. 4. **Raise the unittest key's rate limit** to about 400k/min, so the context probe can check windows of 128k and up. 5. **Harder benchmark tasks**, to separate models that all pass today's four. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-27 18:44:15 +00:00

Benchmark seven new RTX 4090 models, and local-fast on the 4090

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m16s

Details

82f171c2f5

Five runs per task for qwen3.8-27b, qwen3.6-35b-a3b, gemma-4-31b,
gemma-4-26b-a4b, gemma-4-12b, mimo-v2.6-9b and granite-4.2-8b, all
served by llama.cpp b11206 through llama-swap. Every one passed 20 of
20. local-fast (vLLM 0.30.0) passed 3 of 20.

Runs were made under a 350 W power cap, recorded in each note, after
the uncapped card throttled for heat. granite-4.2-8b also has one
uncapped set, kept apart by its note.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dev-agent commented

2026-09-27 19:05:23 +00:00

Two follow-ups measured after the runs:

Context probe (after the unittest key's rate limit was raised to 400k/min): every model's own server refused a prompt at 1.5× its declared window, citing exactly that window. That's 131,072 for granite-4.2-8b, gemma-4-26b-a4b and qwen3.8-27b; 262,144 for gemma-4-12b, mimo-v2.6-9b and qwen3.6-35b-a3b; and 49,152 for gemma-4-31b. All seven declarations are confirmed. (The probe underestimates tokens by about 1.4–1.7×; the servers counted 279k–672k where it estimated 197k–393k.)

tool_choice, tested directly on llama-swap with the executor's tools and 3 samples per case: with "required" on a prompt that needs no tool, qwen3.6-35b-a3b (1/3), mimo-v2.6-9b (1/3) and granite-4.2-8b (0/3) mostly write prose until the 4096-token limit. That's qwen3-14b's bug. All three pass "required" when a tool is needed. The control model, gemma-4-26b-a4b, passes both. Their supports_tool_choice: false declarations are correct; leave them. Named tool choice isn't enforced by llama.cpp b11206 for any model tested, the control included, but the agent only asks for "required".

So qwen3.6's multi-file escalations (prose instead of the completion tool) can't be fixed through LiteLLM. The fix would be agent-side, e.g. a nudge after a prose reply.

🤖 Generated with Claude Code

Two follow-ups measured after the runs: **Context probe (after the unittest key's rate limit was raised to 400k/min):** every model's own server refused a prompt at 1.5× its declared window, citing exactly that window. That's 131,072 for granite-4.2-8b, gemma-4-26b-a4b and qwen3.8-27b; 262,144 for gemma-4-12b, mimo-v2.6-9b and qwen3.6-35b-a3b; and 49,152 for gemma-4-31b. All seven declarations are confirmed. (The probe underestimates tokens by about 1.4–1.7×; the servers counted 279k–672k where it estimated 197k–393k.) **`tool_choice`, tested directly on llama-swap** with the executor's tools and 3 samples per case: with `"required"` on a prompt that needs no tool, qwen3.6-35b-a3b (1/3), mimo-v2.6-9b (1/3) and granite-4.2-8b (0/3) mostly write prose until the 4096-token limit. That's qwen3-14b's bug. All three pass `"required"` when a tool is needed. The control model, gemma-4-26b-a4b, passes both. **Their `supports_tool_choice: false` declarations are correct; leave them.** Named tool choice isn't enforced by llama.cpp b11206 for any model tested, the control included, but the agent only asks for `"required"`. So qwen3.6's multi-file escalations (prose instead of the completion tool) can't be fixed through LiteLLM. The fix would be agent-side, e.g. a nudge after a prose reply. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dongar approved these changes

2026-09-27 20:30:59 +00:00

dev-agent merged commit a716d5ec8d into main

2026-09-27 20:31:46 +00:00

dev-agent deleted branch bench-4090

2026-09-27 20:31:46 +00:00

dev-agent referenced this pull request

2026-09-27 20:57:16 +00:00

Record rate-limit waits in benchmark runs, and leave them out of time #50

dev-agent referenced this pull request

2026-09-28 00:08:52 +00:00

Add four harder benchmark tasks #53