Benchmark Qwen3.8-Flash-Next against qwen3.8-27b #180

2026-10-24T08:40:36Z

dev-agent commented

2026-10-24 08:40:36 +00:00

Benchmark results for Qwen3.8-Flash-Next against today's local model, qwen3.8-27b. Stacked on #177; retarget to main after #177 merges.

Five runs of each of the 8 tasks, through LiteLLM on the RTX 4090, 2026-10-23/24.

Model	Pass /40	Wall time	Prose turns	Notes
qwen3.8-27b (thinking)	36	131 min	0	all 4 failures on spec-parser
flash-next thinking, git workspaces	32	115 min	11	multi-file 2/5, spec-parser 2/5; 3 runs capped at 40 turns
flash-next lowthink, git workspaces	32	108 min	7	spec-parser 1/5; 4 capped, 4 escalated
flash-next thinking, old workspaces	22	110 min	6	10 failures were the missing git repo
flash-next lowthink, old workspaces	31	103 min	5	3 failures were the missing git repo
flash-next nothink, old workspaces	12	—	—	hits the turn cap over and over; not rerun

No run recorded an invented tool, and no LiteLLM fallback answered.

Review e2e (one run each, test_review_live):

qwen3.8-27b passed in 239s; qwen3.8-27b-nothink passed in 46s.
Flash-Next thinking failed: it wrote <tool_call> XML as text with invented tools (lint_fix, call), so its turns counted as prose and the review never posted.
Flash-Next nothink failed: invented tools, then about 20 duplicate "disregard" comments, and no review. This is the loop M7.8 (#178) addresses.

tool_choice: required was sent in every review turn, but it wasn't enforced there.

Reading it: the intervals overlap task by task, so 36 against 32 isn't decisive on five runs. Flash-Next is about 15% faster end to end. But the review failures, prose turns and capped runs all point the same way.

Recommendation: stay on qwen3.8-27b. Switching is the operator's call. If Flash-Next stays of interest, the next step is why tool_choice isn't enforced in the reviewer's requests, before more runs.

The first runs are kept: their task hash predates #177, so the report marks them (old).

🤖 Generated with Claude Code

https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

Benchmark results for Qwen3.8-Flash-Next against today's local model, qwen3.8-27b. Stacked on #177; retarget to main after #177 merges. Five runs of each of the 8 tasks, through LiteLLM on the RTX 4090, 2026-10-23/24. | Model | Pass /40 | Wall time | Prose turns | Notes | |---|---|---|---|---| | qwen3.8-27b (thinking) | **36** | 131 min | 0 | all 4 failures on spec-parser | | flash-next thinking, git workspaces | 32 | 115 min | 11 | multi-file 2/5, spec-parser 2/5; 3 runs capped at 40 turns | | flash-next lowthink, git workspaces | 32 | 108 min | 7 | spec-parser 1/5; 4 capped, 4 escalated | | flash-next thinking, old workspaces | 22 | 110 min | 6 | 10 failures were the missing git repo | | flash-next lowthink, old workspaces | 31 | 103 min | 5 | 3 failures were the missing git repo | | flash-next nothink, old workspaces | 12 | — | — | hits the turn cap over and over; not rerun | No run recorded an invented tool, and no LiteLLM fallback answered. **Review e2e** (one run each, `test_review_live`): - qwen3.8-27b passed in 239s; qwen3.8-27b-nothink passed in 46s. - Flash-Next thinking failed: it wrote `<tool_call>` XML as text with invented tools (`lint_fix`, `call`), so its turns counted as prose and the review never posted. - Flash-Next nothink failed: invented tools, then about 20 duplicate "disregard" comments, and no review. This is the loop M7.8 (#178) addresses. `tool_choice: required` was sent in every review turn, but it wasn't enforced there. **Reading it:** the intervals overlap task by task, so 36 against 32 isn't decisive on five runs. Flash-Next is about 15% faster end to end. But the review failures, prose turns and capped runs all point the same way. **Recommendation:** stay on qwen3.8-27b. Switching is the operator's call. If Flash-Next stays of interest, the next step is why `tool_choice` isn't enforced in the reviewer's requests, before more runs. The first runs are kept: their task hash predates #177, so the report marks them `(old)`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

dev-agent commented

2026-10-24 13:02:02 +00:00

Decision (operator, 2026-10-24): stay on qwen3.8-27b. No alias switch; the Flash-Next evaluation is closed.

dongar approved these changes

2026-10-24 13:07:32 +00:00

dev-agent changed target branch from bench-git-workspace to main

2026-10-24 13:09:22 +00:00

dev-agent added 1 commit 2026-10-24 13:09:22 +00:00

Benchmark Qwen3.8-Flash-Next against qwen3.8-27b

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 2m12s

Details

CI / e2e (pull_request) Successful in 3m56s

Details

3b614bd918

Five runs per task through LiteLLM on the RTX 4090. qwen3.8-27b (thinking)
passed 36 of 40. Flash-Next was run thinking, at reasoning_effort low, and
with no thinking.

The first Flash-Next runs used non-git workspaces, where `git log` fails,
so their task hashes are older and the report marks them (old). On git
workspaces (#177), both thinking and lowthink passed 32 of 40; nothink
passed 12 of 40 on the old workspaces and was not rerun. qwen3.8-27b had no
git-caused failures, so it was not rerun.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

dev-agent merged commit 0d16b5b7d2 into main

2026-10-24 13:10:31 +00:00

dev-agent deleted branch bench-flash-next

2026-10-24 13:10:31 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#180