Add an executor benchmark that keeps its results #47

2026-09-25T00:35:24Z

dev-agent commented

2026-09-25 00:35:24 +00:00

The last 4090-readiness change: an executor benchmark.

Tasks

Four canned milestones. Each is an approved plan over a seed project, judged by a hidden check the model never sees:

Task	Exercises
`write-function`	a small function and its tests
`fix-bug`	fix the bug a failing test exposes, without touching the test
`multi-file`	thread a `--shout` option through core, CLI and tests
`long-output`	one failure among 300 tests, with output far past `max_output_chars`

Each task also has a reference solution. The tests prove that every task is solvable, and that its check fails on the untouched seed.

How runs work

The agent's own client wiring, now shared as make_llm_clients(): the context budget, the declared limits, and tool_choice only where declared. Tuning comes from the config.
Delegation and ask_expert are off, so an escalation is a failure of the model under test.
A fallback answer can never count as a success.

Results are kept

Every run is one JSON line in bench/results/, which is committed. Each line records:

the model's backing name and declared LiteLLM facts;
a --note on how it was served;
the task's content hash;
the agent's commit and tuning;
the outcome, turns, prose turns, invented tools and failed commands;
how many requests the context budget trimmed;
tokens, wall time, and cost from LiteLLM.

report rebuilds the table from every stored run, with Wilson 95% intervals, and marks runs against older task versions (old). Benchmarking a new model means running only that model.

Commands

scripts/evaluate_model.sh ALIAS "how it's served" [RUNS]    # probe → capability matrix → benchmark
uv run python scripts/benchmark.py run --models A,B --runs 5 --note "..."
uv run python scripts/benchmark.py report [--models ...] [--tasks ...]

Shakedown (1 run each, not stored)

Task	`local-fast` (qwen3-8b, vLLM)	`local-coder` (qwen3-14b, llama.cpp)
write-function	fail (escalate)	pass
fix-bug	fail (escalate)	pass
multi-file	pass	pass
long-output	fail (escalate)	fail (escalate)

I read the transcripts, and the failures are genuine model mistakes, not harness bugs. For example, a test written without importing the function it tests, and .lower() applied after the substitution. The model was fixing its own mistakes but hit failure_limit: 2 partway through. One run proves nothing statistically, but it's a first reading on that knob. The guide's new "Benchmarking models" section covers how to read results.

Tests

Unit, 21:
- every task is solvable, and fails on the seed; hidden files never reach the seed;
- long-output really is long;
- task hashes change when a task does;
- success means the check passed, not the model's claim; misbehaviour counts; a fallback never succeeds;
- Wilson intervals, storage round trip, stale flagging, grouping by note, metering.
Live: write-function passes through the production wiring.
scripts/check.sh passes: 398 unit tests, and the integration suite with 19 passed and 7 skipped.

🤖 Generated with Claude Code

The last 4090-readiness change: an executor benchmark. ## Tasks Four canned milestones. Each is an approved plan over a seed project, judged by a **hidden check** the model never sees: | Task | Exercises | |---|---| | `write-function` | a small function and its tests | | `fix-bug` | fix the bug a failing test exposes, without touching the test | | `multi-file` | thread a `--shout` option through core, CLI and tests | | `long-output` | one failure among 300 tests, with output far past `max_output_chars` | Each task also has a reference solution. The tests prove that every task is solvable, and that its check fails on the untouched seed. ## How runs work - **The agent's own client wiring,** now shared as `make_llm_clients()`: the context budget, the declared limits, and `tool_choice` only where declared. Tuning comes from the config. - **Delegation and `ask_expert` are off**, so an escalation is a failure of the model under test. - **A fallback answer can never count as a success.** ## Results are kept Every run is one JSON line in `bench/results/`, which is committed. Each line records: - the model's backing name and declared LiteLLM facts; - a `--note` on how it was served; - the task's content hash; - the agent's commit and tuning; - the outcome, turns, prose turns, invented tools and failed commands; - how many requests the context budget trimmed; - tokens, wall time, and **cost from LiteLLM**. `report` rebuilds the table from every stored run, with Wilson 95% intervals, and marks runs against older task versions `(old)`. Benchmarking a new model means running only that model. ## Commands ``` scripts/evaluate_model.sh ALIAS "how it's served" [RUNS] # probe → capability matrix → benchmark uv run python scripts/benchmark.py run --models A,B --runs 5 --note "..." uv run python scripts/benchmark.py report [--models ...] [--tasks ...] ``` ## Shakedown (1 run each, not stored) | Task | `local-fast` (qwen3-8b, vLLM) | `local-coder` (qwen3-14b, llama.cpp) | |---|---|---| | write-function | fail (escalate) | pass | | fix-bug | fail (escalate) | pass | | multi-file | pass | pass | | long-output | fail (escalate) | fail (escalate) | I read the transcripts, and the failures are genuine model mistakes, not harness bugs. For example, a test written without importing the function it tests, and `.lower()` applied after the substitution. The model was fixing its own mistakes but hit `failure_limit: 2` partway through. One run proves nothing statistically, but it's a first reading on that knob. The guide's new "Benchmarking models" section covers how to read results. ## Tests - Unit, 21: - every task is solvable, and fails on the seed; hidden files never reach the seed; - `long-output` really is long; - task hashes change when a task does; - success means the check passed, not the model's claim; misbehaviour counts; a fallback never succeeds; - Wilson intervals, storage round trip, stale flagging, grouping by note, metering. - Live: `write-function` passes through the production wiring. - `scripts/check.sh` passes: 398 unit tests, and the integration suite with 19 passed and 7 skipped. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-25 00:35:25 +00:00

Add an executor benchmark that keeps its results

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m11s

Details

50b46821d7

Choosing a local executor model, and tuning its knobs, needs measured
success rates across repeated runs, comparable with every model
measured before.

- Four canned milestones under nisse_dev/bench/tasks: write a function,
  fix a bug a failing test exposes, thread an option through several
  files, and find one failure in output far past max_output_chars. Each
  is an approved plan over a seed project, judged by a hidden check the
  model never sees; a reference solution proves each is solvable and
  that its check tells the difference
- Runs use the agent's own client wiring, now shared as
  make_llm_clients, with delegation and ask_expert off: an escalation is
  a failure of the model under test
- Each run is stored as one JSON line in bench/results/, committed, with
  the model's backing name and declared facts, a note on how it was
  served, the task's content hash, the agent's commit and tuning, the
  outcome, turns, misbehaviour counts, context trims, tokens, time and
  LiteLLM's cost; a fallback answer can never count as a success
- The report rebuilds the comparison from every stored run, with Wilson
  95% intervals, marking runs against older task versions
- scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes
  a new model through the context probe, the capability matrix and the
  benchmark in one command
- Clients keep LiteLLM's per-call cost, and BudgetedClient counts the
  requests it had to trim
- The tuning guide gains "Benchmarking models"

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dev-agent commented

2026-09-25 01:25:14 +00:00

Baseline added: both local models, 5 runs per task, default tuning.

Task	`local-fast` (Qwen3-8B, vLLM)	`local-coder` (Qwen3-14B, llama.cpp)
write-function	3/5	5/5
multi-file	3/5	5/5
fix-bug	0/5	2/5
long-output	0/5	0/5

All 22 failures are escalations from failure_limit: 2. There were no prose turns, invented tools or context trims, and no run finished with a wrong result.

fix-bug and long-output start with a test run that fails by design, which spends one of the two allowed failures before any change. For those tasks the effective limit is 1.

Baseline added: both local models, 5 runs per task, default tuning. | Task | `local-fast` (Qwen3-8B, vLLM) | `local-coder` (Qwen3-14B, llama.cpp) | |---|---|---| | write-function | 3/5 | 5/5 | | multi-file | 3/5 | 5/5 | | fix-bug | 0/5 | 2/5 | | long-output | 0/5 | 0/5 | **All 22 failures are escalations from `failure_limit: 2`.** There were no prose turns, invented tools or context trims, and no run finished with a wrong result. `fix-bug` and `long-output` start with a test run that fails by design, which spends one of the two allowed failures before any change. For those tasks the effective limit is 1.

dev-agent added 1 commit 2026-09-25 01:25:16 +00:00

Add the local baseline: qwen3-8b-awq and qwen3-14b, 5 runs per task

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m14s

Details

0cccfe1e94

The 4070's two local models with default tuning, as the baseline the
4090's candidates will be compared with. Every one of the 22 failures
is an escalation from failure_limit: 2; there were no prose turns,
invented tools, context trims, or finished-but-wrong results.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dongar approved these changes

2026-09-25 02:23:54 +00:00

dongar merged commit 8a8d8dc27f into main

2026-09-25 02:23:58 +00:00

dongar deleted branch benchmark

2026-09-25 02:23:58 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#47