dongarandClaude Opus 5.5
50b46821d7
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m11s
Choosing a local executor model, and tuning its knobs, needs measured success rates across repeated runs, comparable with every model measured before. - Four canned milestones under nisse_dev/bench/tasks: write a function, fix a bug a failing test exposes, thread an option through several files, and find one failure in output far past max_output_chars. Each is an approved plan over a seed project, judged by a hidden check the model never sees; a reference solution proves each is solvable and that its check tells the difference - Runs use the agent's own client wiring, now shared as make_llm_clients, with delegation and ask_expert off: an escalation is a failure of the model under test - Each run is stored as one JSON line in bench/results/, committed, with the model's backing name and declared facts, a note on how it was served, the task's content hash, the agent's commit and tuning, the outcome, turns, misbehaviour counts, context trims, tokens, time and LiteLLM's cost; a fallback answer can never count as a success - The report rebuilds the comparison from every stored run, with Wilson 95% intervals, marking runs against older task versions - scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes a new model through the context probe, the capability matrix and the benchmark in one command - Clients keep LiteLLM's per-call cost, and BudgetedClient counts the requests it had to trim - The tuning guide gains "Benchmarking models" Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
32 lines
1.2 KiB
Bash
Executable File
32 lines
1.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Evaluate a model for the executor role, from a declared LiteLLM alias to a
|
|
# benchmark report next to every model measured before it.
|
|
#
|
|
# scripts/evaluate_model.sh ALIAS "how it is served" [RUNS] [--config PATH]
|
|
#
|
|
# 1. The context probe: does the server's window match what LiteLLM declares?
|
|
# 2. The capability matrix: does it hold the tool contract, and is its
|
|
# supports_tool_choice declaration true?
|
|
# 3. The benchmark: canned milestones, RUNS times each (default 5).
|
|
# Stops at the first step that fails; there is no point benchmarking a model
|
|
# that can't hold the tool contract.
|
|
set -euo pipefail
|
|
cd "$(dirname "$0")/.."
|
|
|
|
alias="${1:?usage: evaluate_model.sh ALIAS NOTE [RUNS] [--config PATH]}"
|
|
note="${2:?give a note on how the model is served, e.g. \"vLLM 0.29, AWQ\"}"
|
|
runs="${3:-5}"
|
|
shift $(( $# < 3 ? $# : 3 ))
|
|
config=("$@")
|
|
|
|
echo "== 1/3 context probe"
|
|
uv run python scripts/probe_context.py "$alias" "${config[@]}"
|
|
|
|
echo "== 2/3 capability matrix"
|
|
uv run pytest -m integration -q -rs tests/integration/test_model_capability.py \
|
|
--capability-models="$alias"
|
|
|
|
echo "== 3/3 benchmark ($runs runs per task)"
|
|
uv run python scripts/benchmark.py run --models "$alias" --runs "$runs" \
|
|
--note "$note" "${config[@]}"
|