dongarandClaude Opus 5.5
50b46821d7
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m11s
Choosing a local executor model, and tuning its knobs, needs measured success rates across repeated runs, comparable with every model measured before. - Four canned milestones under nisse_dev/bench/tasks: write a function, fix a bug a failing test exposes, thread an option through several files, and find one failure in output far past max_output_chars. Each is an approved plan over a seed project, judged by a hidden check the model never sees; a reference solution proves each is solvable and that its check tells the difference - Runs use the agent's own client wiring, now shared as make_llm_clients, with delegation and ask_expert off: an escalation is a failure of the model under test - Each run is stored as one JSON line in bench/results/, committed, with the model's backing name and declared facts, a note on how it was served, the task's content hash, the agent's commit and tuning, the outcome, turns, misbehaviour counts, context trims, tokens, time and LiteLLM's cost; a fallback answer can never count as a success - The report rebuilds the comparison from every stored run, with Wilson 95% intervals, marking runs against older task versions - scripts/benchmark.py runs and reports; scripts/evaluate_model.sh takes a new model through the context probe, the capability matrix and the benchmark in one command - Clients keep LiteLLM's per-call cost, and BudgetedClient counts the requests it had to trim - The tuning guide gains "Benchmarking models" Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
32 lines
1017 B
Python
32 lines
1017 B
Python
"""One live benchmark run: the simplest task, through the agent's own
|
|
client wiring, with the test config's executor."""
|
|
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from nisse_dev.agent import make_llm_clients
|
|
from nisse_dev.bench.runner import run_task
|
|
from nisse_dev.bench.tasks import load_tasks
|
|
from nisse_dev.catalog import ModelCatalog
|
|
from nisse_dev.config import AgentConfig
|
|
|
|
pytestmark = pytest.mark.integration
|
|
|
|
|
|
def test_the_simplest_task_passes_live(
|
|
live_config: AgentConfig, tmp_path: Path
|
|
) -> None:
|
|
catalog = ModelCatalog.discover(live_config.llm_base_url, live_config.llm_token)
|
|
normalized, _ = make_llm_clients(live_config, catalog)
|
|
facts = catalog.facts(live_config.roles.executor)
|
|
[task] = load_tasks(["write-function"])
|
|
|
|
record = run_task(task, facts, normalized, live_config.tuning, workdir=tmp_path)
|
|
|
|
assert record.fallbacks == []
|
|
assert record.success, (
|
|
f"{record.outcome}: {record.outcome_note}\n{record.check_output}"
|
|
)
|
|
assert record.cost >= 0
|