dongarandClaude Opus 5.5
6ae2375194
nisse-dev now runs only the conversational loop. The gated state machine it was built alongside is gone: - Removed control.py, planner.py, executor.py, approvals.py, gates.py (command_form moves to loop/reader.py), publish.py and poller.py, the old assembly in agent.py (its shared helpers move to loop/build.py), the store's state-machine view (get_state, transition), their tests, the old end-to-end and live planning/milestone/gate tests, and specs/orchestrator.qnt with its scenarios. - Roles are driver, helper, expert and classifier; driver is required. loop:, roles.planner, roles.executor and tuning.executor are refused with what to use instead; tuning.driver takes the turn cap and failure limit. - The benchmark and the capability matrix measure the driver: the loop's own system prompt and tools, the task's plan as the agreed restatement. Task plans end with finish_milestone, so earlier runs show as (old); stored runs' executor tuning reads as the driver's. - A delegation on the loop records what its helper's alias resolved to (L9), as the old path's did. - The dump shows a work item's revision and what it waits for. - Behaviour the old tests covered and the loop shares stays covered: driver runs that fail on a model call, stop mid-turn, reset the failure count; go-aheads ignoring case and punctuation; a request too big to fit is never sent; the assembly tests, now on build_loop_agent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
32 lines
1.2 KiB
Bash
Executable File
32 lines
1.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Evaluate a model for the driver role, from a declared LiteLLM alias to a
|
|
# benchmark report next to every model measured before it.
|
|
#
|
|
# scripts/evaluate_model.sh ALIAS "how it is served" [RUNS] [--config PATH]
|
|
#
|
|
# 1. The context probe: does the server's window match what LiteLLM declares?
|
|
# 2. The capability matrix: does it hold the tool contract, and is its
|
|
# supports_tool_choice declaration true?
|
|
# 3. The benchmark: canned milestones, RUNS times each (default 5).
|
|
# Stops at the first step that fails; there is no point benchmarking a model
|
|
# that can't hold the tool contract.
|
|
set -euo pipefail
|
|
cd "$(dirname "$0")/.."
|
|
|
|
alias="${1:?usage: evaluate_model.sh ALIAS NOTE [RUNS] [--config PATH]}"
|
|
note="${2:?give a note on how the model is served, e.g. \"vLLM 0.29, AWQ\"}"
|
|
runs="${3:-5}"
|
|
shift $(( $# < 3 ? $# : 3 ))
|
|
config=("$@")
|
|
|
|
echo "== 1/3 context probe"
|
|
uv run python scripts/probe_context.py "$alias" "${config[@]}"
|
|
|
|
echo "== 2/3 capability matrix"
|
|
uv run pytest -m integration -q -rs tests/integration/test_model_capability.py \
|
|
--capability-models="$alias"
|
|
|
|
echo "== 3/3 benchmark ($runs runs per task)"
|
|
uv run python scripts/benchmark.py run --models "$alias" --runs "$runs" \
|
|
--note "$note" "${config[@]}"
|