dongarandClaude Opus 5.5
6ae2375194
nisse-dev now runs only the conversational loop. The gated state machine it was built alongside is gone: - Removed control.py, planner.py, executor.py, approvals.py, gates.py (command_form moves to loop/reader.py), publish.py and poller.py, the old assembly in agent.py (its shared helpers move to loop/build.py), the store's state-machine view (get_state, transition), their tests, the old end-to-end and live planning/milestone/gate tests, and specs/orchestrator.qnt with its scenarios. - Roles are driver, helper, expert and classifier; driver is required. loop:, roles.planner, roles.executor and tuning.executor are refused with what to use instead; tuning.driver takes the turn cap and failure limit. - The benchmark and the capability matrix measure the driver: the loop's own system prompt and tools, the task's plan as the agreed restatement. Task plans end with finish_milestone, so earlier runs show as (old); stored runs' executor tuning reads as the driver's. - A delegation on the loop records what its helper's alias resolved to (L9), as the old path's did. - The dump shows a work item's revision and what it waits for. - Behaviour the old tests covered and the loop shares stays covered: driver runs that fail on a model call, stop mid-turn, reset the failure count; go-aheads ignoring case and punctuation; a request too big to fit is never sent; the assembly tests, now on build_loop_agent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
65 lines
2.8 KiB
Bash
Executable File
65 lines
2.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# The end-to-end test (Development Plan M1.6): the whole agent against real
|
|
# services, driven by a scripted operator. CI runs it on every PR (with
|
|
# --local); locally, run only the tests you're writing (-k) or diagnosing
|
|
# (--lf).
|
|
#
|
|
# scripts/e2e.sh on the bring-up build VM (configured below)
|
|
# scripts/e2e.sh --local on a throwaway local sshd instead
|
|
# scripts/e2e.sh --image REF inside an image's `test` stage (Dockerfile),
|
|
# on its own sshd: checks a release as built
|
|
#
|
|
# Uses tests/integration/nisse-test.yaml and its secrets (.env at the repo
|
|
# root), like the integration tests. Every role runs on a local model
|
|
# (tests/integration/e2e_support.py; NISSE_E2E_MODEL overrides it), so tell
|
|
# the operator before a full run: it keeps the RTX 4090 busy for a while.
|
|
set -euo pipefail
|
|
cd "$(dirname "$0")/.."
|
|
|
|
if [ "${1:-}" = "--image" ]; then
|
|
image=${2:?--image needs an image reference}
|
|
shift 2
|
|
runs=${NISSE_E2E_RUNS:-/root/nisse-e2e/test-runs}
|
|
mkdir -p "$runs"
|
|
secrets=()
|
|
[ -f .env ] && secrets=(--env-file .env)
|
|
# The model, if overridden here, reaches the tests inside too.
|
|
[ -n "${NISSE_E2E_MODEL:-}" ] && secrets+=(-e "NISSE_E2E_MODEL=$NISSE_E2E_MODEL")
|
|
# Host networking: the suite reaches Matrix, Gitea and LiteLLM as here.
|
|
exec docker run --rm --network host "${secrets[@]}" \
|
|
-e NISSE_E2E_RUNS=/runs -v "$runs:/runs" \
|
|
"$image" scripts/e2e.sh --local "$@"
|
|
fi
|
|
if [ "${1:-}" = "--local" ]; then
|
|
shift
|
|
else
|
|
export NISSE_E2E_BUILD_HOST=${NISSE_E2E_BUILD_HOST:-nisse-dev.garrett-hall.com}
|
|
export NISSE_E2E_BUILD_KEY=${NISSE_E2E_BUILD_KEY:-/root/nisse-e2e/ssh/id_ed25519}
|
|
fi
|
|
# Each run's state store is kept here, for nisse_dev.dump after a failure.
|
|
export NISSE_E2E_RUNS=${NISSE_E2E_RUNS:-/root/nisse-e2e/test-runs}
|
|
|
|
# What the run spent on the test key, read from LiteLLM (spend only). The
|
|
# end-to-end tests run on a local model, so anything above $0 means a call
|
|
# went to a paid one: an override, or a proxy fallback.
|
|
spend() {
|
|
(set -a; [ -f .env ] && . ./.env; set +a
|
|
uv run python -c '
|
|
import os, httpx
|
|
r = httpx.get("https://litellm.dongar.net/key/info", timeout=30,
|
|
headers={"Authorization": "Bearer " + os.environ["NISSE_LLM_TOKEN"]})
|
|
i = r.json()["info"]
|
|
print(i["spend"], i.get("max_budget"))' 2>/dev/null) || echo "? ?"
|
|
}
|
|
read -r before budget < <(spend)
|
|
status=0
|
|
uv run pytest -m e2e -rs tests/integration/test_end_to_end_loop.py "$@" || status=$?
|
|
read -r after _ < <(spend)
|
|
python3 -c "
|
|
b, a, cap = '$before', '$after', '$budget'
|
|
try:
|
|
print(f'e2e spend: \${float(a) - float(b):.4f} this run; \${float(a):.2f} of \${float(cap):.2f} today')
|
|
except ValueError:
|
|
print('e2e spend: unknown (could not read the key)')"
|
|
exit "$status"
|