End-to-end tests on a local model, and in CI on every PR
- Every e2e role runs on qwen3.8-27b on the RTX 4090 (e2e_roles(); NISSE_E2E_MODEL overrides, e.g. claude-haiku-4-5). The first full run passed 7 of 8 in 26 minutes for $0.0000; the eighth was a real finding, fixed with the new-project PR. - scripts/e2e.sh reports a run's spend on the test key and the day's total against its budget: above $0 on the local model means a paid fallback. - CI runs the suite as its own job on pull requests (not pushes to main), on a throwaway sshd (--local), cancelling a run a newer push supersedes, so the checks job still reports in minutes. - The plan says so: e2e on every PR; locally only -k / --lf. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
1 parent
b3bf6d5a6e
commit
6db80dfb4b
6 files changed
+91
-9
No files matched your search
+30
-4
@@ -1,13 +1,16 @@
|
||||
#!/usr/bin/env bash
|
||||
# The end-to-end test (Development Plan M1.6): the whole agent against real
|
||||
# services, driven by a scripted operator. Run after each milestone; it
|
||||
# spends model tokens and takes minutes, so it is not part of check.sh or CI.
|
||||
# services, driven by a scripted operator. CI runs it on every PR (with
|
||||
# --local); locally, run only the tests you're writing (-k) or diagnosing
|
||||
# (--lf).
|
||||
#
|
||||
# scripts/e2e.sh on the bring-up build VM (configured below)
|
||||
# scripts/e2e.sh --local on a throwaway local sshd instead
|
||||
#
|
||||
# Uses tests/integration/nisse-test.yaml and its secrets (.env at the repo
|
||||
# root), like the integration tests.
|
||||
# root), like the integration tests. Every role runs on a local model
|
||||
# (tests/integration/e2e_support.py; NISSE_E2E_MODEL overrides it), so tell
|
||||
# the operator before a full run: it keeps the RTX 4090 busy for a while.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
@@ -19,4 +22,27 @@ else
|
||||
fi
|
||||
# Each run's state store is kept here, for nisse_dev.dump after a failure.
|
||||
export NISSE_E2E_RUNS=${NISSE_E2E_RUNS:-/root/nisse-e2e/test-runs}
|
||||
exec uv run pytest -m e2e -rs tests/integration/test_end_to_end.py tests/integration/test_end_to_end_loop.py "$@"
|
||||
|
||||
# What the run spent on the test key, read from LiteLLM (spend only). The
|
||||
# end-to-end tests run on a local model, so anything above $0 means a call
|
||||
# went to a paid one: an override, or a proxy fallback.
|
||||
spend() {
|
||||
(set -a; [ -f .env ] && . ./.env; set +a
|
||||
uv run python -c '
|
||||
import os, httpx
|
||||
r = httpx.get("https://litellm.dongar.net/key/info", timeout=30,
|
||||
headers={"Authorization": "Bearer " + os.environ["NISSE_LLM_TOKEN"]})
|
||||
i = r.json()["info"]
|
||||
print(i["spend"], i.get("max_budget"))' 2>/dev/null) || echo "? ?"
|
||||
}
|
||||
read -r before budget < <(spend)
|
||||
status=0
|
||||
uv run pytest -m e2e -rs tests/integration/test_end_to_end.py tests/integration/test_end_to_end_loop.py "$@" || status=$?
|
||||
read -r after _ < <(spend)
|
||||
python3 -c "
|
||||
b, a, cap = '$before', '$after', '$budget'
|
||||
try:
|
||||
print(f'e2e spend: \${float(a) - float(b):.4f} this run; \${float(a):.2f} of \${float(cap):.2f} today')
|
||||
except ValueError:
|
||||
print('e2e spend: unknown (could not read the key)')"
|
||||
exit "$status"
|
||||
Reference in new issue
Block a user