End-to-end tests on a local model, and in CI on every PR
- Every e2e role runs on qwen3.8-27b on the RTX 4090 (e2e_roles(); NISSE_E2E_MODEL overrides, e.g. claude-haiku-4-5). The first full run passed 7 of 8 in 26 minutes for $0.0000; the eighth was a real finding, fixed with the new-project PR. - scripts/e2e.sh reports a run's spend on the test key and the day's total against its budget: above $0 on the local model means a paid fallback. - CI runs the suite as its own job on pull requests (not pushes to main), on a throwaway sshd (--local), cancelling a run a newer push supersedes, so the checks job still reports in minutes. - The plan says so: e2e on every PR; locally only -k / --lf. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
1 parent
b3bf6d5a6e
commit
6db80dfb4b
6 files changed
+91
-9
No files matched your search
@@ -11,6 +11,17 @@ from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from nisse_dev.gitea import GiteaClient, GiteaError
|
||||
from nisse_dev.llm import RoleMap
|
||||
|
||||
# The end-to-end tests run every role on a local model, which costs nothing
|
||||
# per call; NISSE_E2E_MODEL overrides it (e.g. claude-haiku-4-5 for a quick,
|
||||
# paid run).
|
||||
E2E_MODEL = "qwen3.8-27b"
|
||||
|
||||
|
||||
def e2e_roles() -> RoleMap:
|
||||
model = os.environ.get("NISSE_E2E_MODEL", "").strip() or E2E_MODEL
|
||||
return RoleMap(executor=model, planner=model, helper=model, expert=model)
|
||||
|
||||
|
||||
def task(run: str) -> str:
|
||||
|
||||
@@ -25,7 +25,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from e2e_support import fresh_repo, operator_gitea
|
||||
from e2e_support import e2e_roles, fresh_repo, operator_gitea
|
||||
from e2e_support import task as make_task
|
||||
|
||||
from nisse_dev.agent import Agent, build_agent
|
||||
@@ -84,6 +84,7 @@ def agent(
|
||||
workspace=Path(f"/tmp/nisse-e2e-{run}/sandbox"),
|
||||
build_host=build_host(ssh_host),
|
||||
),
|
||||
roles=e2e_roles(),
|
||||
state_db=tmp_path / "state.db",
|
||||
)
|
||||
transport = InMemoryTransport()
|
||||
|
||||
@@ -21,7 +21,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from e2e_support import dependent_task, fresh_repo, operator_gitea, task
|
||||
from e2e_support import dependent_task, e2e_roles, fresh_repo, operator_gitea, task
|
||||
|
||||
from nisse_dev.config import AgentConfig, LoopTuning
|
||||
from nisse_dev.execution import SSHHost
|
||||
@@ -74,6 +74,7 @@ def running(
|
||||
build_host=build_host(ssh_host),
|
||||
),
|
||||
loop="new",
|
||||
roles=e2e_roles(),
|
||||
state_db=keep / "state.db",
|
||||
# Short, so a bug's grace period runs out within the test.
|
||||
tuning=replace(live_config.tuning, loop=LoopTuning(bug_grace_s=BUG_GRACE_S)),
|
||||
|
||||
Reference in new issue
Block a user