End-to-end tests on a local model, and in CI on every PR
- Every e2e role runs on qwen3.8-27b on the RTX 4090 (e2e_roles(); NISSE_E2E_MODEL overrides, e.g. claude-haiku-4-5). The first full run passed 7 of 8 in 26 minutes for $0.0000; the eighth was a real finding, fixed with the new-project PR. - scripts/e2e.sh reports a run's spend on the test key and the day's total against its budget: above $0 on the local model means a paid fallback. - CI runs the suite as its own job on pull requests (not pushes to main), on a throwaway sshd (--local), cancelling a run a newer push supersedes, so the checks job still reports in minutes. - The plan says so: e2e on every PR; locally only -k / --lf. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
1 parent
b3bf6d5a6e
commit
6db80dfb4b
6 files changed
+91
-9
No files matched your search
@@ -73,9 +73,11 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
|
||||
`CLAUDE.md`. A milestone marked **needs discussion** starts with a design
|
||||
discussion first.
|
||||
- **Every change comes with tests,** through `scripts/check.sh`.
|
||||
- **The end-to-end test follows each milestone** (M1.6): the real agent, real models,
|
||||
a repository of the run's own and a real build host, driven by a scripted operator. It runs
|
||||
after each milestone, not on every push, since it spends model tokens. Manual
|
||||
- **The end-to-end test runs on every PR** (M1.6): the real agent, every role on a
|
||||
local model (so it costs nothing per call), a repository of the run's own and a
|
||||
real build host, driven by a scripted operator. CI runs it as its own job on pull
|
||||
requests, cancelling a run a newer push supersedes. Locally only the tests being
|
||||
written (`-k`) or diagnosed (`--lf`); `scripts/e2e.sh` reports what a run spent. Manual
|
||||
live runs are for the new loop's first runs and phase ends. Findings are filed as
|
||||
issues and either fixed before moving on or deliberately deferred.
|
||||
- **The Quint models move with the loop:** `orchestrator.qnt` while `control.py`
|
||||
|
||||
Reference in new issue
Block a user