Test levels: smoke e2e on PRs, the full set on e2e:full and before a release
- Every PR: lint, unit tests across the runner's cores (pytest-xdist), 16 property walks from a seed that moves with the run number, integration tests, and the end-to-end smoke tests (the slice, a conversation filed as work, a question; `smoke` marker). - A PR labelled e2e:full, a push to main, or a manual run: all 48 walks and the whole end-to-end suite. Adding e2e:full to a PR runs it; any other label runs nothing. - A published release runs the full set before its image is built. - The walk's seeds come from NISSE_WALK_SEEDS / NISSE_WALK_FIRST (48 from 0 by default). Measured here: the unit tier on 2 workers took 3m49s against 6-8 minutes serially. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
1 parent
07a2eb9d10
commit
e35aea5045
7 files changed
+124
-15
No files matched your search
@@ -73,10 +73,14 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
|
||||
`CLAUDE.md`. A milestone marked **needs discussion** starts with a design
|
||||
discussion first.
|
||||
- **Every change comes with tests,** through `scripts/check.sh`.
|
||||
- **The end-to-end test runs on every PR** (M1.6): the real agent, every role on a
|
||||
local model (so it costs nothing per call), a repository of the run's own and a
|
||||
real build host, driven by a scripted operator. CI runs it as its own job on pull
|
||||
requests, after the checks pass. There is one runner, and a run a newer push
|
||||
- **Testing has three levels** (M1.6): every PR runs the checks (unit tests across
|
||||
the runner's cores, 16 property walks from a moving seed, integration) and the
|
||||
end-to-end **smoke** tests; a PR labelled `e2e:full`, a push to `main` and a
|
||||
manual run add all 48 walks and the whole end-to-end suite; a published release
|
||||
runs that full set before its image is built. The end-to-end tests drive the
|
||||
real agent, every role on a local model (so they cost nothing per call), in a
|
||||
repository of the run's own, with a scripted operator; CI runs them after the
|
||||
checks pass. There is one runner, and a run a newer push
|
||||
supersedes has to be cancelled by hand. Locally only the tests being written
|
||||
(`-k`) or diagnosed (`--lf`); `scripts/e2e.sh` reports what a run spent. Manual
|
||||
live runs are for the new loop's first runs and phase ends. Findings are filed as
|
||||
|
||||
Reference in new issue
Block a user