Test levels: smoke e2e on PRs, the full set on e2e:full and before a release

- Every PR: lint, unit tests across the runner's cores (pytest-xdist), 16
  property walks from a seed that moves with the run number, integration
  tests, and the end-to-end smoke tests (the slice, a conversation filed
  as work, a question; `smoke` marker).
- A PR labelled e2e:full, a push to main, or a manual run: all 48 walks and
  the whole end-to-end suite. Adding e2e:full to a PR runs it; any other
  label runs nothing.
- A published release runs the full set before its image is built.
- The walk's seeds come from NISSE_WALK_SEEDS / NISSE_WALK_FIRST (48 from 0
  by default).

Measured here: the unit tier on 2 workers took 3m49s against 6-8 minutes
serially.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-10-07 22:29:41 +00:00
1 parent 07a2eb9d10
commit e35aea5045
7 files changed
+124 -15

No files matched your search

+8 -4
View File
@@ -73,10 +73,14 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
`CLAUDE.md`. A milestone marked **needs discussion** starts with a design
discussion first.
- **Every change comes with tests,** through `scripts/check.sh`.
- **The end-to-end test runs on every PR** (M1.6): the real agent, every role on a
local model (so it costs nothing per call), a repository of the run's own and a
real build host, driven by a scripted operator. CI runs it as its own job on pull
requests, after the checks pass. There is one runner, and a run a newer push
- **Testing has three levels** (M1.6): every PR runs the checks (unit tests across
the runner's cores, 16 property walks from a moving seed, integration) and the
end-to-end **smoke** tests; a PR labelled `e2e:full`, a push to `main` and a
manual run add all 48 walks and the whole end-to-end suite; a published release
runs that full set before its image is built. The end-to-end tests drive the
real agent, every role on a local model (so they cost nothing per call), in a
repository of the run's own, with a scripted operator; CI runs them after the
checks pass. There is one runner, and a run a newer push
supersedes has to be cancelled by hand. Locally only the tests being written
(`-k`) or diagnosed (`--lf`); `scripts/e2e.sh` reports what a run spent. Manual
live runs are for the new loop's first runs and phase ends. Findings are filed as