Test levels: smoke e2e on PRs, the full set on e2e:full and before a release
- Every PR: lint, unit tests across the runner's cores (pytest-xdist), 16 property walks from a seed that moves with the run number, integration tests, and the end-to-end smoke tests (the slice, a conversation filed as work, a question; `smoke` marker). - A PR labelled e2e:full, a push to main, or a manual run: all 48 walks and the whole end-to-end suite. Adding e2e:full to a PR runs it; any other label runs nothing. - A published release runs the full set before its image is built. - The walk's seeds come from NISSE_WALK_SEEDS / NISSE_WALK_FIRST (48 from 0 by default). Measured here: the unit tier on 2 workers took 3m49s against 6-8 minutes serially. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
This commit is contained in:
1 parent
07a2eb9d10
commit
e35aea5045
7 files changed
+124
-15
No files matched your search
@@ -8,6 +8,7 @@ checks every property after every step. A failure prints
|
||||
the seed and the steps, so it replays exactly.
|
||||
"""
|
||||
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import subprocess
|
||||
@@ -46,8 +47,14 @@ from nisse_dev.tools import Workspace
|
||||
from nisse_dev.transport import InboundMessage, InMemoryTransport
|
||||
from nisse_dev.worker import InlineWorker
|
||||
|
||||
SEEDS = range(32)
|
||||
TWO_PROJECT_SEEDS = range(100, 116)
|
||||
# How many walks, and from which seed: all 48 by default; CI runs fewer on a
|
||||
# PR, from a seed that moves with the run number, so PRs still cover new
|
||||
# ground (NISSE_WALK_SEEDS, NISSE_WALK_FIRST). Two thirds walk one project,
|
||||
# a third two.
|
||||
_WALKS = int(os.environ.get("NISSE_WALK_SEEDS") or 48)
|
||||
_FIRST = int(os.environ.get("NISSE_WALK_FIRST") or 0)
|
||||
SEEDS = range(_FIRST, _FIRST + _WALKS * 2 // 3)
|
||||
TWO_PROJECT_SEEDS = range(100 + _FIRST, 100 + _FIRST + _WALKS - _WALKS * 2 // 3)
|
||||
STEPS = 30
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user