Test levels: smoke e2e on PRs, full set on e2e:full and before a release #117

2026-10-07T22:30:02Z

dev-agent commented

2026-10-07 22:30:02 +00:00

Cuts per-PR test time, as agreed. Most PRs run a short acceptance set; the full set runs when a PR is labelled for it, and always before a release.

Levels

Every PR:
- lint;
- unit tests across the runner's cores (pytest -n auto, new dev dependency pytest-xdist), with 16 property walks from a seed that moves with the run number;
- integration tests;
- the smoke end-to-end tests (@pytest.mark.smoke): the slice, a conversation filed as work, and a question.
A PR labelled e2e:full, a push to main, or a manual run: all 48 walks and the whole e2e suite. Adding e2e:full triggers that run (pull_request: labeled); adding any other label runs nothing. I've created the e2e:full label. Worth using on PRs that touch the loop's core: scheduling, milestones, parking, projects.
A published release: release.yml now runs the full set (tests job) before build-and-push, so no image exists until everything has passed.
Walk seeds come from NISSE_WALK_SEEDS and NISSE_WALK_FIRST; locally the default is still all 48 from 0.

Slow apt

The other agent's investigation found archive.ubuntu.com crawling, which took the sshd install in every job from 8 seconds to 3.5–9.5 minutes. The new scripts/ci-sshd.sh skips the install when the image already has sshd, and otherwise points apt at mirrors.edge.kernel.org first. I checked the rewrite on both 22.04 and 24.04 source layouts.

Measured

Locally: the unit tier on 2 workers took 3m 49s, against 6–8 minutes serially on this 2-core box. The runner has 4 cores.
In CI: this PR's own run is the first real test of the label logic, xdist on the runner and the mirror. It should run the smoke e2e only. If you add e2e:full afterwards, it'll run the full set too.

Not in this PR

Thinking mode for qwen in e2e (the long-output finding) waits for your call on a nothink LiteLLM alias.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

Cuts per-PR test time, as agreed. Most PRs run a short acceptance set; the full set runs when a PR is labelled for it, and always before a release. ## Levels - **Every PR:** - lint; - unit tests across the runner's cores (`pytest -n auto`, new dev dependency `pytest-xdist`), with **16 property walks** from a seed that moves with the run number; - integration tests; - the **smoke** end-to-end tests (`@pytest.mark.smoke`): the slice, a conversation filed as work, and a question. - **A PR labelled `e2e:full`, a push to `main`, or a manual run:** all 48 walks and the whole e2e suite. Adding `e2e:full` triggers that run (`pull_request: labeled`); adding any other label runs nothing. I've created the `e2e:full` label. Worth using on PRs that touch the loop's core: scheduling, milestones, parking, projects. - **A published release:** `release.yml` now runs the full set (`tests` job) before `build-and-push`, so no image exists until everything has passed. - **Walk seeds** come from `NISSE_WALK_SEEDS` and `NISSE_WALK_FIRST`; locally the default is still all 48 from 0. ## Slow apt The other agent's investigation found `archive.ubuntu.com` crawling, which took the sshd install in every job from 8 seconds to 3.5–9.5 minutes. The new `scripts/ci-sshd.sh` skips the install when the image already has sshd, and otherwise points apt at `mirrors.edge.kernel.org` first. I checked the rewrite on both 22.04 and 24.04 source layouts. ## Measured - **Locally:** the unit tier on 2 workers took 3m 49s, against 6–8 minutes serially on this 2-core box. The runner has 4 cores. - **In CI:** this PR's own run is the first real test of the label logic, xdist on the runner and the mirror. It should run the smoke e2e only. If you add `e2e:full` afterwards, it'll run the full set too. ## Not in this PR Thinking mode for qwen in e2e (the long-output finding) waits for your call on a `nothink` LiteLLM alias. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dev-agent added 2 commits 2026-10-07 22:30:02 +00:00

Test levels: smoke e2e on PRs, the full set on e2e:full and before a release e35aea5045

- Every PR: lint, unit tests across the runner's cores (pytest-xdist), 16
  property walks from a seed that moves with the run number, integration
  tests, and the end-to-end smoke tests (the slice, a conversation filed
  as work, a question; `smoke` marker).
- A PR labelled e2e:full, a push to main, or a manual run: all 48 walks and
  the whole end-to-end suite. Adding e2e:full to a PR runs it; any other
  label runs nothing.
- A published release runs the full set before its image is built.
- The walk's seeds come from NISSE_WALK_SEEDS / NISSE_WALK_FIRST (48 from 0
  by default).

Measured here: the unit tier on 2 workers took 3m49s against 6-8 minutes
serially.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

CI: install sshd from a fast mirror, or not at all

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 2m10s

Details

CI / e2e (pull_request) Successful in 9m32s

Details

ff5a7a644f

archive.ubuntu.com has been crawling (a few KB/s), taking the sshd install
in every job from 8 seconds to 3.5-9.5 minutes (another agent measured it;
the runner and inference-4090 were healthy). scripts/ci-sshd.sh skips the
install when the image has sshd, and otherwise points apt at
mirrors.edge.kernel.org first (both source layouts, 22.04 and 24.04).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dongar approved these changes

2026-10-07 22:49:02 +00:00

Dismissed

dev-agent added 1 commit 2026-10-07 22:49:14 +00:00

E2E on qwen3.8-27b without thinking

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 2m14s

Details

CI / e2e (pull_request) Successful in 3m6s

Details

28092194fe

The e2e roles move to qwen3.8-27b-nothink, a LiteLLM alias for the same
model with chat_template_kwargs enable_thinking false. With thinking, a
growing share of answers ran to 2-8k tokens (two hit the 8192 cap), which
took one test from 3.5 to 20 minutes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dev-agent dismissed dongar's review 2026-10-07 22:49:14 +00:00

Reason:

New commits pushed, approval review dismissed automatically according to repository settings

dongar approved these changes

2026-10-07 22:49:58 +00:00

dev-agent merged commit 13efa54f16 into main

2026-10-07 22:55:15 +00:00

dev-agent deleted branch test-tiers

2026-10-07 22:55:16 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#117