End-to-end tests on a local model, and in CI on every PR #110

2026-10-07T15:14:43Z

dev-agent commented

2026-10-07 15:14:43 +00:00

Lands before #108, as you asked.

What changes

Local model for e2e. Every e2e role (driver, reader, executor, helpers) runs on qwen3.8-27b on the RTX 4090. That's e2e_roles() in tests/integration/e2e_support.py; NISSE_E2E_MODEL overrides it, e.g. claude-haiku-4-5 for a quick paid run. The integration tier stays on Haiku.
Spend report. scripts/e2e.sh reports what a run spent on the test key, plus the day's total against its budget. On the local model, anything above $0 means a call fell back to a paid model.
E2E in CI. It's a separate job on pull requests only, not pushes to main:
- it uses a throwaway sshd (--local), as the integration tests do;
- a newer push cancels the run it supersedes;
- the 90-minute timeout leaves room over the ~26-minute local run;
- the checks job still reports in minutes.
Running it locally: only the tests being written (-k) or diagnosed (--lf).

Results

First full local run (on #108's branch, so all 8 tests): 7 of 8 passed in 25m 43s, and spend was $0.0000.
- The failure was real: qwen put the new project's code in the same PR as ARCHITECTURE.md, with no build plan. The fix, design items' first milestone limited to documents and enforced, goes into #108.
On this branch: unit (703) and integration (21) tiers pass. This PR's own CI run is the first test of the e2e job and of --local mode.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

Lands before #108, as you asked. ## What changes - **Local model for e2e.** Every e2e role (driver, reader, executor, helpers) runs on **qwen3.8-27b** on the RTX 4090. That's `e2e_roles()` in `tests/integration/e2e_support.py`; `NISSE_E2E_MODEL` overrides it, e.g. `claude-haiku-4-5` for a quick paid run. The integration tier stays on Haiku. - **Spend report.** `scripts/e2e.sh` reports what a run spent on the test key, plus the day's total against its budget. On the local model, anything above $0 means a call fell back to a paid model. - **E2E in CI.** It's a separate job on pull requests only, not pushes to `main`: - it uses a throwaway sshd (`--local`), as the integration tests do; - a newer push cancels the run it supersedes; - the 90-minute timeout leaves room over the ~26-minute local run; - the `checks` job still reports in minutes. - **Running it locally:** only the tests being written (`-k`) or diagnosed (`--lf`). ## Results - **First full local run** (on #108's branch, so all 8 tests): 7 of 8 passed in 25m 43s, and spend was **$0.0000**. - The failure was real: qwen put the new project's code in the same PR as `ARCHITECTURE.md`, with no build plan. The fix, design items' first milestone limited to documents and enforced, goes into #108. - **On this branch:** unit (703) and integration (21) tiers pass. This PR's own CI run is the first test of the e2e job and of `--local` mode. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dev-agent added 1 commit 2026-10-07 15:14:43 +00:00

End-to-end tests on a local model, and in CI on every PR

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 5m26s

Details

CI / e2e (pull_request) Successful in 17m27s

Details

6db80dfb4b

- Every e2e role runs on qwen3.8-27b on the RTX 4090 (e2e_roles();
  NISSE_E2E_MODEL overrides, e.g. claude-haiku-4-5). The first full run
  passed 7 of 8 in 26 minutes for $0.0000; the eighth was a real finding,
  fixed with the new-project PR.
- scripts/e2e.sh reports a run's spend on the test key and the day's total
  against its budget: above $0 on the local model means a paid fallback.
- CI runs the suite as its own job on pull requests (not pushes to main),
  on a throwaway sshd (--local), cancelling a run a newer push supersedes,
  so the checks job still reports in minutes.
- The plan says so: e2e on every PR; locally only -k / --lf.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dongar approved these changes

2026-10-07 15:17:55 +00:00

Dismissed

dongar approved these changes

2026-10-07 15:38:38 +00:00

dongar merged commit 9a86ccb2a4 into main

2026-10-07 15:38:46 +00:00

dongar deleted branch e2e-local-model

2026-10-07 15:38:46 +00:00

dev-agent referenced this pull request

2026-10-07 15:39:21 +00:00

New projects from conversation (M2.5, PR B) #108