Add the end-to-end test, run after each milestone (M1.6) #95

2026-10-02T22:59:13Z

dev-agent commented

2026-10-02 22:59:13 +00:00

Development Plan M1.6: the end-to-end test (added to the plan in #94). It replaces manual live runs after each milestone.

What it is

tests/integration/test_end_to_end.py builds the whole agent with build_agent, as it runs in production. Only the Matrix transport is swapped for the in-memory one, because a test shouldn't hold the operator's Matrix credentials. It runs against real models, the real Gitea sandbox and a real build host, and a scripted operator drives it:

It sends the task before choosing a project (it must be kept), then use.
It answers whatever the agent actually does: approvals, questions, start milestone 1. A fixed script would break when a live model asks something.

Then it asserts what bring-up run 3 checked by hand:

The PR holds only the work: at least one file, none of them build artifacts (#80), on the flat harness branch <task>-milestone-1.
The clone on the build host holds no credential (#89).
After next milestone, the planner's view carries [orchestrator] milestone 1 finished and … published, and nothing written by the executor (#78).

Cleanup always runs, even if startup fails: abandon closes the PR and plan issue, the PR's branch is deleted, and the run's clone is removed from the build host.

Running it

scripts/e2e.sh runs on the bring-up build VM (nisse-dev.garrett-hall.com). With --local it uses a throwaway local sshd instead.
A new marker, e2e, keeps it out of check.sh and CI, since it spends model tokens. It runs after each milestone, per the plan's §3.
With the test config's models (Haiku), a run takes about 30 seconds.
InMemoryTransport gained no-op start and close methods, so the assembled agent can run with it.

Is it real?

Passes on the build VM: sandbox PR #216 held only sandbox/vowels.py, its test and __init__, and was closed and its branch deleted by the cleanup.
Catches regressions: with ARTIFACT_PATTERNS emptied, it fails, because the PR carried six .pyc files. Its artifact check is its own list, independent of the code's, so emptying the code's list can't make it pass.
Leaves nothing behind: no open sandbox PRs, and no clones on the VM.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

Development Plan **M1.6: the end-to-end test** (added to the plan in #94). It replaces manual live runs after each milestone. ## What it is `tests/integration/test_end_to_end.py` builds the **whole agent with `build_agent`**, as it runs in production. Only the Matrix transport is swapped for the in-memory one, because a test shouldn't hold the operator's Matrix credentials. It runs against real models, the real Gitea sandbox and a real build host, and a scripted operator drives it: 1. It sends the task **before** choosing a project (it must be kept), then `use`. 2. It answers whatever the agent actually does: approvals, questions, `start milestone 1`. A fixed script would break when a live model asks something. Then it asserts what bring-up run 3 checked by hand: - **The PR holds only the work:** at least one file, none of them build artifacts (#80), on the flat harness branch `<task>-milestone-1`. - **The clone on the build host holds no credential** (#89). - **After `next milestone`,** the planner's view carries `[orchestrator] milestone 1 finished` and `… published`, and nothing written by the executor (#78). **Cleanup always runs, even if startup fails:** `abandon` closes the PR and plan issue, the PR's branch is deleted, and the run's clone is removed from the build host. ## Running it - **`scripts/e2e.sh`** runs on the bring-up build VM (`nisse-dev.garrett-hall.com`). With **`--local`** it uses a throwaway local `sshd` instead. - **A new marker, `e2e`,** keeps it out of `check.sh` and CI, since it spends model tokens. It runs after each milestone, per the plan's §3. - **With the test config's models** (Haiku), a run takes about 30 seconds. - **`InMemoryTransport` gained no-op `start` and `close` methods,** so the assembled agent can run with it. ## Is it real? - **Passes on the build VM:** sandbox PR #216 held only `sandbox/vowels.py`, its test and `__init__`, and was closed and its branch deleted by the cleanup. - **Catches regressions:** with `ARTIFACT_PATTERNS` emptied, it **fails**, because the PR carried six `.pyc` files. Its artifact check is its own list, independent of the code's, so emptying the code's list can't make it pass. - **Leaves nothing behind:** no open sandbox PRs, and no clones on the VM. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dev-agent added 1 commit 2026-10-02 22:59:14 +00:00

Add the end-to-end test, run after each milestone (M1.6)

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m48s

Details

b9f52be42c

tests/integration/test_end_to_end.py drives the whole assembled agent
(real models, the Gitea sandbox, a real build host) through a milestone
with a scripted operator, and checks what bring-up run 3 checked by hand:
a PR holding only the work, pushed with the push credential; no
credential in the clone; the planner's view carrying facts, not the
executor's turns. It cleans up after itself on the sandbox and the build
host.

scripts/e2e.sh runs it on the bring-up build VM (or --local). It has its
own marker, e2e, kept out of check.sh and CI since it spends tokens.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dongar approved these changes

2026-10-02 23:18:59 +00:00

dongar merged commit 202a0ca117 into main

2026-10-02 23:19:04 +00:00

dongar deleted branch e2e-test

2026-10-02 23:19:04 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#95