Properties against the real loop, and the slice end to end (M2.1, PR D) #99

2026-10-03T15:11:42Z

dev-agent commented

2026-10-03 15:11:42 +00:00

M2.1, PR D of 4: the properties against the real code, and the end-to-end test of the slice. Stacked on #97, whose branch loop-core now holds PR B and PR C (#98 merged into it).

`loop.qnt`'s properties, checked against the real loop

nisse-dev/tests/test_loop_properties.py is a seeded random walk over the new loop: 32 seeds of 20 steps each.

The actions: go-aheads, stale go-aheads, answers, PR comments, merges, the comment-then-merge race, closed PRs, abandon, and a crash and restart.
The model: an adaptive scripted one.
The check: every property after every step: one item per issue, waits are explainable, go-aheads bind, milestone after merge, one PR per milestone, review fixes land, and closed items leave nothing behind.

It found two code bugs, both fixed with their own regression tests:

Abandon left the item's PR open (the model's fix 1, now in code).
After a review fix became a follow-up PR, the next fix went to the merged branch and was lost. The design gave a follow-up its own branch; the code didn't.

The end-to-end test of the slice

tests/integration/test_end_to_end_loop.py runs the whole slice live, with real models, the Gitea sandbox and the build VM.

The scripted operator: nisse-testing-operator in Gitea, an operator only in nisse-test.yaml, plus the item's thread through the in-memory transport.
What it does:
1. files and assigns an issue;
2. restatement → go → milestone → PR, with only the work and "Closes #N";
3. asks for a specific change on the PR, and checks the pushed fix contains it;
4. approves and merges;
5. checks the item and issue close.

Its first run found a bug every unit test had missed. start_milestone records a fact while the tool call is in flight, which put a user message between the tool_use and its tool_result. Both APIs reject that, so every later request in the run failed with a 400.

Fix: views now keep each result directly after its call (views._results_follow_calls), for every role.
Why the unit tests missed it: they used scripted models that never checked message order. Now the loop tests' scripted models check both formats' real ordering rules (tests/model_contract.py), so this class of bug fails in unit tests. Confirmed: 23 loop tests failed with the stricter check before the fix.

Making the e2e tests reliable

Independent of history: each run asks for a uniquely named function and works against a base branch of its own (e2e-base-<run>, deleted afterwards). Runs no longer depend on what earlier ones merged. An earlier passing run had merged count_vowels into the sandbox's main (#238), which broke the next runs.
A review request that can't already be met: a marker comment with a fresh ID.
The scripted operator gives up after five fruitless answers, instead of spinning, which it did 3,431 times in the first run. That run's leftover issue (#234) and VM clone are cleaned up.
Each run keeps its state store under /root/nisse-e2e/test-runs/<run>/, for nisse_dev.dump after a failure. pytest prunes its own temp directories.
Both e2e tests pass, twice in a row (66–84 s), on the build VM. The sandbox's main is unchanged and no branches are left behind.

Also: the driver's prompt says review fixes stay in the current milestone. Live, it had tried to start a second one.

624 unit tests pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

M2.1, **PR D of 4: the properties against the real code, and the end-to-end test of the slice.** **Stacked on #97**, whose branch `loop-core` now holds PR B and PR C (#98 merged into it). ## `loop.qnt`'s properties, checked against the real loop `nisse-dev/tests/test_loop_properties.py` is a seeded random walk over the new loop: 32 seeds of 20 steps each. - **The actions:** go-aheads, stale go-aheads, answers, PR comments, merges, the comment-then-merge race, closed PRs, abandon, and a crash and restart. - **The model:** an adaptive scripted one. - **The check:** every property after every step: one item per issue, waits are explainable, go-aheads bind, milestone after merge, one PR per milestone, review fixes land, and closed items leave nothing behind. It found **two code bugs**, both fixed with their own regression tests: 1. **Abandon left the item's PR open** (the model's fix 1, now in code). 2. **After a review fix became a follow-up PR, the next fix went to the merged branch** and was lost. The design gave a follow-up its own branch; the code didn't. ## The end-to-end test of the slice `tests/integration/test_end_to_end_loop.py` runs the whole slice live, with real models, the Gitea sandbox and the build VM. - **The scripted operator:** `nisse-testing-operator` in Gitea, an operator only in `nisse-test.yaml`, plus the item's thread through the in-memory transport. - **What it does:** 1. files and assigns an issue; 2. restatement → `go` → milestone → PR, with only the work and "Closes #N"; 3. asks for a specific change on the PR, and checks the pushed fix contains it; 4. approves and merges; 5. checks the item and issue close. **Its first run found a bug every unit test had missed.** `start_milestone` records a fact while the tool call is in flight, which put a user message between the `tool_use` and its `tool_result`. Both APIs reject that, so every later request in the run failed with a 400. - **Fix:** views now keep each result directly after its call (`views._results_follow_calls`), for every role. - **Why the unit tests missed it:** they used scripted models that never checked message order. Now the loop tests' scripted models check both formats' real ordering rules (`tests/model_contract.py`), so this class of bug fails in unit tests. Confirmed: 23 loop tests failed with the stricter check before the fix. ## Making the e2e tests reliable - **Independent of history:** each run asks for a uniquely named function and works against **a base branch of its own** (`e2e-base-<run>`, deleted afterwards). Runs no longer depend on what earlier ones merged. An earlier passing run had merged `count_vowels` into the sandbox's `main` (#238), which broke the next runs. - **A review request that can't already be met:** a marker comment with a fresh ID. - **The scripted operator gives up after five fruitless answers,** instead of spinning, which it did 3,431 times in the first run. That run's leftover issue (#234) and VM clone are cleaned up. - **Each run keeps its state store** under `/root/nisse-e2e/test-runs/<run>/`, for `nisse_dev.dump` after a failure. pytest prunes its own temp directories. - **Both e2e tests pass, twice in a row** (66–84 s), on the build VM. The sandbox's `main` is unchanged and no branches are left behind. **Also:** the driver's prompt says review fixes stay in the current milestone. Live, it had tried to start a second one. 624 unit tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dongar changed target branch from loop-core to main

2026-10-03 16:42:57 +00:00

dongar added 4 commits 2026-10-03 16:42:57 +00:00

Check loop.qnt's properties against the real loop (M2.1, PR D part 1)

CI / checks (push) Skipped

266b5dbbd7

test_loop_properties.py walks the new loop with a seeded mix of operator
and Gitea actions (go-aheads, stale go-aheads, answers, comments, merges,
the comment-then-merge race, closes, abandon, a crash and restart) and an
adaptive scripted model, checking every loop.qnt property after every
step. It found two bugs, both fixed with regression tests: abandoning an
item left its open PR open (the model's fix 1, now in code), and after a
review fix became a follow-up PR, the next fix went to the merged branch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

Add the end-to-end test of the new loop's slice (M2.1, PR D part 2)

CI / checks (push) Skipped

897181b0f4

From an issue the scripted operator files and assigns in Gitea, through
the restatement, go-ahead, milestone and PR, a review comment answered
with a pushed fix, to an approved and merged PR and a closed item and
issue. The operator acts in Gitea through its own test account
(NISSE_OPERATOR_TOKEN) and in the item's thread through the in-memory
transport. Runs from scripts/e2e.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

Make nisse-testing-operator a Gitea operator in the test config only

CI / checks (push) Skipped

30ab5c9b9c

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

Fix the run-breaking tool-result order; e2e test runs the slice (M2.1, PR D)

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 2m17s

Details

37965aefca

The first live run of the new loop's e2e test found that a fact recorded
while start_milestone was in flight landed between the tool call and its
result; both API formats reject that, so every later request in the run
failed. Views now keep each result right after its call. Scripted test
models check the APIs' ordering rules, so this can't pass unit tests
again.

The e2e tests no longer depend on the sandbox's history: each run asks for
a uniquely named function and works against a base branch of its own,
deleted afterwards. The review step asks for a change that can't already
be there. The scripted operator gives up after five fruitless answers, and
each run keeps its state store for inspection. The driver's prompt says
review fixes stay in the current milestone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dongar approved these changes

2026-10-03 16:43:06 +00:00

dongar merged commit bf416cad09 into main

2026-10-03 16:43:10 +00:00

dongar deleted branch loop-properties

2026-10-03 16:43:10 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#99

Properties against the real loop, and the slice end to end (M2.1, PR D) #99

loop.qnt's properties, checked against the real loop

The end-to-end test of the slice

Making the e2e tests reliable

`loop.qnt`'s properties, checked against the real loop