Compare commits

...
10 Commits
Author SHA1 Message Date
dongar 09232dcbdd Merge pull request 'Fixes from the rest of the live session' (#122) from live-findings into main
CI / checks (push) Successful in 2m45s
CI / e2e (push) Skipped
Reviewed-on: #122
Reviewed-by: Don Garrett <dgarrett@acm.org>
2026-10-08 02:36:51 +00:00
dongarandClaude Opus 5.5 0064ec7696 Fixes from the rest of the live session
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 2m13s
CI / e2e (pull_request) Successful in 3m22s
- A near miss for abandon ("abort", "cancel", "drop it", ...) in an item's
  thread is asked about, and changes nothing.
- The driver is told only the operator stops or closes an item, so it
  doesn't claim to have done either.
- An issue assigned again after its item closed starts a new item that
  notes the one before (Gitea's timeline gives the assignment time);
  item_for_issue returns the latest item.
- Abandoning an issue Gitea won't close because of open dependencies says
  so, naming them.
- An agreed item isn't offered restate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-08 02:16:51 +00:00
dev-agent 613a2a7f72 Merge pull request 'Fixes from the first live session of the new loop' (#119) from live-fixes into main
CI / checks (push) Successful in 2m30s
CI / e2e (push) Skipped
2026-10-08 01:01:04 +00:00
dongarandClaude Opus 5.5 659915d309 Fixes from the first live session of the new loop
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 2m33s
CI / e2e (pull_request) Successful in 10m38s
Found live on 2026-10-07 (bring-up on nisse-testing-operator/nisse-bringup,
Opus as driver):

- The driver's prose nudge went into the system prompt, so the retry ended
  on the model's own reply. Opus 5.5 refuses that ("does not support
  assistant message prefill. The conversation must end with a user
  message"). The nudge is now a harness fact in the conversation. The
  delegation harness's prose path did the same and is fixed likewise.
- A helper's first request ended with its own delegation_started event,
  rendered as an assistant message: every delegation would have failed on
  Opus. The helper now sees it as the harness handing over its task.
- model_contract checks that every request ends on the user's side, so
  scripted tests catch this from now on (each fix is mutation-checked).
- `status` (and `details`) in an item's thread were taken as replies: they
  woke the driver and cancelled a bug's countdown. They now answer at once
  and change nothing.
- The driver had no way to answer a remark without changing the item, so it
  answered in prose. Every run woken by a message (or a review comment) now
  offers `reply`, which answers where asked and returns to the very wait
  that was interrupted.

Also: scripts/bringup-repo.sh makes the bring-up's test repository
(make_repo/delete_repo, split out of fresh_repo), and made repositories get
the agent's labels (bug, discuss, design).

Tests: test_loop_live_fixes.py (one per finding); the walk sends `status`
into threads (status_changes_nothing) and its scripted driver sometimes
answers remarks with reply.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-08 00:02:50 +00:00
dev-agent 572bd9f8eb Merge pull request 'M1.5 accepted: tests over a published image' (#118) from image-accept into main
CI / checks (push) Successful in 2m40s
CI / e2e (push) Skipped
2026-10-07 23:14:45 +00:00
dongarandClaude Opus 5.5 d7e4fd0938 M1.5 accepted: the tests run over a published image
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 2m10s
CI / e2e (pull_request) Successful in 2m54s
- The Dockerfile's test stage stacks on TEST_BASE: this build's agent by
  default, or a published image. It adds only the test tools (uv export
  of the dev group into the image's environment), never reinstalling the
  agent, so the suite tests the image as published.
- scripts/e2e.sh --image passes NISSE_E2E_MODEL into the container.
- The plan marks M1.5 done: v0.1.0's published image passed the whole e2e
  suite (9 of 9, 11m37s), with the release's own tests over it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 23:09:08 +00:00
dev-agent 13efa54f16 Merge pull request 'Test levels: smoke e2e on PRs, full set on e2e:full and before a release' (#117) from test-tiers into main
CI / checks (push) Successful in 2m42s
CI / e2e (push) Skipped
2026-10-07 22:55:15 +00:00
dongarandClaude Opus 5.5 28092194fe E2E on qwen3.8-27b without thinking
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 2m14s
CI / e2e (pull_request) Successful in 3m6s
The e2e roles move to qwen3.8-27b-nothink, a LiteLLM alias for the same
model with chat_template_kwargs enable_thinking false. With thinking, a
growing share of answers ran to 2-8k tokens (two hit the 8192 cap), which
took one test from 3.5 to 20 minutes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 22:49:13 +00:00
dongarandClaude Opus 5.5 ff5a7a644f CI: install sshd from a fast mirror, or not at all
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 2m10s
CI / e2e (pull_request) Successful in 9m32s
archive.ubuntu.com has been crawling (a few KB/s), taking the sshd install
in every job from 8 seconds to 3.5-9.5 minutes (another agent measured it;
the runner and inference-4090 were healthy). scripts/ci-sshd.sh skips the
install when the image has sshd, and otherwise points apt at
mirrors.edge.kernel.org first (both source layouts, 22.04 and 24.04).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 22:29:41 +00:00
dongarandClaude Opus 5.5 e35aea5045 Test levels: smoke e2e on PRs, the full set on e2e:full and before a release
- Every PR: lint, unit tests across the runner's cores (pytest-xdist), 16
  property walks from a seed that moves with the run number, integration
  tests, and the end-to-end smoke tests (the slice, a conversation filed
  as work, a question; `smoke` marker).
- A PR labelled e2e:full, a push to main, or a manual run: all 48 walks and
  the whole end-to-end suite. Adding e2e:full to a PR runs it; any other
  label runs nothing.
- A published release runs the full set before its image is built.
- The walk's seeds come from NISSE_WALK_SEEDS / NISSE_WALK_FIRST (48 from 0
  by default).

Measured here: the unit tier on 2 workers took 3m49s against 6-8 minutes
serially.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 22:29:41 +00:00
24 changed files with 755 additions and 93 deletions

No files matched your search

+42 -20
View File
@@ -1,4 +1,10 @@
name: CI
# Three levels of testing:
# - every PR: lint, unit tests (16 property walks), integration tests, and
# the smoke end-to-end tests (`smoke` marker);
# - a PR labelled e2e:full, a push to main and a manual run: all 48 walks
# and the whole end-to-end suite;
# - a published release: the full set again before its image (release.yml).
on:
# Only main on push. A PR branch is already covered by the pull_request
# event, and running both fired two identical suites concurrently — which
@@ -7,11 +13,18 @@ on:
push:
branches: [main]
pull_request:
# labeled: adding e2e:full runs the full set; any other label, nothing.
types: [opened, synchronize, reopened, labeled]
# Manual re-run, e.g. after rotating a credential or raising a budget cap.
workflow_dispatch:
env:
# Full on e2e:full, and outside PRs.
FULL: ${{ github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'e2e:full') }}
jobs:
checks:
if: github.event.action != 'labeled' || github.event.label.name == 'e2e:full'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
@@ -26,12 +39,10 @@ jobs:
- name: Install sshd
# The tests reach "the build VM" over real SSH, through a throwaway
# sshd the test fixture starts (nisse_dev/testing.py).
run: |
SUDO=$(command -v sudo || true)
$SUDO apt-get update -qq
$SUDO apt-get install -y -qq openssh-server openssh-client > /dev/null
$SUDO mkdir -p /run/sshd
# sshd the test fixture starts (nisse_dev/testing.py). Skipped when the image has it; otherwise from
# a kernel.org mirror: archive.ubuntu.com has been slow enough to add
# minutes to every job.
run: scripts/ci-sshd.sh
- name: Lint
run: uv run ruff check .
@@ -40,7 +51,12 @@ jobs:
run: uv run ruff format --check .
- name: Unit tests
run: uv run pytest -m "not integration and not e2e"
# Across the runner's cores (pytest-xdist). On a PR, 16 property
# walks from a seed that moves with the run number; all 48 otherwise.
env:
NISSE_WALK_SEEDS: ${{ env.FULL == 'true' && '48' || '16' }}
NISSE_WALK_FIRST: ${{ env.FULL == 'true' && '0' || github.run_number }}
run: uv run pytest -n auto -m "not integration and not e2e"
- name: Integration tests
# Everything but the secrets is in tests/integration/nisse-test.yaml,
@@ -58,12 +74,15 @@ jobs:
e2e:
# The end-to-end suite (scripts/e2e.sh): the whole agent against real
# services, every role on a local model, so it spends nothing but keeps
# the RTX 4090 busy for ~25 minutes. On PRs only, and only after the
# checks pass: a PR that's red already doesn't spend the GPU. There is
# one runner and Gitea doesn't honour cancel-in-progress, so a run a
# newer push supersedes holds the runner until it's cancelled by hand.
# Locally, run only what you're writing (-k) or diagnosing (--lf).
if: github.event_name != 'push'
# the RTX 4090 busy: ~10 minutes for the smoke tests, ~25+ for all. On
# PRs and manual runs, and only after the checks pass: a PR that's red
# already doesn't spend the GPU. There is one runner and Gitea doesn't
# honour cancel-in-progress, so a run a newer push supersedes holds the
# runner until it's cancelled by hand. Locally, run only what you're
# writing (-k) or diagnosing (--lf).
if: >-
github.event_name != 'push'
&& (github.event.action != 'labeled' || github.event.label.name == 'e2e:full')
needs: checks
runs-on: ubuntu-latest
timeout-minutes: 90
@@ -79,12 +98,10 @@ jobs:
run: uv sync --all-packages
- name: Install sshd
# The build VM is a throwaway sshd here (scripts/e2e.sh --local).
run: |
SUDO=$(command -v sudo || true)
$SUDO apt-get update -qq
$SUDO apt-get install -y -qq openssh-server openssh-client > /dev/null
$SUDO mkdir -p /run/sshd
# The build VM is a throwaway sshd here (scripts/e2e.sh --local). Skipped when the image has it; otherwise from
# a kernel.org mirror: archive.ubuntu.com has been slow enough to add
# minutes to every job.
run: scripts/ci-sshd.sh
- name: End-to-end tests
env:
@@ -93,4 +110,9 @@ jobs:
NISSE_GITEA_TOKEN: ${{ secrets.NISSE_GITEA_TOKEN }}
NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }}
NISSE_E2E_RUNS: /tmp/nisse-e2e-runs
run: scripts/e2e.sh --local -v
run: |
if [ "$FULL" = true ]; then
scripts/e2e.sh --local -v
else
scripts/e2e.sh --local -v -m "e2e and smoke"
fi
+41 -1
View File
@@ -3,7 +3,8 @@ name: Release
# registry (Development Plan M1.5, X4): gitea.dongar.net/dev/nisse-dev
# tagged with the version (the release tag without its v) and latest. Cut a
# release in Gitea (Releases → New release, tag vX.Y.Z); a tag pushed for any
# other reason builds nothing.
# other reason builds nothing. The full test set runs first: no image without
# it passing.
#
# The registry needs a personal access token with write:package
# (REGISTRY_TOKEN, an organization secret of `dev`, belonging to the
@@ -23,7 +24,46 @@ env:
IMAGE: gitea.dongar.net/dev/nisse-dev
jobs:
tests:
# Everything, before an image exists: lint, unit tests with all 48
# property walks, integration tests and the whole end-to-end suite.
runs-on: ubuntu-latest
timeout-minutes: 150
env:
NISSE_LLM_TOKEN: ${{ secrets.NISSE_LLM_TOKEN }}
NISSE_MATRIX_TOKEN: ${{ secrets.NISSE_MATRIX_TOKEN }}
NISSE_GITEA_TOKEN: ${{ secrets.NISSE_GITEA_TOKEN }}
NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }}
NISSE_E2E_RUNS: /tmp/nisse-e2e-runs
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install uv
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Sync workspace
run: uv sync --all-packages
- name: Install sshd
run: scripts/ci-sshd.sh
- name: Lint
run: uv run ruff check . && uv run ruff format --check .
- name: Unit tests
run: uv run pytest -n auto -m "not integration and not e2e"
- name: Integration tests
run: uv run pytest -m "integration and not e2e" -rs
- name: End-to-end tests
run: scripts/e2e.sh --local -v
build-and-push:
needs: tests
runs-on: ubuntu-latest
permissions:
contents: read
+13 -8
View File
@@ -10,6 +10,12 @@
# docker build -t nisse-dev . the agent
# docker build --target test -t nisse-dev:test . plus the tests, for
# scripts/e2e.sh --image
# docker build --target test --build-arg TEST_BASE=<published image> \
# -t nisse-dev:test . the tests over a release as
# published (M1.5 acceptance)
# What the test stage stacks on: this build's agent, or a published image.
ARG TEST_BASE=runtime
FROM python:3.12-slim AS build
COPY --from=ghcr.io/astral-sh/uv:0.12.10 /uv /usr/local/bin/uv
@@ -40,9 +46,10 @@ VOLUME /var/lib/nisse-dev
USER nisse
ENTRYPOINT ["nisse-dev"]
# The image's own package plus the tests, so the end-to-end suite runs
# against what was built, not the source tree.
FROM runtime AS test
# The tests over an image's own package (this build's, or a published one,
# TEST_BASE): only the test tools are added, never the agent reinstalled, so
# the end-to-end suite runs against what was built, not the source beside it.
FROM ${TEST_BASE} AS test
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends git openssh-server \
@@ -57,11 +64,9 @@ COPY nisse-ops nisse-ops
COPY tests tests
COPY scripts scripts
COPY conftest.py ./
ENV UV_PROJECT_ENVIRONMENT=/opt/nisse
# Test tools too, all installed (not editable), and `uv run` never re-syncs:
# the tests import the package as built, not the source tree beside them.
RUN uv sync --frozen --all-packages --no-editable --inexact
ENV UV_NO_SYNC=1
ENV UV_PROJECT_ENVIRONMENT=/opt/nisse UV_NO_SYNC=1
RUN uv export --frozen --only-group dev --no-hashes --no-emit-project > /tmp/dev.txt \
&& uv pip install --quiet --python /opt/nisse/bin/python -r /tmp/dev.txt
ENTRYPOINT []
# The default target: the agent itself.
+3
View File
@@ -569,6 +569,9 @@ Work arrives as Gitea issues or conversation, and leaves as reviewed PRs.
is. The agent acknowledges it ("queued myproject#14, 1 ahead"). Your issue is the
item's record. Only issues filed or assigned by allowlisted operators count (L11).
- **Unassigned issues are backlog,** never work, though the agent will discuss them.
- **An issue assigned again after its item closed is a new item,** which notes the
one before it; an issue whose item is still open, or whose assignment predates
the close, stays with that item.
- **Open Gitea dependencies make an issue blocked** (L1).
- **Work that starts in conversation gets a record** once it is work on a project,
not at its first message: the transcript already keeps the conversation. The
+15 -6
View File
@@ -73,10 +73,14 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
`CLAUDE.md`. A milestone marked **needs discussion** starts with a design
discussion first.
- **Every change comes with tests,** through `scripts/check.sh`.
- **The end-to-end test runs on every PR** (M1.6): the real agent, every role on a
local model (so it costs nothing per call), a repository of the run's own and a
real build host, driven by a scripted operator. CI runs it as its own job on pull
requests, after the checks pass. There is one runner, and a run a newer push
- **Testing has three levels** (M1.6): every PR runs the checks (unit tests across
the runner's cores, 16 property walks from a moving seed, integration) and the
end-to-end **smoke** tests; a PR labelled `e2e:full`, a push to `main` and a
manual run add all 48 walks and the whole end-to-end suite; a published release
runs that full set before its image is built. The end-to-end tests drive the
real agent, every role on a local model (so they cost nothing per call), in a
repository of the run's own, with a scripted operator; CI runs them after the
checks pass. There is one runner, and a run a newer push
supersedes has to be cancelled by hand. Locally only the tests being written
(`-k`) or diagnosed (`--lf`); `scripts/e2e.sh` reports what a run spent. Manual
live runs are for the new loop's first runs and phase ends. Findings are filed as
@@ -271,8 +275,9 @@ in prose, asking mid-work (L5, #42, #77).
- Deferred: config reload without a restart (with M2.5's entry step if wanted);
the trust dial's configuration (M3.3).
**M1.5 The agent as a container image.** *Pre-approved.* **Built** (this PR);
accepted once a tag's image passes the e2e suite.
**M1.5 The agent as a container image.** *Pre-approved.* **Done** (#115; accepted
2026-10-07: `v0.1.0`'s published image passed the whole e2e suite, 9 of 9, run with
the release's own tests over it).
- A Dockerfile for `nisse-dev`, stateless with a volume for state; a Gitea Actions
workflow that builds it and pushes to Gitea's registry on each release tag (X4).
- `bringup.sh` runs the image instead of a venv.
@@ -283,6 +288,10 @@ accepted once a tag's image passes the e2e suite.
package, for `scripts/e2e.sh --image REF`. Images are built only on release:
`release.yml` pushes `gitea.dongar.net/dev/nisse-dev:<version>` and `:latest`
on a published release (`REGISTRY_TOKEN`, the `dev` organization's).
- Acceptance runs the tests over a published image: `docker build --target test
--build-arg TEST_BASE=gitea.dongar.net/dev/nisse-dev:<version>` from the
release's checkout, then `scripts/e2e.sh --image`; the test stage adds only the
test tools, so the agent under test is the one published.
- The live bring-up's switch to the image waits for the next live test (it needs
its own test repository too); `bringup.sh plan` shows what it will run.
+10
View File
@@ -33,6 +33,7 @@ from nisse_dev.canonical import (
AssistantText,
DelegationCompleted,
DelegationStarted,
Fact,
PlanSubmitted,
StoredEvent,
ToolCall,
@@ -414,6 +415,15 @@ class Delegation:
# and its tree state is unknown. Treat it as a failure rather
# than inventing a handback on its behalf.
if any(isinstance(e, AssistantText) for e in produced):
# Said something but did nothing: told so, as the
# harness, so the next request ends on the user's side
# (models refuse assistant prefill).
transcript.as_actor("orchestrator").append(
Fact(
text="Your last reply was prose. Use your tools to "
"do the work, then call hand_back."
)
)
continue
return DelegationResult(
"failed",
+25
View File
@@ -261,6 +261,31 @@ class GiteaClient:
who = (event.get("user") or {}).get("login")
return who
def assigned_at(self, repo: str, index: int, assignee: str) -> str | None:
"""When the issue was last assigned to ``assignee`` (ISO 8601), from
its timeline, oldest first and paged to the end; None if the
timeline doesn't say."""
when = None
for page in range(1, MAX_PR_PAGES + 1):
events = (
self._request(
"GET",
f"/repos/{repo}/issues/{index}/timeline",
params={"limit": PR_PAGE_SIZE, "page": page},
)
or []
)
for event in events:
if (
event.get("type") == "assignees"
and not event.get("removed_assignee")
and (event.get("assignee") or {}).get("login") == assignee
):
when = event.get("created_at")
if len(events) < PR_PAGE_SIZE:
break
return when
def repository(self, repo: str) -> dict[str, Any]:
"""The repository as this account sees it, permissions included."""
return dict(self._request("GET", f"/repos/{repo}"))
+133 -31
View File
@@ -22,6 +22,7 @@ import time
import uuid
from collections.abc import Callable, Iterable, Sequence
from dataclasses import dataclass, replace
from datetime import UTC, datetime
from nisse_dev.cancel import Cancel
from nisse_dev.canonical import (
@@ -87,6 +88,19 @@ _NEXT = re.compile(rf"(?:do|work on) {_ISSUE} (?:next|first|now)")
_NAMED = re.compile(r"(go|approve|abandon|stop) (?:([a-z0-9][a-z0-9-]*))?#(\d+)")
# At top level these need an item; without one they are ambiguous (L5).
_NEEDS_ITEM = frozenset({"go", "approve", "abandon"})
# Near-misses for abandon: asked about, never taken for it or for a reply.
_NEAR_ABANDON = frozenset(
{
"abort",
"abort it",
"cancel",
"cancel it",
"drop it",
"quit",
"kill it",
"scrap it",
}
)
BUG_GRACE_S = 600.0
GENERAL_IDLE_S = 86400.0
PROGRESS_EVERY_S = 60.0 # progress in prose at a readable rate (L5, #42)
@@ -113,9 +127,16 @@ If the request is unclear, ask_human instead of guessing: a restatement asks
no questions, so the operator can agree to it as it stands. At any point,
asking is better than guessing.
Only the operator stops or closes an item (their stop and abandon commands):
never say you have stopped or closed anything yourself.
Before each group of tool calls, say in a sentence what you're doing and why:
the operator sees it as your progress.
When the operator says something that needs an answer but no change (a
remark, a status question), answer with reply: the item goes back to what it
was waiting for. Every reply of yours is a tool call, never bare prose.
The project's documents are in its repository: read ARCHITECTURE.md and any
build plans under docs/plans/ before you restate.
@@ -500,7 +521,8 @@ class LoopAgent:
self._oplog.emit("loop.intake_unavailable", cause=str(exc))
return
for issue in issues:
if item_for_issue(self._store, issue.number, project.name) is not None:
before = item_for_issue(self._store, issue.number, project.name)
if before is not None and not self._assigned_again(project, issue, before):
continue
if _MARKER.search(issue.body or ""):
continue # D1: a record of work started in conversation
@@ -517,7 +539,7 @@ class LoopAgent:
except GiteaError as exc:
self._oplog.emit("loop.intake_unavailable", cause=str(exc))
continue # taken on the next round
self._open(project, issue, tuple(blocked_by))
self._open(project, issue, tuple(blocked_by), follows=before)
# -- PR review and merge (D4, D5) -------------------------------------------
@@ -587,6 +609,8 @@ class LoopAgent:
loop, position = replace(loop, pr=0, pr_url="", branch=""), READY
elif new and item.position == WAITING:
position = READY
# A comment can be answered with a reply, back to this wait.
loop = replace(loop, back_to=loop.wait, back_note=loop.note)
if position != item.position or loop != loop_of(item):
if position == READY:
loop = replace(loop, wait="", note="")
@@ -683,8 +707,24 @@ class LoopAgent:
return False
return who in self._gitea_operators
def _assigned_again(self, project: Project, issue: Issue, before: WorkItem) -> bool:
"""D1: an issue is taken once, and again only when it's assigned to
the agent anew after its last item closed (abandoned, say, then
given back): never on the same assignment twice."""
if before.status == "open" or not before.closed_at:
return False
try:
when = self._gitea.assigned_at(project.repo, issue.number, self._account)
except GiteaError:
return False
return bool(when) and _instant(when) > _instant(before.closed_at)
def _open(
self, project: Project, issue: Issue, blocked_by: tuple[str, ...]
self,
project: Project,
issue: Issue,
blocked_by: tuple[str, ...],
follows: WorkItem | None = None,
) -> None:
item_id = f"{project.name}-{issue.number}-{uuid.uuid4().hex[:6]}"
cause = f"assigned {project.repo}#{issue.number}"
@@ -702,6 +742,7 @@ class LoopAgent:
issue=issue.number,
blocked_by=blocked_by,
labels=labels,
follows=follows.id if follows else "",
)
self._threads[(project.name, issue.number)] = root
self._store.open_work_item(
@@ -723,6 +764,12 @@ class LoopAgent:
if project.labels
else ""
)
+ (
f"; assigned again after its earlier item {follows.id} closed "
f"({follows.outcome})"
if follows
else ""
)
)
)
text = f"{issue.title}\n\n{issue.body}".strip()
@@ -824,6 +871,31 @@ class LoopAgent:
elif not self._store.active_item():
self._transport.post("Nothing is running.", thread=msg.thread)
return
if command in _NEAR_ABANDON:
# L5: a near-miss for a command is asked about, never read as a
# reply (the driver can't stop or close anything itself).
self._store.transcript(None, "operator").append(
UserMessage(text=text, source=f"transport:{msg.sender}")
)
self._transport.post(
f"Did you mean `abandon`? That closes {self._label(item)} and its "
"issue; say abandon to confirm. (`stop` ends a run in progress.)",
thread=loop_of(item).thread,
)
return
if command in ("status", "details"):
# About the item, not a reply to it: nothing changes, a
# countdown runs on (L5). Recorded at the agent's level.
self._store.transcript(None, "operator").append(
UserMessage(text=text, source=f"transport:{msg.sender}")
)
said = (
self._details(item)
if command == "details"
else self._status_line(item, [i.id for i in self._queue()])
)
self._transport.post(said, thread=loop_of(item).thread)
return
self._store.transcript(item.id, "operator").append(
UserMessage(text=text, source=f"transport:{msg.sender}")
)
@@ -844,8 +916,6 @@ class LoopAgent:
self._close(item, loop, "abandoned", f"abandoned by {msg.sender}")
elif command == "done" and loop.new_project:
self._provisioned(item, loop)
elif command == "details":
self._transport.post(self._details(item), thread=loop.thread)
elif (
item.position == WAITING
and item.scope != GENERAL
@@ -903,12 +973,13 @@ class LoopAgent:
)
self._go(item, loop, f"{msg.sender} (in plain language)")
elif item.position == WAITING:
# An answer, or changes to a restatement: the driver reads it.
# An answer, changes to a restatement, or a remark: the driver
# reads it, and may reply and go back to this wait (L5).
save(
self._store,
item.id,
READY,
replace(loop, wait="", note=""),
replace(loop, wait="", note="", back_to=loop.wait, back_note=loop.note),
cause="the operator answered",
)
self._schedule()
@@ -1272,28 +1343,28 @@ class LoopAgent:
f"{self._repos()}."
)
queue = [i.id for i in self._queue()]
lines = []
for item in items:
loop = loop_of(item)
what = item.position
if item.position == WAITING and loop.wait == CHAT:
what = "talking; your turn"
elif item.position == WAITING:
what = f"waiting for {loop.wait} ({loop.note or f'rev-{loop.rev}'})"
if loop.grace_until:
left = max(0, int(loop.grace_until - self._clock()) // 60)
what += f", going ahead in {left} min unless you reply"
elif item.position == BLOCKED:
what = f"blocked by {', '.join(loop.blocked_by)}"
elif item.position == READY:
what = f"ready, {queue.index(item.id) + 1} of {len(queue)} in line"
name = (
f'conversation "{item.title}"'
if item.scope == GENERAL
else f"{self._label(item)} {item.title}"
)
lines.append(f"{name}: {what}")
return "\n".join(lines)
return "\n".join(self._status_line(item, queue) for item in items)
def _status_line(self, item: WorkItem, queue: list[str]) -> str:
loop = loop_of(item)
what = item.position
if item.position == WAITING and loop.wait == CHAT:
what = "talking; your turn"
elif item.position == WAITING:
what = f"waiting for {loop.wait} ({loop.note or f'rev-{loop.rev}'})"
if loop.grace_until:
left = max(0, int(loop.grace_until - self._clock()) // 60)
what += f", going ahead in {left} min unless you reply"
elif item.position == BLOCKED:
what = f"blocked by {', '.join(loop.blocked_by)}"
elif item.position == READY:
what = f"ready, {queue.index(item.id) + 1} of {len(queue)} in line"
name = (
f'conversation "{item.title}"'
if item.scope == GENERAL
else f"{self._label(item)} {item.title}"
)
return f"{name}: {what}"
# -- running (L1, L2) ------------------------------------------------------
@@ -1648,7 +1719,9 @@ class LoopAgent:
loop = replace(loop, prepared=True)
save(self._store, item.id, RUNNING, loop, cause="driver run")
# D1: a `discuss` item talks before it restates.
hold = "discuss" in loop.labels and not loop.talked and not loop.started
hold = (
"discuss" in loop.labels and not loop.talked
) or loop.started # agreed: a restatement is behind it (D1)
if loop.question:
hold = True # nothing but answering (below)
tools = [
@@ -1663,12 +1736,15 @@ class LoopAgent:
CLOSE_ITEM,
*([WAIT_FOR_REVIEW] if loop.pr else []),
*([DELEGATE, ASK_EXPERT] if self._helpers else []),
*([REPLY] if loop.back_to else []),
*EXECUTOR_TOOLS,
]
),
]
if loop.question:
tools = [REPLY, *_LOOK]
elif loop.back_to and not loop.started:
tools = [*tools, REPLY] # before the go-ahead, a remark answered
roles = self._roles_of(item)
workspace = project.workspace
system = SYSTEM.format(
@@ -1771,6 +1847,13 @@ class LoopAgent:
replace(loop, rev=rev, wait=GO_AHEAD, note="", grace_until=grace),
None,
)
elif outcome.kind == "wait" and outcome.tool == "reply" and loop.back_to:
# A project item's reply: answered where asked (the thread or the
# PR), then back to the very wait the message interrupted (L5).
self._answer_where_asked(item, loop, str(args.get("text", "")).strip())
self._wait(
item, replace(loop, wait=loop.back_to, note=loop.back_note), None
)
elif outcome.kind == "wait" and outcome.tool == "reply":
text = str(args.get("text", "")).strip()
self._transport.post(text, thread=loop.thread)
@@ -2068,6 +2151,8 @@ class LoopAgent:
def _wait(self, item: WorkItem, loop: Loop, say: str | None) -> None:
if say:
self._transport.post(say, thread=loop.thread)
# Waiting again: whatever woke the item is settled.
loop = replace(loop, question=False, back_to="", back_note="")
save(self._store, item.id, WAITING, loop, cause=f"waiting for {loop.wait}")
def _close(self, item: WorkItem, loop: Loop, outcome: str, note: str) -> None:
@@ -2094,8 +2179,19 @@ class LoopAgent:
self._gitea.post_comment(project.repo, loop.issue, f"{outcome}: {note}")
self._gitea.close_issue(project.repo, loop.issue)
except GiteaError as exc:
try:
blockers = self._gitea.open_dependencies(project.repo, loop.issue)
except GiteaError:
blockers = []
why = (
f"Gitea won't close an issue while it depends on open ones "
f"({', '.join(blockers)}): close it when they're done, or "
"remove the dependency"
if blockers
else str(exc)
)
self._transport.post(
f"closed here, but could not close #{loop.issue}: {exc}",
f"closed here, but could not close #{loop.issue}: {why}",
thread=loop.thread,
)
if outcome == "abandoned":
@@ -2103,3 +2199,9 @@ class LoopAgent:
def _fact(self, item: WorkItem, text: str) -> None:
self._store.transcript(item.id, "orchestrator").append(Fact(text=text))
def _instant(stamp: str) -> datetime:
"""An ISO 8601 time from Gitea or the state store, comparable either way."""
when = datetime.fromisoformat(stamp)
return when if when.tzinfo else when.replace(tzinfo=UTC)
+5 -2
View File
@@ -12,7 +12,7 @@ from dataclasses import dataclass
from typing import Any, Literal
from nisse_dev.cancel import Cancel, skip_calls
from nisse_dev.canonical import AssistantText, ToolCall, ToolResult
from nisse_dev.canonical import AssistantText, Fact, ToolCall, ToolResult
from nisse_dev.llm import LLMClient, LLMError
from nisse_dev.loop.tools import WAITING, names
from nisse_dev.state import Transcript
@@ -100,7 +100,10 @@ class Driver:
if nudged:
return RunOutcome("failed", note="kept replying in prose")
nudged = True
system = f"{system}\n\n{_PROSE_NUDGE}"
# In the conversation, as the harness: a request must end on
# the user's side (models refuse assistant prefill), and the
# nudge is then in the log as what it was.
transcript.as_actor("orchestrator").append(Fact(text=_PROSE_NUDGE))
continue
for index, call in enumerate(calls):
if self._cancel.requested:
+9 -6
View File
@@ -108,9 +108,12 @@ def item_for_thread(
def item_for_issue(
store: StateStore, issue: int, project: str | None = None
) -> WorkItem | None:
"""Any item, open or closed, recorded by this issue (in ``project``,
when given): intake never takes an issue twice (D1)."""
for item in store.work_items():
if loop_of(item).issue == issue and project in (None, item.scope):
return item
return None
"""The latest item, open or closed, recorded by this issue (in
``project``, when given): intake takes an issue once, and again only on a
fresh assignment after its item closed (D1)."""
found = [
item
for item in store.work_items()
if loop_of(item).issue == issue and project in (None, item.scope)
]
return found[-1] if found else None
+8
View File
@@ -22,6 +22,7 @@ from dataclasses import replace
from nisse_dev.canonical import (
AssistantText,
DelegationCompleted,
DelegationStarted,
Fact,
GateOpened,
PlanSubmitted,
@@ -105,6 +106,13 @@ def project(events: list[StoredEvent], viewer: str) -> list[StoredEvent]:
if viewer == "helper":
out.append(_said(stored, "approved plan", event.plan))
continue
if isinstance(event, DelegationStarted):
# The harness handing a task over: to the helper given it, a
# message from the harness, never its own turn (a request must
# end on the user's side; models refuse assistant prefill).
if actor == viewer:
out.append(_said(stored, "orchestrator", f"your task: {event.brief}"))
continue
if actor == viewer:
out.append(stored)
elif actor == "operator" and isinstance(event, UserMessage):
+7
View File
@@ -20,6 +20,13 @@ def assert_renderable(events: list[Any]) -> None:
def _anthropic(messages: list[dict[str, Any]]) -> None:
# A request ends on the user's side: current models refuse assistant
# prefill ("The conversation must end with a user message"), found live
# on 2026-10-07 when the driver's prose nudge left its own reply last.
assert not messages or messages[-1]["role"] == "user", (
f"anthropic messages.{len(messages) - 1}: the conversation ends with an "
"assistant message (prefill), which the API refuses"
)
for index, message in enumerate(messages):
if message["role"] != "assistant":
continue
+27
View File
@@ -229,3 +229,30 @@ def test_closing_comments_first_then_closes_issue_and_pr() -> None:
("POST", "/api/v1/repos/o/r/issues/20/comments", {"body": "abandoned by don"}),
("PATCH", "/api/v1/repos/o/r/pulls/20", {"state": "closed"}),
]
def test_assigned_at_is_the_last_assignment_across_pages() -> None:
def assigned(login: str, at: str, removed: bool = False) -> dict:
return {
"type": "assignees",
"assignee": {"login": login},
"removed_assignee": removed,
"created_at": at,
}
filler = {"type": "comment", "created_at": "t0"}
pages = {
1: [assigned("dev-agent", "t1")] + [filler] * (PR_PAGE_SIZE - 1),
2: [
assigned("dev-agent", "t2", removed=True),
assigned("someone", "t3"),
assigned("dev-agent", "t4"),
],
}
def handler(request: httpx.Request) -> httpx.Response:
assert request.url.path == "/api/v1/repos/o/r/issues/6/timeline"
return httpx.Response(200, json=pages[int(request.url.params["page"])])
assert client(handler).assigned_at("o/r", 6, "dev-agent") == "t4"
assert client(handler).assigned_at("o/r", 6, "nobody") is None
+4
View File
@@ -49,6 +49,7 @@ class FakeGitea:
self.comments: list[tuple[int, str]] = []
self.closed: list[int] = []
self.deps: dict[int, list[int]] = {}
self.reassigned: dict[int, str] = {} # issue -> when assigned again
def create_issue(self, repo: str, title: str, body: str) -> int:
n = max([i.number for i in self.issues] + [0]) + 1
@@ -67,6 +68,9 @@ class FakeGitea:
def assigned_by(self, repo: str, index: int, assignee: str) -> str:
return self.assigner
def assigned_at(self, repo: str, index: int, assignee: str) -> str | None:
return self.reassigned.get(index)
def post_comment(self, repo: str, index: int, body: str) -> None:
self.comments.append((index, body))
+238
View File
@@ -0,0 +1,238 @@
"""What the first live session of the new loop found (2026-10-07), each as a
test: a request ending on the assistant's side (refused by models without
prefill: the driver's prose nudge, a helper's first request), `status` in an
item's thread taken as a reply, no way for the driver to answer a remark;
later, near-miss commands, the driver claiming a stop, a reassigned issue
ignored, an unexplained failed abandon and restating an agreed item.
"""
from pathlib import Path
from typing import Any
from model_contract import assert_renderable
from test_loop import DON, RESTATED, FakeGitea, Harness, ScriptedModel, call, issue
from test_loop_help import ROLES, hand_back
from test_loop_milestones import RESTATE, Forge, loop_agent, say
from test_loop_milestones import item as the_item
from test_loop_milestones import world as world # noqa: PLC0414 -- the fixture
from nisse_dev.canonical import AssistantText, Fact
from nisse_dev.delegation import Caps
from nisse_dev.gitea import Comment
from nisse_dev.loop.help import Helpers
from nisse_dev.loop.items import GO_AHEAD, MERGE, WAITING, loop_of
class Clock:
def __init__(self) -> None:
self.now = 1000.0
def __call__(self) -> float:
return self.now
def test_the_prose_nudge_keeps_requests_ending_on_the_user_side(
tmp_path: Path,
) -> None:
# ScriptedModel checks every request as the API would, now including
# "the conversation must end with a user message".
h = Harness(
tmp_path,
ScriptedModel([AssistantText(text="Let me think.", model="m")], RESTATED),
FakeGitea(issue()),
)
h.agent.intake()
assert loop_of(h.item()).wait == GO_AHEAD # the nudge worked
facts = [
e.event.text
for e in h.store.transcript(h.item().id, "orchestrator").read()
if isinstance(e.event, Fact)
]
assert any("Your last reply was prose" in f for f in facts)
class ContractHelper:
"""A helper that checks its requests as the API would."""
def __init__(self, *turns: list[Any]) -> None:
self.turns = list(turns)
def complete(self, events, tools, model, system=None) -> list[Any]:
assert_renderable(events)
return self.turns.pop(0)
def test_a_helpers_requests_end_on_the_user_side(world: dict[str, Any]) -> None:
helper = ContractHelper(
[AssistantText(text="Looking around first.", model="h")], # prose
[call("write_file", path="helped.py", content="H = 1\n")],
[hand_back()],
)
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="words"),
call("delegate", task="write helped.py"),
call("finish_milestone", summary="with help", final="yes"),
],
)
helpers = Helpers(
client=helper, # type: ignore[arg-type]
roles=ROLES,
client_for=lambda model: helper, # type: ignore[arg-type,return-value]
caps=Caps(turns=5),
)
agent, transport = loop_agent(world, model, helpers=helpers)
agent.intake()
say(transport, world, "go")
forge: Forge = world["forge"]
[pr] = forge.prs
assert forge.files(pr) == ["helped.py"]
def test_status_in_a_thread_is_not_a_reply(tmp_path: Path) -> None:
clock = Clock()
model = ScriptedModel(RESTATED) # called once only: status runs nothing
h = Harness(
tmp_path,
model,
FakeGitea(issue(labels=("bug",))),
clock=clock,
bug_grace_s=600,
)
h.agent.intake()
grace = loop_of(h.item()).grace_until
h.say("status")
assert h.posted()[-1].startswith("#14 Add count_words: waiting for go-ahead")
loop = loop_of(h.item())
assert (h.item().position, loop.wait, loop.grace_until) == (
WAITING,
GO_AHEAD,
grace,
)
h.say("details")
assert loop_of(h.item()).grace_until == grace
def test_a_remark_gets_a_reply_and_the_restatement_stays_up(tmp_path: Path) -> None:
h = Harness(
tmp_path,
ScriptedModel(
RESTATED,
[call("reply", text="Noted: I'll keep it to text.py.")],
),
FakeGitea(issue()),
)
h.agent.intake()
h.say("just a thought: keep it small")
loop = loop_of(h.item())
assert (loop.wait, loop.rev) == (GO_AHEAD, 1)
assert h.posted()[-1] == "Noted: I'll keep it to text.py."
def test_a_review_comment_can_be_answered_without_a_fix(
world: dict[str, Any],
) -> None:
forge: Forge = world["forge"]
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="words"),
call("write_file", path="w.py", content="W = 1\n"),
call("finish_milestone", summary="words", final="yes"),
],
[call("reply", text="It's a constant on purpose, for the test.")],
)
agent, transport = loop_agent(world, model)
agent.intake()
say(transport, world, "go")
[pr] = forge.prs
forge.comments[pr] = [Comment(7, "don", "why a constant", "t")]
agent.watch()
agent._schedule()
loop = loop_of(the_item(world))
assert (loop.wait, loop.pr) == (MERGE, pr)
assert forge.posted[-1][0] == pr # answered on the PR, where asked
assert "It's a constant on purpose" in forge.posted[-1][1]
# --- the session's later findings ----------------------------------------------
def test_a_near_miss_for_abandon_is_asked_about(tmp_path: Path) -> None:
model = ScriptedModel(RESTATED) # called once only: nothing reaches it
h = Harness(tmp_path, model, FakeGitea(issue()))
h.agent.intake()
h.say("abort")
assert h.posted()[-1].startswith("Did you mean `abandon`?")
assert (h.item().status, loop_of(h.item()).wait) == ("open", GO_AHEAD)
h.say("abandon")
assert h.item().status == "closed"
def test_the_driver_is_told_it_cant_stop_or_close(tmp_path: Path) -> None:
seen: list[str] = []
class Spy(ScriptedModel):
def complete(self, events, tools, model, system=None): # type: ignore[no-untyped-def]
seen.append(system or "")
return super().complete(events, tools, model, system)
h = Harness(tmp_path, Spy(RESTATED), FakeGitea(issue()))
h.agent.intake()
assert "never say you have stopped or closed anything yourself" in seen[0]
def test_an_issue_assigned_again_after_its_item_closed_is_taken_again(
tmp_path: Path,
) -> None:
gitea = FakeGitea(issue())
h = Harness(tmp_path, ScriptedModel(RESTATED, RESTATED), gitea)
h.agent.intake()
h.say("abandon")
[old] = h.store.work_items()
gitea.closed.remove(14) # the operator reopened it
h.agent.intake()
assert len(h.store.work_items()) == 1 # the same assignment: not again
gitea.reassigned[14] = "2999-01-01T00:00:00Z" # assigned anew, later
h.agent.intake()
items = h.store.work_items()
assert len(items) == 2
new = next(i for i in items if i.id != old.id)
assert new.status == "open" and loop_of(new).follows == old.id
def test_abandoning_a_blocked_issue_says_why_it_stays_open(tmp_path: Path) -> None:
from nisse_dev.gitea import GiteaError
class Strict(FakeGitea):
def close_issue(self, repo: str, index: int) -> None:
if self.open_dependencies(repo, index):
raise GiteaError("gitea PATCH returned 500: blocked")
super().close_issue(repo, index)
gitea = Strict(issue(14), issue(15, title="other"))
gitea.deps[14] = [15]
h = Harness(tmp_path, ScriptedModel(RESTATED), gitea)
h.agent._running = True # hold the queue: only the abandon matters
h.agent.intake()
item = next(i for i in h.store.work_items() if loop_of(i).issue == 14)
h.transport.receive(DON, "abandon", thread=loop_of(item).thread)
said = " ".join(h.posted())
assert "won't close an issue while it depends on open ones" in said
assert "don/sandbox#15" in said
def test_an_agreed_item_isnt_offered_restate(world: dict[str, Any]) -> None:
offered: list[set[str]] = []
class Spy(ScriptedModel):
def complete(self, events, tools, model, system=None): # type: ignore[no-untyped-def]
offered.append({t["function"]["name"] for t in tools})
return super().complete(events, tools, model, system)
model = Spy(RESTATE, [call("ask_human", question="anything else?")])
agent, transport = loop_agent(world, model)
agent.intake()
say(transport, world, "go")
assert "restate" in offered[0] and "restate" not in offered[1]
+4
View File
@@ -61,6 +61,7 @@ class Forge:
self.issues = [Issue(14, "Add count_words", "count words", "don", ())]
self.deps: dict[int, list[int]] = {} # issue -> the issues it depends on
self.assigned: list[int] = [] # issues the agent assigned to itself
self.reassigned: dict[int, str] = {} # issue -> when assigned again
# intake
def assigned_issues(self, repo: str, assignee: str) -> list[Issue]:
@@ -84,6 +85,9 @@ class Forge:
def assigned_by(self, repo: str, index: int, assignee: str) -> str:
return "don"
def assigned_at(self, repo: str, index: int, assignee: str) -> str | None:
return self.reassigned.get(index)
# PRs
def _pr(self, n: int) -> PullRequest:
p = self.prs[n]
+37 -4
View File
@@ -8,6 +8,7 @@ checks every property after every step. A failure prints
the seed and the steps, so it replays exactly.
"""
import os
import random
import re
import subprocess
@@ -46,8 +47,14 @@ from nisse_dev.tools import Workspace
from nisse_dev.transport import InboundMessage, InMemoryTransport
from nisse_dev.worker import InlineWorker
SEEDS = range(32)
TWO_PROJECT_SEEDS = range(100, 116)
# How many walks, and from which seed: all 48 by default; CI runs fewer on a
# PR, from a seed that moves with the run number, so PRs still cover new
# ground (NISSE_WALK_SEEDS, NISSE_WALK_FIRST). Two thirds walk one project,
# a third two.
_WALKS = int(os.environ.get("NISSE_WALK_SEEDS") or 48)
_FIRST = int(os.environ.get("NISSE_WALK_FIRST") or 0)
SEEDS = range(_FIRST, _FIRST + _WALKS * 2 // 3)
TWO_PROJECT_SEEDS = range(100 + _FIRST, 100 + _FIRST + _WALKS - _WALKS * 2 // 3)
STEPS = 30
@@ -76,8 +83,15 @@ class PolicyModel:
return [call("ask_human", question=f"question {self.n}?")]
if roll < 0.12:
return [ToolCall(call_id=f"p{self.n}", tool="nonsense", arguments={})]
if "reply" in offered and "propose_work" not in offered:
if "reply" in offered and not offered & {
"propose_work",
"restate",
"ask_human",
"start_milestone",
}:
return [call("reply", text=f"because {self.n}")] # a question run
if "reply" in offered and "propose_work" not in offered and roll < 0.2:
return [call("reply", text=f"noted {self.n}")] # a remark answered
if "reply" in offered: # a conversation: talk, or propose work
if self.rng.random() < 0.1:
return [
@@ -251,6 +265,7 @@ class Walk:
if item and loop:
weights.update({"go": 3, "answer": 2, "abandon": 1, "go-stale": 1})
weights.update({"plain-yes": 2, "wait": 2, "question": 2})
weights.update({"status-here": 1})
if loop.new_project:
weights.update({"done": 2})
if item.scope != "general": # conversations aren't named by number
@@ -266,6 +281,18 @@ class Walk:
elif action == "done" and loop:
# The operator says a new project is ready; no entry was added.
self.transport.receive(DON, "done", thread=loop.thread)
elif action == "status-here" and loop:
# L5: a command about the item, not a reply: nothing changes,
# not even a countdown.
before = (item.position, loop.wait, loop.rev, loop.grace_until)
self.transport.receive(DON, "status", thread=loop.thread)
after_item = self.world["store"].work_item(item.id)
after = loop_of(after_item)
now = (after_item.position, after.wait, after.rev, after.grace_until)
assert now == before, (
f"status_changes_nothing: {before} -> {now}: "
f"seed {self.seed}, steps {self.steps}"
)
elif action == "question" and loop:
# L5: a question never changes the item it's asked about.
before = (item.position, loop.wait, loop.rev, loop.pr, loop.branch)
@@ -373,7 +400,13 @@ class Walk:
where = f"seed {self.seed}, steps {self.steps}"
store = self.world["store"]
items = self.items()
issues = [(i.scope, loop_of(i).issue) for i in items if loop_of(i).issue]
# Open items: an issue assigned again after its item closed gets a new
# one (D1), which loop.qnt doesn't model.
issues = [
(i.scope, loop_of(i).issue)
for i in items
if loop_of(i).issue and i.status == "open"
]
assert len(issues) == len(set(issues)), f"one_item_per_issue: {where}"
for item in items:
loop = loop_of(item)
+2
View File
@@ -4,6 +4,7 @@ members = ["nisse-core", "nisse-dev", "nisse-ops"]
[dependency-groups]
dev = [
"pytest>=8",
"pytest-xdist>=3",
"ruff>=0.6",
]
@@ -19,6 +20,7 @@ addopts = "--strict-markers --ignore=tests/integration/seed"
markers = [
"integration: end-to-end tests that exercise installed entry points or external services",
"e2e: the whole agent against real services, run after each milestone (scripts/e2e.sh)",
"smoke: the end-to-end tests every PR runs; the rest on e2e:full and before a release",
]
[tool.ruff]
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env bash
# The bring-up's test repository (Development Plan, live runs):
# nisse-testing-operator/nisse-bringup, made as the end-to-end tests make
# theirs (tests/integration/e2e_support.py, make_repo), but kept between runs.
# Seeded with the test package, the bring-up agent's account a collaborator,
# main protected, and a deploy key of its own at $E2E/ssh/nisse-bringup.key.
#
# scripts/bringup-repo.sh create make it
# scripts/bringup-repo.sh reset delete it and make it again
# scripts/bringup-repo.sh delete delete it
#
# Needs NISSE_OPERATOR_TOKEN (.env here) and the bring-up agent's token
# (NISSE_GITEA_TOKEN in $E2E/.env), which names the agent's account.
set -euo pipefail
cd "$(dirname "$0")/.."
E2E=${NISSE_E2E:-/root/nisse-e2e}
case "${1:-}" in create|reset|delete) ;; *) sed -n '2,15p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;; esac
set -a; [ -f .env ] && . ./.env; set +a
exec uv run python - "$1" "$E2E" <<'PY'
import sys
from pathlib import Path
sys.path.insert(0, "tests/integration")
from e2e_support import OWNER, delete_repo, make_repo, operator_gitea # noqa: E402
from nisse_dev.envfile import load_env_file # noqa: E402
from nisse_dev.gitea import GiteaClient # noqa: E402
action, e2e = sys.argv[1], Path(sys.argv[2])
url = "https://gitea.dongar.net"
operator = operator_gitea(url)
name = "nisse-bringup"
if action in ("reset", "delete"):
delete_repo(operator, f"{OWNER}/{name}")
print(f"deleted {OWNER}/{name}")
if action in ("create", "reset"):
agent_env: dict[str, str] = {}
load_env_file(e2e / ".env", agent_env)
agent = GiteaClient(url, agent_env["NISSE_GITEA_TOKEN"]).whoami()
repo = make_repo(operator, name, e2e / "ssh", agent=agent)
print(f"made {repo.name}: {agent} a collaborator; push key {repo.push_key}")
PY
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# The throwaway sshd the tests need (nisse_dev/testing.py), in a CI job.
# Skipped when the job's image already has it. Otherwise apt fetches from a
# kernel.org mirror rather than archive.ubuntu.com, which has been slow
# enough (a few KB/s) to add minutes to every job.
set -euo pipefail
SUDO=$(command -v sudo || true)
if command -v /usr/sbin/sshd > /dev/null && command -v ssh > /dev/null; then
echo "sshd already installed"
else
MIRROR=${APT_MIRROR:-http://mirrors.edge.kernel.org/ubuntu}
# Both layouts: deb822 (ubuntu.sources, 24.04+) and sources.list (older).
for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
[ -f "$f" ] && $SUDO sed -i -E \
"s#https?://(archive|security)\.ubuntu\.com/ubuntu/?#$MIRROR/#g" "$f"
done
$SUDO apt-get update -qq
$SUDO apt-get install -y -qq openssh-server openssh-client > /dev/null
fi
$SUDO mkdir -p /run/sshd
+2
View File
@@ -23,6 +23,8 @@ if [ "${1:-}" = "--image" ]; then
mkdir -p "$runs"
secrets=()
[ -f .env ] && secrets=(--env-file .env)
# The model, if overridden here, reaches the tests inside too.
[ -n "${NISSE_E2E_MODEL:-}" ] && secrets+=(-e "NISSE_E2E_MODEL=$NISSE_E2E_MODEL")
# Host networking: the suite reaches Matrix, Gitea and LiteLLM as here.
exec docker run --rm --network host "${secrets[@]}" \
-e NISSE_E2E_RUNS=/runs -v "$runs:/runs" \
+42 -15
View File
@@ -15,8 +15,10 @@ from nisse_dev.llm import RoleMap
# The end-to-end tests run every role on a local model, which costs nothing
# per call; NISSE_E2E_MODEL overrides it (e.g. claude-haiku-4-5 for a quick,
# paid run).
E2E_MODEL = "qwen3.8-27b"
# paid run). The no-thinking alias (LiteLLM: chat_template_kwargs
# enable_thinking false): with thinking, some answers ran to 6-8k tokens,
# minutes each, and stretched a test from 3.5 to 20 minutes.
E2E_MODEL = "qwen3.8-27b-nothink"
def e2e_roles() -> RoleMap:
@@ -59,16 +61,14 @@ class FreshRepo:
push_key: Path # this repository's deploy key, private half
@contextlib.contextmanager
def fresh_repo(
operator: GiteaClient, run: str, keys: Path, *, agent: str, seed: bool = True
) -> Iterator[FreshRepo]:
"""A repository of its own for one test run, as the operator would make a
project's (D4, D8): seeded with a small Python package, the ``agent``
account a collaborator with write access, the default branch protected (pushes only by the
owner, an approval to merge), and a deploy key of its own, written under
``keys``. Deleted afterwards, unless NISSE_E2E_KEEP_REPOS is set."""
name = f"nisse-e2e-{run}"
def make_repo(
operator: GiteaClient, name: str, keys: Path, *, agent: str, seed: bool = True
) -> FreshRepo:
"""A repository as the operator would make a project's (D4, D8): seeded
with a small Python package (or only a README), the ``agent`` account a
collaborator with write access, the default branch protected (pushes only
by the owner, an approval to merge), and a deploy key of its own, written
under ``keys``. A failure part-way deletes what was made."""
full = f"{OWNER}/{name}"
operator._request(
"POST",
@@ -145,8 +145,35 @@ def fresh_repo(
"required_approvals": 1,
},
)
yield FreshRepo(full, key)
# The labels the agent reads (D1), as a project would have them.
for label, color in (
("bug", "#ee0701"),
("discuss", "#0e8a16"),
("design", "#5319e7"),
):
operator._request(
"POST", f"/repos/{full}/labels", json={"name": label, "color": color}
)
except BaseException:
delete_repo(operator, full)
raise
return FreshRepo(full, key)
def delete_repo(operator: GiteaClient, full: str) -> None:
with contextlib.suppress(GiteaError):
operator._request("DELETE", f"/repos/{full}")
@contextlib.contextmanager
def fresh_repo(
operator: GiteaClient, run: str, keys: Path, *, agent: str, seed: bool = True
) -> Iterator[FreshRepo]:
"""A repository of its own for one test run (make_repo), deleted
afterwards unless NISSE_E2E_KEEP_REPOS is set."""
repo = make_repo(operator, f"nisse-e2e-{run}", keys, agent=agent, seed=seed)
try:
yield repo
finally:
if not os.environ.get("NISSE_E2E_KEEP_REPOS"):
with contextlib.suppress(GiteaError):
operator._request("DELETE", f"/repos/{full}")
delete_repo(operator, repo.name)
@@ -239,6 +239,7 @@ class Operator:
return self.transport.outbound[-1].text if self.transport.outbound else ""
@pytest.mark.smoke
def test_the_slice_end_to_end(
running: tuple[RunningLoop, InMemoryTransport, str],
) -> None:
@@ -501,6 +502,7 @@ def test_a_two_milestone_feature(
operator.tidy(issues, prs)
@pytest.mark.smoke
def test_a_conversation_becomes_filed_work(
running: tuple[RunningLoop, InMemoryTransport, str],
) -> None:
@@ -636,6 +638,7 @@ def test_a_conversation_becomes_a_new_project(
assert any(f.startswith("docs/plans/") for f in files), files
@pytest.mark.smoke
def test_a_question_leaves_the_restatement_up(
running: tuple[RunningLoop, InMemoryTransport, str],
) -> None:
Generated
+23
View File
@@ -12,6 +12,7 @@ members = [
[manifest.dependency-groups]
dev = [
{ name = "pytest", specifier = ">=8" },
{ name = "pytest-xdist", specifier = ">=3" },
{ name = "ruff", specifier = ">=0.6" },
]
@@ -46,6 +47,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
]
[[package]]
name = "execnet"
version = "2.1.2"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/bf/89/780e11f9588d9e7128a3f87788354c7946a9cbb1401ad38a48c4db9a4f07/execnet-2.1.2.tar.gz", hash = "sha256:63d83bfdd9a23e35b9c6a3261412324f964c2ec8dcd8d3c6916ee9373e0befcd", size = 166622, upload-time = "2025-11-12T09:56:37.75Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ab/84/02fc1827e8cdded4aa65baef11296a9bbe595c474f0d6d758af082d849fd/execnet-2.1.2-py3-none-any.whl", hash = "sha256:67fba928dd5a544b783f6056f449e5e3931a5c378b128bc18501f7ea79e296ec", size = 40708, upload-time = "2025-11-12T09:56:36.333Z" },
]
[[package]]
name = "h11"
version = "0.16.0"
@@ -177,6 +187,19 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/24/25/1de2678b631f5a49215c6c96fff41ba892b0a34df68d6d80292b1b48aa7f/pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c", size = 386536, upload-time = "2026-06-19T10:58:31.347Z" },
]
[[package]]
name = "pytest-xdist"
version = "3.8.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "execnet" },
{ name = "pytest" },
]
sdist = { url = "https://files.pythonhosted.org/packages/78/b4/439b179d1ff526791eb921115fca8e44e596a13efeda518b9d845a619450/pytest_xdist-3.8.0.tar.gz", hash = "sha256:7e578125ec9bc6050861aa93f2d59f1d8d085595d6551c2c90b6f4fad8d3a9f1", size = 88069, upload-time = "2025-07-01T13:30:59.346Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl", hash = "sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88", size = 46396, upload-time = "2025-07-01T13:30:56.632Z" },
]
[[package]]
name = "pyyaml"
version = "6.0.3"