Compare commits

..
1 Commits
Author SHA1 Message Date
dongarandClaude Opus 5.5 dd6196c742 New projects from conversation (M2.5, PR B)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 4m56s
- propose_project(name, purpose, decisions) in a conversation goes up for
  a go-ahead (checked: a slug, not an existing project, no trailing
  question).
- On go, the provisioning checklist (D1): a repository with a README,
  protection on its default branch, the agent as a collaborator, a build
  VM, a deploy key, and the project entry as YAML with <...> to fill in.
- On "done", the agent reads its configuration again and adds the new
  project live (build_loop_agent(load=...); only additions), then checks
  what was provided (D4, verified, not assumed): permissions and
  protection as it sees them, then on a throwaway branch with an empty
  commit, a branch push must succeed and a push to the default branch and
  merging its own PR must be refused. The check's PR is closed and its
  branch deleted. Failures are said per item; nothing starts.
- Checked: the tracking issue (the distilled decisions, carrying the new
  item's ID), and the project's item, agreed and labelled `design`, whose
  first milestone is ARCHITECTURE.md and the build plan. A conversation
  waiting on provisioning doesn't close as idle.
- The walk replays exactly from its seed (item IDs too), and event IDs
  stay unique across simulated restarts, as a server's are.
- The test config holds output to 8192 tokens, under the test key's
  per-request cost cap.

Tests: new-project unit tests (checklist and its YAML parse as config, done
before the entry, what isn't set up, a checked project's record and first
PR, an existing name), protection enforced by a hook on the fake forge,
the walk proposing projects and saying done, and an e2e new project in a
repository the test operator provisions from the agent's checklist.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-06 14:07:37 +00:00
34 changed files with 101 additions and 1860 deletions

No files matched your search

-9
View File
@@ -1,9 +0,0 @@
.git
.env
**/.venv
**/__pycache__
**/.pytest_cache
**/.ruff_cache
_apalache-out
tmp
bench
-40
View File
@@ -54,43 +54,3 @@ jobs:
# key) and deletes it afterwards. # key) and deletes it afterwards.
NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }} NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }}
run: uv run pytest -m "integration and not e2e" -rs run: uv run pytest -m "integration and not e2e" -rs
e2e:
# The end-to-end suite (scripts/e2e.sh): the whole agent against real
# services, every role on a local model, so it spends nothing but keeps
# the RTX 4090 busy for ~25 minutes. On PRs only, and only after the
# checks pass: a PR that's red already doesn't spend the GPU. There is
# one runner and Gitea doesn't honour cancel-in-progress, so a run a
# newer push supersedes holds the runner until it's cancelled by hand.
# Locally, run only what you're writing (-k) or diagnosing (--lf).
if: github.event_name != 'push'
needs: checks
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
- name: Install uv
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Sync workspace
run: uv sync --all-packages
- name: Install sshd
# The build VM is a throwaway sshd here (scripts/e2e.sh --local).
run: |
SUDO=$(command -v sudo || true)
$SUDO apt-get update -qq
$SUDO apt-get install -y -qq openssh-server openssh-client > /dev/null
$SUDO mkdir -p /run/sshd
- name: End-to-end tests
env:
NISSE_LLM_TOKEN: ${{ secrets.NISSE_LLM_TOKEN }}
NISSE_MATRIX_TOKEN: ${{ secrets.NISSE_MATRIX_TOKEN }}
NISSE_GITEA_TOKEN: ${{ secrets.NISSE_GITEA_TOKEN }}
NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }}
NISSE_E2E_RUNS: /tmp/nisse-e2e-runs
run: scripts/e2e.sh --local -v
-63
View File
@@ -1,63 +0,0 @@
name: Release
# A published release builds the agent's image and pushes it to Gitea's
# registry (Development Plan M1.5, X4): gitea.dongar.net/dev/nisse-dev
# tagged with the version (the release tag without its v) and latest. Cut a
# release in Gitea (Releases → New release, tag vX.Y.Z); a tag pushed for any
# other reason builds nothing.
#
# The registry needs a personal access token with write:package
# (REGISTRY_TOKEN, an organization secret of `dev`, belonging to the
# registry-push bot), never the built-in job token: Gitea's registry
# resolves an Actions job token to an empty package scope (go-gitea/gitea
# #23642 and others). Packages belong to the repository's owner, which is
# why this repository is the `dev` organization's: the bot writes packages
# through the org's registry-push team. Never on pull_request: registry
# credentials stay off that trigger.
on:
release:
types: [published]
env:
REGISTRY: gitea.dongar.net
REGISTRY_USER: registry-push # the token's account
IMAGE: gitea.dongar.net/dev/nisse-dev
jobs:
build-and-push:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Derive the image version from the release's tag
id: vars
env:
TAG: ${{ github.event.release.tag_name }}
run: echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
- name: Log in to Gitea's container registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ env.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and push the agent
uses: docker/build-push-action@v5
with:
context: .
target: agent
push: true
build-args: |
VERSION=${{ steps.vars.outputs.version }}
labels: |
org.opencontainers.image.version=${{ steps.vars.outputs.version }}
org.opencontainers.image.revision=${{ github.sha }}
tags: |
${{ env.IMAGE }}:${{ steps.vars.outputs.version }}
${{ env.IMAGE }}:latest
-68
View File
@@ -1,68 +0,0 @@
# The nisse-dev agent as a container image (Development Plan M1.5, X4).
#
# Stateless: the state store and SSH known-hosts live in a volume
# (/var/lib/nisse-dev); configuration is mounted read-only
# (/etc/nisse-dev/nisse-dev.yaml); secrets come as environment variables
# (docker --env-file); SSH and deploy keys are mounted read-only wherever the
# configuration says. The agent runs no model-chosen command here (L13):
# those go to build VMs over SSH.
#
# docker build -t nisse-dev . the agent
# docker build --target test -t nisse-dev:test . plus the tests, for
# scripts/e2e.sh --image
FROM python:3.12-slim AS build
COPY --from=ghcr.io/astral-sh/uv:0.12.10 /uv /usr/local/bin/uv
ENV UV_PROJECT_ENVIRONMENT=/opt/nisse UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
WORKDIR /src
COPY pyproject.toml uv.lock ./
COPY nisse-core nisse-core
COPY nisse-dev nisse-dev
COPY nisse-ops nisse-ops
# The agent and what it needs, installed (not editable): the source tree
# isn't part of the image.
RUN uv sync --frozen --no-dev --package nisse-dev --no-editable
FROM python:3.12-slim AS runtime
ARG VERSION=dev
LABEL org.opencontainers.image.title=nisse-dev org.opencontainers.image.version=$VERSION
RUN apt-get update \
&& apt-get install -y --no-install-recommends openssh-client \
&& rm -rf /var/lib/apt/lists/* \
&& useradd --create-home --uid 10001 nisse \
&& mkdir -p /var/lib/nisse-dev /etc/nisse-dev \
&& chown nisse /var/lib/nisse-dev
COPY --from=build /opt/nisse /opt/nisse
ENV PATH=/opt/nisse/bin:$PATH \
NISSE_CONFIG=/etc/nisse-dev/nisse-dev.yaml \
NISSE_ENV=/run/nisse-dev/.env
VOLUME /var/lib/nisse-dev
USER nisse
ENTRYPOINT ["nisse-dev"]
# The image's own package plus the tests, so the end-to-end suite runs
# against what was built, not the source tree.
FROM runtime AS test
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends git openssh-server \
&& rm -rf /var/lib/apt/lists/* \
&& mkdir -p /run/sshd
COPY --from=build /usr/local/bin/uv /usr/local/bin/uv
WORKDIR /src
COPY pyproject.toml uv.lock ./
COPY nisse-core nisse-core
COPY nisse-dev nisse-dev
COPY nisse-ops nisse-ops
COPY tests tests
COPY scripts scripts
COPY conftest.py ./
ENV UV_PROJECT_ENVIRONMENT=/opt/nisse
# Test tools too, all installed (not editable), and `uv run` never re-syncs:
# the tests import the package as built, not the source tree beside them.
RUN uv sync --frozen --all-packages --no-editable --inexact
ENV UV_NO_SYNC=1
ENTRYPOINT []
# The default target: the agent itself.
FROM runtime AS agent
+10 -47
View File
@@ -31,7 +31,7 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
| **L6** Context | partial | Context fitting, per-role views and orchestrator facts (M1.2); no scope or agent memory | | **L6** Context | partial | Context fitting, per-role views and orchestrator facts (M1.2); no scope or agent memory |
| **L7** Canonical log | partial | Work item and actor on every event, `fact` (M1.1, M1.2); no `classifier_verdict`, `knowledge_retrieved` | | **L7** Canonical log | partial | Work item and actor on every event, `fact` (M1.1, M1.2); no `classifier_verdict`, `knowledge_retrieved` |
| **L8** Durability | mostly done | WAL store, intents, boot reconciliation, PR adoption, comment redelivery; to move onto the new loop | | **L8** Durability | mostly done | WAL store, intents, boot reconciliation, PR adoption, comment redelivery; to move onto the new loop |
| **L9** Models and roles | mostly done | LiteLLM, eval capture; on the new loop: `driver`/`helper`/`expert`/`classifier` pinned per item, capability profiles in briefings, delegation and `ask_expert` with triggers (M2.6) | | **L9** Models and roles | partial | LiteLLM, delegation, `ask_expert`, eval capture; roles are still planner/executor, no capability profiles |
| **L10** Knowledge vault | none | | | **L10** Knowledge vault | none | |
| **L11** Injection | partial | Operator allowlists on every channel; no classifier | | **L11** Injection | partial | Operator allowlists on every channel; no classifier |
| **L12** Secrets, logging | partial | Structured state-change log; secrets in `.env`; no per-operation handout or redaction | | **L12** Secrets, logging | partial | Structured state-change log; secrets in `.env`; no per-operation handout or redaction |
@@ -73,12 +73,9 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
`CLAUDE.md`. A milestone marked **needs discussion** starts with a design `CLAUDE.md`. A milestone marked **needs discussion** starts with a design
discussion first. discussion first.
- **Every change comes with tests,** through `scripts/check.sh`. - **Every change comes with tests,** through `scripts/check.sh`.
- **The end-to-end test runs on every PR** (M1.6): the real agent, every role on a - **The end-to-end test follows each milestone** (M1.6): the real agent, real models,
local model (so it costs nothing per call), a repository of the run's own and a a repository of the run's own and a real build host, driven by a scripted operator. It runs
real build host, driven by a scripted operator. CI runs it as its own job on pull after each milestone, not on every push, since it spends model tokens. Manual
requests, after the checks pass. There is one runner, and a run a newer push
supersedes has to be cancelled by hand. Locally only the tests being written
(`-k`) or diagnosed (`--lf`); `scripts/e2e.sh` reports what a run spent. Manual
live runs are for the new loop's first runs and phase ends. Findings are filed as live runs are for the new loop's first runs and phase ends. Findings are filed as
issues and either fixed before moving on or deliberately deferred. issues and either fixed before moving on or deliberately deferred.
- **The Quint models move with the loop:** `orchestrator.qnt` while `control.py` - **The Quint models move with the loop:** `orchestrator.qnt` while `control.py`
@@ -223,37 +220,12 @@ deletes afterwards.
`ARCHITECTURE.md` and the build plan. A conversation waiting on provisioning `ARCHITECTURE.md` and the build plan. A conversation waiting on provisioning
doesn't close as idle. doesn't close as idle.
**M2.6 Roles and capability profiles.** *Pre-approved.* **Done** (this PR). **M2.6 Roles and capability profiles.** *Pre-approved.* `driver`, `helper`,
`driver`, `helper`, `expert`, `classifier` (L9); escalation triggers; delegation and `expert`, `classifier` (L9); escalation triggers; delegation and `ask_expert` on the
`ask_expert` on the new loop. new loop.
- `roles.driver` (unset: the planner's model) and `roles.classifier` (unset: the
expert's), pinned per item; the driver and the go-ahead reader run on the item's.
- `profiles:` in configuration (tools, strengths, weaknesses), used in briefings:
the driver is told what its helper and expert are good at, the helper who will
read its handback; never a model's name.
- `delegate(task)` and `ask_expert(question)` for the driver. A helper works on a
branch off the milestone's; its work comes back as uncommitted changes on a
handback and is discarded otherwise, so harness branches never hold commits only
the clone has (the walk found a fix stranded that way when a PR merged mid-help).
- Triggers: `failure_limit` failed commands in a row, or the turn cap, mid-
milestone, call a helper in once per milestone; if it fails too, the item waits
on the operator as before. A delegation cut off by a crash is discarded at boot.
- Evaluation capture: delegations, `ask_expert` and the reader's verdicts.
- Deferred: the context-won't-fit trigger (L6); the action classifier (M3.2).
**M2.7 Conversation.** *Pre-approved.* **Done** (this PR). Explanations, progress **M2.7 Conversation.** *Pre-approved.* Explanations, progress in prose, asking
in prose, asking mid-work (L5, #42, #77). mid-work (L5, #42, #77).
- Questions never change state: a message ending in `?` to a waiting project item
is answered in a question run (`reply` and look-only tools), and the item goes
back to the very wait it was in, so a restatement up stays up and a go binds to
it.
- A restatement gives its reasons and the main alternative; a revision says what
changed (`restate(changed)`, sent back without it).
- Progress: the driver's and a helper's own words for a turn are posted, trimmed,
at most once a minute per item; `details` shows the last tool calls. A message
arriving mid-run is acknowledged at once.
- Deferred: streaming (#42); the question property in `specs/loop.qnt` (the walk
checks it against the code).
**M1.4 Projects** (D8). *Pre-approved.* **Done** (this PR). **M1.4 Projects** (D8). *Pre-approved.* **Done** (this PR).
- The project entry in configuration: repository, default branch, build VM, label - The project entry in configuration: repository, default branch, build VM, label
@@ -271,20 +243,11 @@ in prose, asking mid-work (L5, #42, #77).
- Deferred: config reload without a restart (with M2.5's entry step if wanted); - Deferred: config reload without a restart (with M2.5's entry step if wanted);
the trust dial's configuration (M3.3). the trust dial's configuration (M3.3).
**M1.5 The agent as a container image.** *Pre-approved.* **Built** (this PR); **M1.5 The agent as a container image.** *Pre-approved.*
accepted once a tag's image passes the e2e suite.
- A Dockerfile for `nisse-dev`, stateless with a volume for state; a Gitea Actions - A Dockerfile for `nisse-dev`, stateless with a volume for state; a Gitea Actions
workflow that builds it and pushes to Gitea's registry on each release tag (X4). workflow that builds it and pushes to Gitea's registry on each release tag (X4).
- `bringup.sh` runs the image instead of a venv. - `bringup.sh` runs the image instead of a venv.
- Acceptance: a tagged release produces an image; the end-to-end test runs from it. - Acceptance: a tagged release produces an image; the end-to-end test runs from it.
- The image runs as its own user; state in `/var/lib/nisse-dev`, configuration
mounted at `/etc/nisse-dev/nisse-dev.yaml`, secrets as environment variables,
keys mounted read-only. Its `test` stage carries the tests over the installed
package, for `scripts/e2e.sh --image REF`. Images are built only on release:
`release.yml` pushes `gitea.dongar.net/dev/nisse-dev:<version>` and `:latest`
on a published release (`REGISTRY_TOKEN`, the `dev` organization's).
- The live bring-up's switch to the image waits for the next live test (it needs
its own test repository too); `bringup.sh plan` shows what it will run.
**M2.8 Switch over.** *Pre-approved, after acceptance.* `nisse-dev` runs the new **M2.8 Switch over.** *Pre-approved, after acceptance.* `nisse-dev` runs the new
loop; `control.py`, its tests and `orchestrator.qnt` are removed. loop; `control.py`, its tests and `orchestrator.qnt` are removed.
-21
View File
@@ -45,27 +45,6 @@ roles: # L9: role -> LiteLLM model alias
planner: claude-opus-5-5 # high-end judgement: plans a weaker executor follows planner: claude-opus-5-5 # high-end judgement: plans a weaker executor follows
helper: claude-opus-5-5 # takes over what the executor got stuck on helper: claude-opus-5-5 # takes over what the executor got stuck on
expert: claude-haiku-4-5 # one question at a time, no tools: cheap is enough expert: claude-haiku-4-5 # one question at a time, no tools: cheap is enough
# The new loop's roles (loop: new): unset, driver is the planner's model and
# classifier the expert's. Each work item pins the mapping when it opens.
# driver: claude-opus-5-5 # drives the loop: talks, restates, works
# classifier: claude-haiku-4-5 # reads plain-language go-aheads (later: actions, L4)
# Capability profiles (L9), from benchmarks: what one role is told about
# another (never a model's name). Optional; a model without one is briefed as
# "no profile recorded".
profiles:
qwen3.8-27b:
tools: reliable (37/40 on the executor benchmark, PR #55)
strengths: careful multi-step edits that follow a clear brief
weaknesses: speed (minutes per hard task) and open-ended design
claude-haiku-4-5:
tools: reliable
strengths: quick, well-scoped changes and short answers
weaknesses: long multi-file work (15/20 on the harder benchmark tasks)
claude-opus-5-5:
tools: reliable
strengths: design, debugging, and work other models got stuck on
weaknesses: cost; use it for what needs it
matrix: # L5 matrix: # L5
homeserver: https://matrix.example homeserver: https://matrix.example
+1 -52
View File
@@ -46,8 +46,6 @@ REQUIRED_KEYS = (
"gitea.operators", "gitea.operators",
) )
OPTIONAL_KEYS = ( OPTIONAL_KEYS = (
"roles.driver",
"roles.classifier",
"llm.max_output_tokens", "llm.max_output_tokens",
"llm.request_timeout_s", "llm.request_timeout_s",
"loop", "loop",
@@ -381,48 +379,6 @@ class BuildHost:
port: int = 22 port: int = 22
@dataclass(frozen=True)
class Profile:
"""A model's capability profile (L9), from benchmarks: what one role is
told about another, never the model's name."""
tools: str = "" # how reliably it keeps the tool-call contract
strengths: str = ""
weaknesses: str = ""
def describe(self) -> str:
parts = [
f"tool calls: {self.tools}" if self.tools else "",
f"strong at {self.strengths}" if self.strengths else "",
f"weak at {self.weaknesses}" if self.weaknesses else "",
]
return "; ".join(p for p in parts if p) or "no profile recorded"
def _parse_profiles(raw: Any, errors: list[str]) -> dict[str, "Profile"]:
"""`profiles:`, model alias -> {tools, strengths, weaknesses}."""
if raw is None:
return {}
if not isinstance(raw, dict):
errors.append("profiles: must be a mapping of model alias to profile")
return {}
profiles: dict[str, Profile] = {}
for model, spec in raw.items():
where = f"profiles.{model}"
if not isinstance(spec, dict):
errors.append(f"{where}: must be a mapping")
continue
unknown = set(spec) - {"tools", "strengths", "weaknesses"}
if unknown:
errors.append(f"{where}.{min(unknown)}: unknown setting")
continue
if not all(isinstance(v, str) for v in spec.values()):
errors.append(f"{where}: values are text")
continue
profiles[str(model)] = Profile(**{k: v.strip() for k, v in spec.items()})
return profiles
@dataclass(frozen=True) @dataclass(frozen=True)
class Secret: class Secret:
value: str = field(repr=False) value: str = field(repr=False)
@@ -470,7 +426,6 @@ class AgentConfig:
gitea_token: str = field(repr=False) gitea_token: str = field(repr=False)
gitea_operators: frozenset[str] gitea_operators: frozenset[str]
projects: tuple[Project, ...] projects: tuple[Project, ...]
profiles: Mapping[str, Profile] = field(default_factory=dict)
max_output_tokens: int = DEFAULT_MAX_OUTPUT_TOKENS max_output_tokens: int = DEFAULT_MAX_OUTPUT_TOKENS
request_timeout_s: float = DEFAULT_REQUEST_TIMEOUT_S request_timeout_s: float = DEFAULT_REQUEST_TIMEOUT_S
tuning: Tuning = field(default_factory=Tuning) tuning: Tuning = field(default_factory=Tuning)
@@ -565,10 +520,7 @@ class AgentConfig:
) -> "AgentConfig": ) -> "AgentConfig":
tuning = _parse_tuning(tree.get("tuning"), errors) tuning = _parse_tuning(tree.get("tuning"), errors)
projects = _parse_projects(tree.get("projects"), env, errors) projects = _parse_projects(tree.get("projects"), env, errors)
profiles = _parse_profiles(tree.get("profiles"), errors) flat = _flatten({k: v for k, v in tree.items() if k != "projects"})
flat = _flatten(
{k: v for k, v in tree.items() if k not in ("projects", "profiles")}
)
flat.pop("tuning", None) flat.pop("tuning", None)
known = set(REQUIRED_KEYS) | set(OPTIONAL_KEYS) known = set(REQUIRED_KEYS) | set(OPTIONAL_KEYS)
@@ -622,8 +574,6 @@ class AgentConfig:
planner=str(values["roles.planner"]), planner=str(values["roles.planner"]),
helper=str(values["roles.helper"]), helper=str(values["roles.helper"]),
expert=str(values["roles.expert"]), expert=str(values["roles.expert"]),
driver=str(values.get("roles.driver") or ""),
classifier=str(values.get("roles.classifier") or ""),
), ),
matrix=MatrixConfig( matrix=MatrixConfig(
homeserver=str(values["matrix.homeserver"]).rstrip("/"), homeserver=str(values["matrix.homeserver"]).rstrip("/"),
@@ -636,7 +586,6 @@ class AgentConfig:
gitea_token=str(values["gitea.token"]), gitea_token=str(values["gitea.token"]),
gitea_operators=_people(values["gitea.operators"]), gitea_operators=_people(values["gitea.operators"]),
projects=projects, projects=projects,
profiles=profiles,
max_output_tokens=max_output, max_output_tokens=max_output,
request_timeout_s=request_timeout, request_timeout_s=request_timeout,
tuning=tuning, tuning=tuning,
+3 -16
View File
@@ -54,8 +54,8 @@ Stay inside the sub-task you were given. It comes from a plan a human already
approved; widening it is not yours to do. If the sub-task turns out to be approved; widening it is not yours to do. If the sub-task turns out to be
wrong or impossible, say so in the handback rather than substituting your own. wrong or impossible, say so in the handback rather than substituting your own.
When you are done, call hand_back exactly once. Write it for {reader}, When you are done, call hand_back exactly once. Write it for a materially
which cannot see your reasoning and will act on your note alone: weaker model that cannot see your reasoning and will act on your note alone:
- what changed - what changed
- what is now true that was not before - what is now true that was not before
@@ -172,17 +172,10 @@ class Delegation:
caps: Caps | None = None, caps: Caps | None = None,
now: Callable[[], float] = time.monotonic, now: Callable[[], float] = time.monotonic,
cancel: Cancel | None = None, cancel: Cancel | None = None,
reader: str = "",
on_turn: Callable[[str], None] | None = None,
) -> None: ) -> None:
self._client = client self._client = client
self._store = store self._store = store
self._workspace = workspace self._workspace = workspace
# Who acts on the handback, from its capability profile (L9): never
# a model's name.
self._reader = reader or "a materially weaker model"
# The delegate's own text for each turn, for progress (#42).
self._on_turn = on_turn
self._git = git_repo self._git = git_repo
self._model = model self._model = model
self._backing_model = backing_model self._backing_model = backing_model
@@ -330,8 +323,7 @@ class Delegation:
self, transcript: Transcript, brief: str, branch: str, started_at: float self, transcript: Transcript, brief: str, branch: str, started_at: float
) -> DelegationResult: ) -> DelegationResult:
system = ( system = (
f"{SYSTEM_PROMPT.format(reader=self._reader)}\n\n" f"{SYSTEM_PROMPT}\n\n{self._workspace.describe()}"
f"{self._workspace.describe()}"
f"\n\nYour sub-task:\n{brief}" f"\n\nYour sub-task:\n{brief}"
) )
touched: set[str] = set() touched: set[str] = set()
@@ -403,11 +395,6 @@ class Delegation:
transcript.append(event) transcript.append(event)
if isinstance(event, ToolCall): if isinstance(event, ToolCall):
calls.append(event) calls.append(event)
said = " ".join(
e.text.strip() for e in produced if isinstance(e, AssistantText)
).strip()
if said and self._on_turn is not None:
self._on_turn(said)
if not calls: if not calls:
# A delegate that stops without handing back has not reported, # A delegate that stops without handing back has not reported,
-7
View File
@@ -255,13 +255,6 @@ class GitRepo:
self._git("commit", "-q", "-m", message) self._git("commit", "-q", "-m", message)
return self.head_sha() return self.head_sha()
def changed_paths(self, since: str) -> list[str]:
"""Every path the work changes relative to ``since``, committed or
not, new files included; build artifacts aren't work (#80)."""
tracked = self._git("diff", "--name-only", since).splitlines()
new = self._git("ls-files", "--others", "--exclude-standard").splitlines()
return sorted({p for p in [*tracked, *new] if p})
def left_out(self) -> list[str]: def left_out(self) -> list[str]:
"""Changes not in the last commit: build artifacts the harness left """Changes not in the last commit: build artifacts the harness left
out, to be reported rather than silently dropped (D4).""" out, to be reported rather than silently dropped (D4)."""
+13 -58
View File
@@ -12,7 +12,7 @@ config change cannot split a task across two planners.
import json import json
import time import time
from dataclasses import dataclass from dataclasses import asdict, dataclass
from typing import Any, ClassVar, Protocol from typing import Any, ClassVar, Protocol
import httpx import httpx
@@ -39,13 +39,6 @@ class LLMError(RuntimeError):
RATE_LIMIT_STATUS = 429 RATE_LIMIT_STATUS = 429
RATE_LIMIT_RETRY_S = 20.0 RATE_LIMIT_RETRY_S = 20.0
# The proxy, or what's behind it, briefly gone: a 502/503/504 or a dropped
# connection. Outages delay, never fail (L5): waited out with backoff before
# the work is stopped. Not a 500 (it may be the request) and not a timeout
# (a hung model has had its request_timeout_s already).
OUTAGE_STATUSES = frozenset({502, 503, 504})
OUTAGE_BACKOFF_S = (5.0, 15.0, 45.0)
# Generous by default. Output is billed by what is produced, not by the # Generous by default. Output is billed by what is produced, not by the
# ceiling, and a ceiling too low does not fail loudly — it truncates a tool # ceiling, and a ceiling too low does not fail loudly — it truncates a tool
# call mid-argument and hands back something that parses. Measured: Opus # call mid-argument and hands back something that parses. Measured: Opus
@@ -114,40 +107,21 @@ class LLMClient(Protocol):
@dataclass(frozen=True) @dataclass(frozen=True)
class RoleMap: class RoleMap:
executor: str # the old loop's (control.py), until M2.8 executor: str
planner: str # the old loop's, until M2.8 planner: str
helper: str # delegations (L9) helper: str
expert: str # ask_expert (L9) expert: str
driver: str = "" # drives the new loop; unset, the planner's model
classifier: str = "" # reads go-aheads now, actions later (L4); unset, expert's
ROLES: ClassVar[tuple[str, ...]] = ( ROLES: ClassVar[tuple[str, ...]] = ("executor", "planner", "helper", "expert")
"executor",
"planner",
"helper",
"expert",
"driver",
"classifier",
)
_FALLBACK: ClassVar[dict[str, str]] = {"driver": "planner", "classifier": "expert"}
def resolve(self, role: str) -> str: def resolve(self, role: str) -> str:
if role not in self.ROLES: if role not in self.ROLES:
raise LLMError(f"unknown role: {role!r}") raise LLMError(f"unknown role: {role!r}")
model = str(getattr(self, role)) return str(getattr(self, role))
if not model and role in self._FALLBACK:
return self.resolve(self._FALLBACK[role])
return model
def pinned(self) -> dict[str, str]: def pinned(self) -> dict[str, str]:
"""The full resolved mapping, to record at task start (L9).""" """The full resolved mapping, to record at task start (L9)."""
return {role: self.resolve(role) for role in self.ROLES} return asdict(self)
@classmethod
def from_pinned(cls, pinned: dict[str, str], default: "RoleMap") -> "RoleMap":
"""A work item's pinned mapping; a role it predates comes from
``default``."""
return cls(**{r: pinned.get(r) or default.resolve(r) for r in cls.ROLES})
class LiteLLMClient: class LiteLLMClient:
@@ -388,39 +362,20 @@ class AnthropicPassthroughClient:
def _post_with_rate_limit_retry( def _post_with_rate_limit_retry(
http: httpx.Client, path: str, body: dict[str, Any], sleep: Any http: httpx.Client, path: str, body: dict[str, Any], sleep: Any
) -> tuple[httpx.Response, float]: ) -> tuple[httpx.Response, float]:
"""The response, and how long was spent waiting: out a rate limit (once) """The response, and how long was spent waiting out a rate limit."""
or an outage (with backoff), not on the model."""
waited = 0.0 waited = 0.0
rate_limited = False for attempt in (1, 2):
outages = iter(OUTAGE_BACKOFF_S)
while True:
try: try:
response = http.post(path, json=body) response = http.post(path, json=body)
except httpx.TimeoutException as exc:
raise UpstreamUnavailable(f"LLM request failed: {exc}") from exc
except httpx.HTTPError as exc: except httpx.HTTPError as exc:
# Could not reach the proxy at all: nothing about the request was # Could not reach the proxy at all: nothing about the request was
# wrong, so callers holding work in progress should keep it. # wrong, so callers holding work in progress should keep it.
pause = next(outages, None)
if pause is None:
raise UpstreamUnavailable(f"LLM request failed: {exc}") from exc raise UpstreamUnavailable(f"LLM request failed: {exc}") from exc
sleep(pause) if response.status_code != RATE_LIMIT_STATUS or attempt == 2:
waited += pause return response, waited
continue
if response.status_code == RATE_LIMIT_STATUS and not rate_limited:
rate_limited = True
sleep(RATE_LIMIT_RETRY_S) sleep(RATE_LIMIT_RETRY_S)
waited += RATE_LIMIT_RETRY_S waited += RATE_LIMIT_RETRY_S
continue raise AssertionError("unreachable")
if response.status_code in OUTAGE_STATUSES and not _looks_like_budget_refusal(
response.text
):
pause = next(outages, None)
if pause is not None:
sleep(pause)
waited += pause
continue
return response, waited
# Model families we have a native renderer adapter for. L9: delegations take # Model families we have a native renderer adapter for. L9: delegations take
+15 -426
View File
@@ -15,7 +15,6 @@ a message is recorded and changes its item's state, but never interrupts a
run (L1). run (L1).
""" """
import contextlib
import re import re
import threading import threading
import time import time
@@ -24,24 +23,12 @@ from collections.abc import Callable, Iterable, Sequence
from dataclasses import dataclass, replace from dataclasses import dataclass, replace
from nisse_dev.cancel import Cancel from nisse_dev.cancel import Cancel
from nisse_dev.canonical import ( from nisse_dev.canonical import Fact, PlanSubmitted, UserMessage
DelegationCompleted,
DelegationStarted,
Fact,
PlanSubmitted,
ToolCall,
ToolResult,
UserMessage,
)
from nisse_dev.config import GENERAL from nisse_dev.config import GENERAL
from nisse_dev.delegation import DELEGATION_INTENT, Delegation
from nisse_dev.expert import BudgetExhausted, ExpertDesk
from nisse_dev.gates import command_form from nisse_dev.gates import command_form
from nisse_dev.git import GitError from nisse_dev.git import GitError
from nisse_dev.gitea import GiteaClient, GiteaError, Issue from nisse_dev.gitea import GiteaClient, GiteaError, Issue
from nisse_dev.llm import LLMError, RoleMap
from nisse_dev.loop.driver import Driver, RunOutcome from nisse_dev.loop.driver import Driver, RunOutcome
from nisse_dev.loop.help import ASK_EXPERT, DELEGATE, Helpers
from nisse_dev.loop.items import ( from nisse_dev.loop.items import (
ANSWER, ANSWER,
BLOCKED, BLOCKED,
@@ -89,9 +76,6 @@ _NAMED = re.compile(r"(go|approve|abandon|stop) (?:([a-z0-9][a-z0-9-]*))?#(\d+)"
_NEEDS_ITEM = frozenset({"go", "approve", "abandon"}) _NEEDS_ITEM = frozenset({"go", "approve", "abandon"})
BUG_GRACE_S = 600.0 BUG_GRACE_S = 600.0
GENERAL_IDLE_S = 86400.0 GENERAL_IDLE_S = 86400.0
PROGRESS_EVERY_S = 60.0 # progress in prose at a readable rate (L5, #42)
PROGRESS_CHARS = 400
DETAILS = 10 # tool calls shown by `details`
FILE_ISSUE = "file_issue" # the intent of opening a record issue (L8) FILE_ISSUE = "file_issue" # the intent of opening a record issue (L8)
# Carried by a record issue the agent opens for work started in conversation: # Carried by a record issue the agent opens for work started in conversation:
# intake never takes it for new work, even before its item exists (D1). # intake never takes it for new work, even before its item exists (D1).
@@ -107,14 +91,9 @@ The issue and the conversation about it are below.
Before any work, find out what you need, then use restate to put your Before any work, find out what you need, then use restate to put your
understanding to the operator: what you take the request to mean, how it fits understanding to the operator: what you take the request to mean, how it fits
the project, your first step, and why that way rather than the main the project, and your first step. Wait for their go-ahead. If the request is
alternative. A later restatement says what changed. Wait for their go-ahead. unclear, ask_human instead of guessing: a restatement asks no questions, so
If the request is unclear, ask_human instead of guessing: a restatement asks the operator can agree to it as it stands.
no questions, so the operator can agree to it as it stands. At any point,
asking is better than guessing.
Before each group of tool calls, say in a sentence what you're doing and why:
the operator sees it as your progress.
The project's documents are in its repository: read ARCHITECTURE.md and any The project's documents are in its repository: read ARCHITECTURE.md and any
build plans under docs/plans/ before you restate. build plans under docs/plans/ before you restate.
@@ -129,8 +108,7 @@ the same milestone (don't start another), run the tests, and call
finish_milestone again to push the fix. When a review needs no change from finish_milestone again to push the fix. When a review needs no change from
you, or you are only waiting for the operator to merge (you can't merge it you, or you are only waiting for the operator to merge (you can't merge it
yourself), call wait_for_review. The next milestone starts only after the yourself), call wait_for_review. The next milestone starts only after the
previous PR has merged. A milestone's summary says what was done, what was previous PR has merged. When the last PR has merged, call close_item with
verified, and what is left open. When the last PR has merged, call close_item with
what was done, what was verified, and what is left open. If a PR is closed what was done, what was verified, and what is left open. If a PR is closed
without merging, ask the operator what next. without merging, ask the operator what next.
@@ -140,16 +118,10 @@ edge cases, and the tests that are its acceptance criteria. Commit it with the
first milestone, and mark each milestone done in it in that milestone's PR. first milestone, and mark each milestone done in it in that milestone's PR.
Keep ARCHITECTURE.md current: a design change updates it in the same PR. Keep ARCHITECTURE.md current: a design change updates it in the same PR.
{labels}{helpers}Never label the issue yourself; you may suggest a label to the operator. {labels}Never label the issue yourself; you may suggest a label to the operator.
{workspace}""" {workspace}"""
QUESTION = """
The operator has asked a question (the last message). Answer it with reply,
from what you know and what you can look at; don't change anything. The item
then goes back to waiting exactly as it was, so a restatement up stays up."""
CONVERSATION = """You are nisse-dev, a software agent, in a conversation with CONVERSATION = """You are nisse-dev, a software agent, in a conversation with
the operator in a Matrix thread. It isn't about a project yet: it may be a the operator in a Matrix thread. It isn't about a project yet: it may be a
question, thoughts on feasibility or hardware, or the start of some work. question, thoughts on feasibility or hardware, or the start of some work.
@@ -178,11 +150,9 @@ LABELS = {
"discuss": "This issue is labelled discuss: talk it through with the " "discuss": "This issue is labelled discuss: talk it through with the "
"operator (ask_human) before restating, and restate only once you agree " "operator (ask_human) before restating, and restate only once you agree "
"on what is wanted.", "on what is wanted.",
"design": "This issue is labelled design: the design comes before any " "design": "This issue is labelled design: agree the design with the "
"code. Your first milestone is documents only: ARCHITECTURE.md and a " "operator before any code. Your restatement puts the design to them; "
"build plan under docs/plans/, in a PR of their own (if you restate, put " "code waits for their go-ahead on it.",
"the design to the operator). Code starts in a later milestone, after that "
"PR has merged; the harness refuses a design milestone with anything else.",
} }
@@ -240,7 +210,6 @@ class LoopAgent:
# A new project's entry, once the operator has added it (D1, D8): # A new project's entry, once the operator has added it (D1, D8):
# the projects configured now that this agent doesn't know yet. # the projects configured now that this agent doesn't know yet.
reload: Callable[[set[str]], Sequence[Project]] | None = None, reload: Callable[[set[str]], Sequence[Project]] | None = None,
helpers: Helpers | None = None,
clock: Callable[[], float] = time.time, clock: Callable[[], float] = time.time,
intake_interval_s: float = INTAKE_INTERVAL_S, intake_interval_s: float = INTAKE_INTERVAL_S,
) -> None: ) -> None:
@@ -262,7 +231,6 @@ class LoopAgent:
self._grace = bug_grace_s self._grace = bug_grace_s
self._idle = general_idle_s self._idle = general_idle_s
self._reload = reload self._reload = reload
self._helpers = helpers
self._clock = clock self._clock = clock
self._interval = intake_interval_s self._interval = intake_interval_s
# thread root -> revision on screen there, read on the transport's # thread root -> revision on screen there, read on the transport's
@@ -274,9 +242,6 @@ class LoopAgent:
# The project whose workspace was last worked on: switching to # The project whose workspace was last worked on: switching to
# another parks it (L1). None after a boot: unknown, so all are. # another parks it (L1). None after a boot: unknown, so all are.
self._last_project: str | None = None self._last_project: str | None = None
# Per item: when progress was last posted (L5: a readable rate).
self._progressed: dict[str, float] = {}
self._active_name = "" # the running item, for "noted" on arrival
self._running = False self._running = False
self._stop = threading.Event() self._stop = threading.Event()
self._intake_thread: threading.Thread | None = None self._intake_thread: threading.Thread | None = None
@@ -351,7 +316,6 @@ class LoopAgent:
self._store.set_active(None, cause="boot") self._store.set_active(None, cause="boot")
self._reconcile_publishing() self._reconcile_publishing()
self._reconcile_filing() self._reconcile_filing()
self._reconcile_delegations()
for item in self._store.work_items("open"): for item in self._store.work_items("open"):
loop = loop_of(item) loop = loop_of(item)
self._threads[(item.scope or "", loop.issue)] = loop.thread self._threads[(item.scope or "", loop.issue)] = loop.thread
@@ -372,30 +336,6 @@ class LoopAgent:
thread=loop.thread, thread=loop.thread,
) )
def _reconcile_delegations(self) -> None:
"""L8, L9: a delegation cut off by a crash is discarded, its branch
deleted and the milestone's left as it was. The item, cut off mid-
run, then waits on the operator like any other."""
for intent in self._store.pending_intents():
if intent.kind != DELEGATION_INTENT or not intent.work_item:
continue
item = self._store.work_item(intent.work_item)
branch = str(intent.payload.get("branch", ""))
base = str(intent.payload.get("base", ""))
project = self._projects.get(item.scope or "") if item else None
milestones = project.milestones if project else None
if milestones is not None and branch and base:
with contextlib.suppress(GitError):
milestones.git.reset_to(base, base)
with contextlib.suppress(GitError):
milestones.git._git("branch", "-q", "-D", branch)
self._store.abort_intent(intent.id, cause="cut off by a restart")
if item is not None:
self._fact(
item,
f"a helper's work on {branch} was cut off by a restart and discarded",
)
def _reconcile_filing(self) -> None: def _reconcile_filing(self) -> None:
"""L8: a record issue cut off by a crash. An issue carrying the new """L8: a record issue cut off by a crash. An issue carrying the new
item's ID was opened: its item is started now. Otherwise nothing was item's ID was opened: its item is started now. Otherwise nothing was
@@ -579,9 +519,6 @@ class LoopAgent:
if pr.merged: if pr.merged:
self._fact(item, f"PR #{loop.pr} merged") self._fact(item, f"PR #{loop.pr} merged")
loop, position = replace(loop, pr=0, pr_url="", branch=""), READY loop, position = replace(loop, pr=0, pr_url="", branch=""), READY
if "design" in loop.labels and not loop.design_merged:
self._fact(item, "the design has merged: code may follow")
loop = replace(loop, design_merged=True)
elif pr.state != "open": elif pr.state != "open":
self._fact(item, f"PR #{loop.pr} was closed without merging") self._fact(item, f"PR #{loop.pr} was closed without merging")
loop, position = replace(loop, pr=0, pr_url="", branch=""), READY loop, position = replace(loop, pr=0, pr_url="", branch=""), READY
@@ -763,16 +700,6 @@ class LoopAgent:
# Straight to the flag: the run is what it has to reach (L2). # Straight to the flag: the run is what it has to reach (L2).
self._cancel.set() self._cancel.set()
about = self._shown.get(thread) if thread else None about = self._shown.get(thread) if thread else None
if self._running and command not in ("stop", "status"):
# L5: say what happens to a message that can't be read yet.
here = thread and thread == self._active_thread
self._transport.post(
"Noted; I'll read it when this turn ends."
if here
else f"Noted; I'm in the middle of {self._active_name}, and I'll "
"read it when that turn ends.",
thread=thread or msg.event_id or None,
)
self._worker.submit( self._worker.submit(
self._settled("your message", lambda: self.handle(msg, about)) self._settled("your message", lambda: self.handle(msg, about))
) )
@@ -844,30 +771,6 @@ class LoopAgent:
self._close(item, loop, "abandoned", f"abandoned by {msg.sender}") self._close(item, loop, "abandoned", f"abandoned by {msg.sender}")
elif command == "done" and loop.new_project: elif command == "done" and loop.new_project:
self._provisioned(item, loop) self._provisioned(item, loop)
elif command == "details":
self._transport.post(self._details(item), thread=loop.thread)
elif (
item.position == WAITING
and item.scope != GENERAL
and is_question(text)
and loop.wait in (GO_AHEAD, ANSWER, MERGE)
):
# L5: questions never change state. Answered, then back to this
# very wait: a restatement up stays up, a go still binds to it.
save(
self._store,
item.id,
READY,
replace(
loop,
question=True,
back_to=loop.wait,
back_note=loop.note,
wait="",
note="",
),
cause="the operator asked a question",
)
elif command == "stop": elif command == "stop":
# Stopping a run took effect on arrival; a stop is never an answer. # Stopping a run took effect on arrival; a stop is never an answer.
if not loop.note.startswith("stopped ("): if not loop.note.startswith("stopped ("):
@@ -1144,8 +1047,7 @@ class LoopAgent:
project.repo, project.repo,
title, title,
f"{brief}\n\nFrom a conversation with the operator, agreed there " f"{brief}\n\nFrom a conversation with the operator, agreed there "
f"as rev-{loop.rev}. First: ARCHITECTURE.md and the build plan, " f"as rev-{loop.rev}. First: ARCHITECTURE.md and the build plan."
"in a PR of documents only; code after it has merged."
f"\n\n<!-- nisse-item: {new_id} -->", f"\n\n<!-- nisse-item: {new_id} -->",
) )
self._gitea.assign_issue(project.repo, number, self._account) self._gitea.assign_issue(project.repo, number, self._account)
@@ -1188,30 +1090,7 @@ class LoopAgent:
for e in self._store.transcript(item.id, "driver").read() for e in self._store.transcript(item.id, "driver").read()
if isinstance(e.event, PlanSubmitted) if isinstance(e.event, PlanSubmitted)
] ]
if not proposals: return bool(proposals) and self._reader.agrees(proposals[-1], text)
return False
classifier = self._roles_of(item).resolve("classifier")
agreed = self._reader.agrees(proposals[-1], text, model=classifier)
# L9 evaluation capture: the reading, as an input/output pair.
log = self._store.transcript(item.id, "classifier")
log.append(
DelegationStarted(
brief=f"proposal:\n{proposals[-1]}\n\nreply:\n{text}",
role="classifier",
model=classifier,
caps={},
)
)
log.append(
DelegationCompleted(
handback="a go-ahead" if agreed else "not a go-ahead",
artifacts=[],
tokens=0,
duration_s=0.0,
exit_status="answered",
)
)
return agreed
def _which(self, command: str) -> str: def _which(self, command: str) -> str:
"""L5: an unnamed command at top level is ambiguous: ask.""" """L5: an unnamed command at top level is ambiguous: ask."""
@@ -1332,237 +1211,6 @@ class LoopAgent:
) )
self._last_project = here self._last_project = here
# -- help (L9) -------------------------------------------------------------
def _roles_of(self, item: WorkItem) -> RoleMap:
"""The roles pinned when the item opened, so a configuration change
never splits one item across models (L9)."""
default = (
self._helpers.roles
if self._helpers
else RoleMap(
executor="",
planner=self._roles.get("planner", ""),
helper="",
expert="",
)
)
pinned = item.data.get("roles") or self._roles
return RoleMap.from_pinned(dict(pinned), default)
def _delegate(
self, item_id: str, task: str, *, trigger: str = ""
) -> tuple[str, bool]:
"""A helper takes a sub-task on a branch off the milestone's: merged
back on a handback, discarded otherwise."""
item = self._store.work_item(item_id)
assert item is not None
loop = loop_of(item)
milestones = self._at(item).milestones
workspace = self._at(item).workspace
if self._helpers is None or milestones is None or workspace is None:
return "there's no helper here", False
if not loop.branch:
return "start a milestone first: a helper works on its branch", False
if not task:
return "say what the sub-task is", False
roles = self._roles_of(item)
number = loop.helps + 1
branch = f"{loop.branch}-help-{number}"
save(
self._store,
item.id,
item.position,
replace(loop, helps=number),
cause=f"helper {number} called in",
)
helper = roles.resolve("helper")
delegation = Delegation(
self._helpers.delegate_client(helper),
self._store,
workspace,
milestones.git,
model=helper,
caps=self._helpers.caps,
cancel=self._cancel,
reader=self._helpers.reader(roles),
on_turn=lambda said: self._progress(
f"{item.id}:helper", loop.thread, said, prefix="helper: "
),
)
git = milestones.git
before = git.head_sha() # where the milestone is, uncommitted work aside
brief = (
f'{task}\n\nThe work item: issue #{loop.issue}, "{item.title}", '
f"milestone {loop.milestone}." + (f" {trigger}" if trigger else "")
)
result = delegation.run(
self._store.transcript(item.id, "helper"), brief=brief, branch=branch
)
try:
# The milestone takes the helper's work (or, on failure, only
# its own back) as uncommitted changes: harness branches never
# hold commits only this clone has (publishing makes them), so
# parking, a later milestone and a follow-up PR all carry it.
git.reset_to(loop.branch, branch if result.usable else loop.branch)
git._git("reset", "-q", "--soft", before)
if branch in git.local_branches():
git._git("branch", "-q", "-D", branch)
except GitError as exc:
self._fact(item, f"after helper {number}: {exc}")
for intent in self._store.pending_intents():
# Cut off rather than finished: discarded all the same, never
# resumed blindly (L8, L9).
if (
intent.kind == DELEGATION_INTENT
and intent.payload.get("branch") == branch
):
self._store.abort_intent(intent.id, cause=f"delegation {result.status}")
if result.usable and result.handback is not None:
self._fact(item, f"helper {number} finished: {result.handback.render()}")
return f"the helper finished:\n{result.handback.render()}", True
self._fact(
item, f"helper {number} didn't finish ({result.status}): {result.note}"
)
said = (
f"the helper didn't finish ({result.status}: {result.note}); its work "
"was discarded and the milestone is as it was"
)
return said, False
def _ask_expert(self, item_id: str, question: str) -> tuple[str, bool]:
item = self._store.work_item(item_id)
assert item is not None
if self._helpers is None:
return "there's no expert here", False
if not question:
return "ask a question", False
desk = ExpertDesk(
self._helpers.client,
self._roles_of(item),
budget=self._helpers.expert_budget,
)
try:
answer = desk.ask(self._store.transcript(item.id, "expert"), question)
except BudgetExhausted as exc:
return str(exc), False
except LLMError as exc:
return f"the expert couldn't answer: {exc}", False
return answer.render(), True
def _help_when_stuck(self, item: WorkItem, loop: Loop, outcome: RunOutcome) -> bool:
"""L9's observable triggers: the driver stuck (commands failing in a
row) or out of turns mid-milestone. A helper is called in, once per
milestone; True when it finished and the driver goes on."""
if (
self._helpers is None
or outcome.kind not in ("stuck", "capped")
or not loop.branch
or loop.helped_at == loop.milestone
):
return False
save(
self._store,
item.id,
RUNNING,
replace(loop, helped_at=loop.milestone),
cause="helper called in",
)
self._transport.post(
f"I got stuck ({outcome.note[:200]}), so a helper is taking over the "
f"rest of milestone {loop.milestone}.",
thread=loop.thread,
)
self._running = True
try:
_, ok = self._delegate(
item.id,
f"Finish milestone {loop.milestone}, so its tests pass.",
trigger=f"The driver got stuck: {outcome.note}",
)
finally:
self._running = False
self._cancel.clear()
if not ok:
return False
item = self._store.work_item(item.id) or item
save(self._store, item.id, READY, loop_of(item), cause="a helper finished")
return True
def _changed_problem(self, item_id: str, args: dict[str, object]) -> str | None:
"""L5: a revision says what changed from the one before."""
item = self._store.work_item(item_id)
if item is None or loop_of(item).rev == 0:
return None
if str(args.get("changed", "")).strip():
return None
return (
f"not posted: this is a revision of rev-{loop_of(item).rev}, so say "
"in changed what's different and why"
)
def _progress(self, item_id: str, thread: str, said: str, prefix: str = "") -> None:
"""A turn's own words, posted at a readable rate (L5, #42): at most
one a minute per item, the latest when the minute is up."""
now = self._clock()
if now - self._progressed.get(item_id, float("-inf")) < PROGRESS_EVERY_S:
return
self._progressed[item_id] = now
text = said if len(said) <= PROGRESS_CHARS else said[:PROGRESS_CHARS] + " …"
self._transport.post(f"{prefix}{text}", thread=thread)
def _details(self, item: WorkItem) -> str:
"""L5: raw tool activity, on request: the item's last tool calls."""
events = self._store.transcript(item.id, "driver").read()
results = {
e.event.call_id: e.event for e in events if isinstance(e.event, ToolResult)
}
lines = []
for e in events:
if not isinstance(e.event, ToolCall):
continue
call, result = e.event, results.get(e.event.call_id)
target = call.arguments.get("path") or call.arguments.get("command") or ""
outcome = (
"no result yet"
if result is None
else ("ok" if result.success else "failed")
+ (f": {result.result.strip()[:120]}" if result.result.strip() else "")
)
who = "" if e.actor == "driver" else f"{e.actor}: "
lines.append(f"- {who}{call.tool}({str(target)[:80]}) → {outcome}")
if not lines:
return f"No tool calls yet on {self._label(item)}."
return f"The last tool calls on {self._label(item)}:\n" + "\n".join(
lines[-DETAILS:]
)
def _design_problem(self, item_id: str) -> str | None:
"""D1, D2: a `design` item agrees its design before any code. Until
a PR of it has merged, a milestone holds documents only."""
item = self._store.work_item(item_id)
if item is None:
return None
loop = loop_of(item)
milestones = self._at(item).milestones
if "design" not in loop.labels or loop.design_merged or milestones is None:
return None
try:
changed = milestones.changed()
except GitError:
return None # publishing reports what's wrong with the workspace
code = [
p for p in changed if p != "ARCHITECTURE.md" and not p.startswith("docs/")
]
if not code:
return None
return (
"not published: this milestone is the design, so it holds documents "
"only (ARCHITECTURE.md, docs/); code starts in a later milestone, "
"after this one's PR has merged. Take these out of it (they can "
"wait in the plan): " + ", ".join(code)
)
def _converse_run(self, item: WorkItem) -> None: def _converse_run(self, item: WorkItem) -> None:
"""A conversation's turn (L1, L5): it talks, and may propose work on """A conversation's turn (L1, L5): it talks, and may propose work on
a project. No workspace, nothing to run.""" a project. No workspace, nothing to run."""
@@ -1649,8 +1297,6 @@ class LoopAgent:
save(self._store, item.id, RUNNING, loop, cause="driver run") save(self._store, item.id, RUNNING, loop, cause="driver run")
# D1: a `discuss` item talks before it restates. # D1: a `discuss` item talks before it restates.
hold = "discuss" in loop.labels and not loop.talked and not loop.started hold = "discuss" in loop.labels and not loop.talked and not loop.started
if loop.question:
hold = True # nothing but answering (below)
tools = [ tools = [
*([] if hold else [RESTATE]), *([] if hold else [RESTATE]),
ASK_HUMAN, ASK_HUMAN,
@@ -1662,29 +1308,19 @@ class LoopAgent:
FINISH_MILESTONE, FINISH_MILESTONE,
CLOSE_ITEM, CLOSE_ITEM,
*([WAIT_FOR_REVIEW] if loop.pr else []), *([WAIT_FOR_REVIEW] if loop.pr else []),
*([DELEGATE, ASK_EXPERT] if self._helpers else []),
*EXECUTOR_TOOLS, *EXECUTOR_TOOLS,
] ]
), ),
] ]
if loop.question:
tools = [REPLY, *_LOOK]
roles = self._roles_of(item)
workspace = project.workspace workspace = project.workspace
system = SYSTEM.format( system = SYSTEM.format(
repo=project.repo, repo=project.repo,
issue=loop.issue, issue=loop.issue,
title=item.title, title=item.title,
labels="".join(f"{LABELS[x]}\n\n" for x in loop.labels if x in LABELS), labels="".join(f"{LABELS[x]}\n\n" for x in loop.labels if x in LABELS),
helpers=(
f"{self._helpers.briefing(roles)}\n\n"
if self._helpers and loop.started
else ""
),
workspace=workspace.describe() if workspace else "", workspace=workspace.describe() if workspace else "",
) + (QUESTION if loop.question else "") )
self._active_thread = loop.thread self._active_thread = loop.thread
self._active_name = self._label(item)
self._running = True self._running = True
try: try:
outcome = self._driver.run( outcome = self._driver.run(
@@ -1692,24 +1328,8 @@ class LoopAgent:
system=system, system=system,
tools=tools, tools=tools,
workspace=workspace, workspace=workspace,
handlers={ handlers={"start_milestone": lambda args: self._start(item.id, args)},
"start_milestone": lambda args: self._start(item.id, args), checks={"restate": _restatement_problem},
"delegate": lambda args: self._delegate(
item.id, str(args.get("task", "")).strip()
),
"ask_expert": lambda args: self._ask_expert(
item.id, str(args.get("question", "")).strip()
),
},
checks={
"restate": lambda args: (
_restatement_problem(args)
or self._changed_problem(item.id, args)
),
"finish_milestone": lambda args: self._design_problem(item.id),
},
model=roles.resolve("driver"),
on_turn=lambda said: self._progress(item.id, loop.thread, said),
) )
finally: finally:
self._running = False self._running = False
@@ -1721,31 +1341,6 @@ class LoopAgent:
assert item is not None assert item is not None
loop = loop_of(item) loop = loop_of(item)
args = outcome.arguments or {} args = outcome.arguments or {}
if loop.question:
# Back to the wait the question interrupted, as it was (L5).
if outcome.kind == "wait" and outcome.tool == "reply":
self._transport.post(
str(args.get("text", "")).strip(), thread=loop.thread
)
else:
self._transport.post(
f"I couldn't answer that ({outcome.note or outcome.kind}).",
thread=loop.thread,
)
self._wait(
item,
replace(
loop,
wait=loop.back_to,
note=loop.back_note,
question=False,
back_to="",
back_note="",
),
None,
)
self._store.set_active(None, cause="question answered")
return
if outcome.kind == "wait" and outcome.tool == "restate": if outcome.kind == "wait" and outcome.tool == "restate":
text = str(args.get("text", "")).strip() text = str(args.get("text", "")).strip()
rev = loop.rev + 1 rev = loop.rev + 1
@@ -1761,11 +1356,7 @@ class LoopAgent:
f"It's a bug, so I'll go ahead in {round(self._grace / 60)} min " f"It's a bug, so I'll go ahead in {round(self._grace / 60)} min "
"unless you reply; say go to start now, or reply with changes." "unless you reply; say go to start now, or reply with changes."
) )
changed = str(args.get("changed", "")).strip() self._transport.post(f"rev-{rev}: {text}\n\n{after}", thread=loop.thread)
since = f"\n\nChanged from rev-{rev - 1}: {changed}" if rev > 1 else ""
self._transport.post(
f"rev-{rev}: {text}{since}\n\n{after}", thread=loop.thread
)
self._wait( self._wait(
item, item,
replace(loop, rev=rev, wait=GO_AHEAD, note="", grace_until=grace), replace(loop, rev=rev, wait=GO_AHEAD, note="", grace_until=grace),
@@ -1879,8 +1470,6 @@ class LoopAgent:
elif outcome.kind == "stopped": elif outcome.kind == "stopped":
note = f"stopped ({outcome.note}). Why did you stop it? Reply to continue." note = f"stopped ({outcome.note}). Why did you stop it? Reply to continue."
self._wait(item, replace(loop, wait=ANSWER, note=note), note) self._wait(item, replace(loop, wait=ANSWER, note=note), note)
elif self._help_when_stuck(item, loop, outcome):
pass # ready again: the driver goes on from the helper's handback
else: else:
note = f"I stopped short: {outcome.note}. Reply to continue." note = f"I stopped short: {outcome.note}. Reply to continue."
self._wait(item, replace(loop, wait=ANSWER, note=note), note) self._wait(item, replace(loop, wait=ANSWER, note=note), note)
+6 -23
View File
@@ -15,14 +15,11 @@ from nisse_dev.agent import (
remote_url, remote_url,
) )
from nisse_dev.cancel import Cancel from nisse_dev.cancel import Cancel
from nisse_dev.catalog import select_client
from nisse_dev.config import AgentConfig, ConfigError from nisse_dev.config import AgentConfig, ConfigError
from nisse_dev.config import Project as ProjectEntry from nisse_dev.config import Project as ProjectEntry
from nisse_dev.delegation import Caps
from nisse_dev.gitea import GiteaClient from nisse_dev.gitea import GiteaClient
from nisse_dev.loop.agent import LoopAgent, Project from nisse_dev.loop.agent import LoopAgent, Project
from nisse_dev.loop.driver import Driver from nisse_dev.loop.driver import Driver
from nisse_dev.loop.help import Helpers
from nisse_dev.loop.milestones import Milestones from nisse_dev.loop.milestones import Milestones
from nisse_dev.loop.reader import GoAheadReader from nisse_dev.loop.reader import GoAheadReader
from nisse_dev.matrix import MatrixTransport from nisse_dev.matrix import MatrixTransport
@@ -127,28 +124,14 @@ def build_loop_agent(
project_runtime(config, entry, store, gitea, account) project_runtime(config, entry, store, gitea, account)
for entry in config.projects for entry in config.projects
] ]
normalized, passthrough = make_llm_clients(config, catalog) normalized, _passthrough = make_llm_clients(config, catalog)
cancel = Cancel() cancel = Cancel()
# The configured driver; each work item runs on the one it pinned (L9). # The driver is the planner's model for now (M2.6 brings the driver role).
driver = Driver( driver = Driver(
normalized, normalized,
config.roles.resolve("driver"), config.roles.resolve("planner"),
max_turns=config.tuning.executor.max_turns, max_turns=config.tuning.executor.max_turns,
cancel=cancel, cancel=cancel,
failure_limit=config.tuning.executor.failure_limit,
)
tuning = config.tuning
helpers = Helpers(
client=normalized,
roles=config.roles,
client_for=lambda model: select_client(model, normalized, passthrough, catalog),
caps=Caps(
turns=tuning.delegation.max_turns,
tokens=tuning.delegation.max_tokens,
wall_clock_s=tuning.delegation.wall_clock_s,
),
expert_budget=tuning.expert.consult_budget,
profiles=config.profiles,
) )
transport = MatrixTransport(config.matrix) transport = MatrixTransport(config.matrix)
loop = LoopAgent( loop = LoopAgent(
@@ -164,9 +147,9 @@ def build_loop_agent(
cancel=cancel, cancel=cancel,
oplog=oplog, oplog=oplog,
roles=config.roles.pinned(), roles=config.roles.pinned(),
# The classifier role reads plain-language go-aheads (L5, L9). # The driver's model reads plain-language go-aheads until the
reader=GoAheadReader(normalized, config.roles.resolve("classifier")), # classifier role arrives (M2.6).
helpers=helpers, reader=GoAheadReader(normalized, config.roles.resolve("planner")),
reload=lambda known: [ reload=lambda known: [
project_runtime(config, entry, store, gitea, account) project_runtime(config, entry, store, gitea, account)
for entry in (load or _from_file(config))().projects for entry in (load or _from_file(config))().projects
+3 -33
View File
@@ -12,7 +12,7 @@ from dataclasses import dataclass
from typing import Any, Literal from typing import Any, Literal
from nisse_dev.cancel import Cancel, skip_calls from nisse_dev.cancel import Cancel, skip_calls
from nisse_dev.canonical import AssistantText, ToolCall, ToolResult from nisse_dev.canonical import ToolCall, ToolResult
from nisse_dev.llm import LLMClient, LLMError from nisse_dev.llm import LLMClient, LLMError
from nisse_dev.loop.tools import WAITING, names from nisse_dev.loop.tools import WAITING, names
from nisse_dev.state import Transcript from nisse_dev.state import Transcript
@@ -29,8 +29,7 @@ _PROSE_NUDGE = (
@dataclass(frozen=True) @dataclass(frozen=True)
class RunOutcome: class RunOutcome:
# stuck: failed commands in a row hit the failure limit (L9's trigger) kind: Literal["wait", "capped", "stopped", "failed"]
kind: Literal["wait", "capped", "stuck", "stopped", "failed"]
tool: str = "" # the waiting tool called tool: str = "" # the waiting tool called
arguments: dict[str, Any] | None = None arguments: dict[str, Any] | None = None
note: str = "" note: str = ""
@@ -44,12 +43,10 @@ class Driver:
*, *,
max_turns: int = DEFAULT_MAX_TURNS, max_turns: int = DEFAULT_MAX_TURNS,
cancel: Cancel | None = None, cancel: Cancel | None = None,
failure_limit: int = 0, # 0: no limit
) -> None: ) -> None:
self._client = client self._client = client
self._model = model self._model = model
self._max_turns = max_turns self._max_turns = max_turns
self._failure_limit = failure_limit
self._cancel = cancel or Cancel() self._cancel = cancel or Cancel()
def run( def run(
@@ -62,8 +59,6 @@ class Driver:
handlers: Mapping[str, Callable[[dict[str, Any]], tuple[str, bool]]] handlers: Mapping[str, Callable[[dict[str, Any]], tuple[str, bool]]]
| None = None, | None = None,
checks: Mapping[str, Callable[[dict[str, Any]], str | None]] | None = None, checks: Mapping[str, Callable[[dict[str, Any]], str | None]] | None = None,
model: str | None = None,
on_turn: Callable[[str], None] | None = None,
) -> RunOutcome: ) -> RunOutcome:
"""``handlers`` answer control tools that don't end the run (e.g. """``handlers`` answer control tools that don't end the run (e.g.
start_milestone): each returns a result and whether it succeeded. start_milestone): each returns a result and whether it succeeded.
@@ -71,8 +66,6 @@ class Driver:
failed result and the run goes on, so the model puts it right.""" failed result and the run goes on, so the model puts it right."""
handlers = handlers or {} handlers = handlers or {}
checks = checks or {} checks = checks or {}
model = model or self._model # the work item's pinned driver (L9)
failures: list[str] = [] # the failed commands in a row, latest last
offered = names(tools) offered = names(tools)
nudged = False nudged = False
for turn in range(self._max_turns): for turn in range(self._max_turns):
@@ -81,7 +74,7 @@ class Driver:
return RunOutcome("stopped", note=f"before turn {turn + 1}") return RunOutcome("stopped", note=f"before turn {turn + 1}")
try: try:
produced = self._client.complete( produced = self._client.complete(
transcript.view(), tools=tools, model=model, system=system transcript.view(), tools=tools, model=self._model, system=system
) )
except LLMError as exc: except LLMError as exc:
return RunOutcome("failed", note=f"the driver's call failed: {exc}") return RunOutcome("failed", note=f"the driver's call failed: {exc}")
@@ -90,12 +83,6 @@ class Driver:
transcript.append(event) transcript.append(event)
if isinstance(event, ToolCall): if isinstance(event, ToolCall):
calls.append(event) calls.append(event)
# The model's own words for the turn: its progress, in prose (#42).
said = " ".join(
e.text.strip() for e in produced if isinstance(e, AssistantText)
).strip()
if said and calls and on_turn is not None:
on_turn(said)
if not calls: if not calls:
if nudged: if nudged:
return RunOutcome("failed", note="kept replying in prose") return RunOutcome("failed", note="kept replying in prose")
@@ -145,21 +132,4 @@ class Driver:
transcript.append( transcript.append(
ToolResult(call_id=call.call_id, result=result, success=ok) ToolResult(call_id=call.call_id, result=result, success=ok)
) )
if call.tool == "run_command":
failures = [] if ok else [*failures, _brief(call, result)]
if self._failure_limit and len(failures) >= self._failure_limit:
skip_calls(
transcript, calls[index + 1 :], "not run: you're stuck"
)
return RunOutcome(
"stuck",
note=f"{len(failures)} commands failed in a row: "
+ "; ".join(failures),
)
return RunOutcome("capped", note=f"hit the {self._max_turns}-turn cap") return RunOutcome("capped", note=f"hit the {self._max_turns}-turn cap")
def _brief(call: ToolCall, result: str) -> str:
"""A failed command and the end of what it said, for a helper's brief."""
command = str(call.arguments.get("command", ""))[:120]
return f"`{command}`: {result.strip()[-300:]}"
-73
View File
@@ -1,73 +0,0 @@
"""Help for the driver (L9): a helper for a sub-task (delegation), an expert
for one question, both by role, briefed from capability profiles.
The driver asks when it chooses to (its instructions treat asking as
normal), and the harness asks for it on observable triggers: commands failing
in a row, or the turn cap. Delegation runs on a branch off the milestone's,
merged back on a handback and discarded otherwise (L9: never build on a
half-edited tree).
"""
from collections.abc import Callable, Mapping
from dataclasses import dataclass, field
from nisse_dev.config import Profile
from nisse_dev.delegation import Caps
from nisse_dev.expert import DEFAULT_BUDGET
from nisse_dev.llm import LLMClient, RoleMap
from nisse_dev.loop.tools import _tool
DELEGATE = _tool(
"delegate",
"Hand a sub-task of the current milestone to a helper, who works on the "
"milestone's branch with the same tools and reports back what changed, "
"what is now true and what next. Use it when the sub-task is beyond what "
"you can do reliably, or you are going round in circles. Needs a "
"milestone started.",
task="The sub-task, self-contained: what to do, where, and how to know it's done.",
)
ASK_EXPERT = _tool(
"ask_expert",
"Ask an expert one self-contained question: a design trade-off, how a "
"library behaves. It can't see the repository. The budget per item is "
"small.",
question="The question, with whatever context it needs.",
)
@dataclass(frozen=True)
class Helpers:
client: LLMClient # for the expert
roles: RoleMap # the configured mapping; each item pins its own (L9)
# The helper's client by model (vendor passthrough where it fits, L9);
# unset, ``client``.
client_for: Callable[[str], LLMClient] | None = None
caps: Caps = field(default_factory=Caps)
expert_budget: int = DEFAULT_BUDGET
profiles: Mapping[str, Profile] = field(default_factory=dict)
def delegate_client(self, model: str) -> LLMClient:
return self.client_for(model) if self.client_for else self.client
def profile(self, model: str) -> str:
"""What one role is told about another: its capability profile,
never its name (L9)."""
found = self.profiles.get(model)
return found.describe() if found else "no profile recorded"
def briefing(self, roles: RoleMap) -> str:
"""The driver's instructions on asking for help."""
return (
"Asking for help is normal, not a failure. delegate hands a "
"sub-task to a helper ("
+ self.profile(roles.resolve("helper"))
+ "); ask_expert asks one question of an expert ("
+ self.profile(roles.resolve("expert"))
+ "). If commands keep failing, a helper is called in for you."
)
def reader(self, roles: RoleMap) -> str:
"""Who acts on a helper's handback: the driver, by its profile."""
return "the driver, a model with this profile: " + self.profile(
roles.resolve("driver")
)
-8
View File
@@ -55,14 +55,6 @@ class Loop:
proposal_title: str = "" proposal_title: str = ""
proposal_kind: str = "" # "work" on a project, or a new "project" proposal_kind: str = "" # "work" on a project, or a new "project"
new_project: str = "" # a new project being provisioned (D1) new_project: str = "" # a new project being provisioned (D1)
design_merged: bool = False # a `design` item's design PR has merged (D1, D2)
helps: int = 0 # delegations so far, for their branch names (L9)
helped_at: int = 0 # the milestone a helper was last called in for, unasked
# A question from the operator is answered without changing the item
# (L5): the wait it interrupted, to go back to.
question: bool = False
back_to: str = ""
back_note: str = ""
# Where the operator last spoke (D5: answer where asked): "" for the # Where the operator last spoke (D5: answer where asked): "" for the
# Matrix thread, or the PR it came from, with the inline location and a # Matrix thread, or the PR it came from, with the inline location and a
# little of what they said, for quoting. # little of what they said, for quoting.
@@ -161,15 +161,6 @@ class Milestones:
if git.remote_sha(self._remote, parked, credential=self._credential): if git.remote_sha(self._remote, parked, credential=self._credential):
git.delete_remote_branch(self._remote, parked, credential=self._credential) git.delete_remote_branch(self._remote, parked, credential=self._credential)
@property
def git(self) -> GitRepo:
"""The clone, for a delegation's branch (L9)."""
return self._git
def changed(self) -> list[str]:
"""What the current milestone changes, from the default branch."""
return self._git.changed_paths(f"origin/{self._base}")
# -- a new project's checks (D1, D4) ----------------------------------------- # -- a new project's checks (D1, D4) -----------------------------------------
def verify_access(self, account: str) -> list[str]: def verify_access(self, account: str) -> list[str]:
+2 -3
View File
@@ -50,8 +50,7 @@ class GoAheadReader:
self._client = client self._client = client
self._model = model self._model = model
def agrees(self, proposal: str, reply: str, model: str = "") -> bool: def agrees(self, proposal: str, reply: str) -> bool:
"""``model``: the work item's pinned classifier (L9)."""
if is_question(reply): if is_question(reply):
return False return False
prompt = f"The proposal:\n\n{proposal}\n\nThe operator's reply:\n\n{reply}" prompt = f"The proposal:\n\n{proposal}\n\nThe operator's reply:\n\n{reply}"
@@ -66,7 +65,7 @@ class GoAheadReader:
) )
], ],
tools=[VERDICT], tools=[VERDICT],
model=model or self._model, model=self._model,
system=SYSTEM, system=SYSTEM,
) )
except LLMError: except LLMError:
+2 -5
View File
@@ -25,12 +25,9 @@ def _tool(tool: str, description: str, /, **params: str) -> dict[str, Any]:
RESTATE = _tool( RESTATE = _tool(
"restate", "restate",
"Put your understanding of the request to the operator, in their item's " "Put your understanding of the request to the operator, in their item's "
"thread: what you take it to mean, how it fits the project, your first " "thread: what you take it to mean, how it fits the project, and your "
"step, and why that way (and the main alternative you considered). Ends " "first step. Ends your turn; work starts when they agree.",
"your turn; work starts when they agree.",
text="Your restatement, in plain language.", text="Your restatement, in plain language.",
changed="What changed from your previous restatement, and why; leave it "
"empty for the first.",
) )
ASK_HUMAN = _tool( ASK_HUMAN = _tool(
"ask_human", "ask_human",
-3
View File
@@ -143,9 +143,6 @@ class FakeTransport:
def build_offline(tmp_path: Path, origin: Path, monkeypatch: Any) -> Any: def build_offline(tmp_path: Path, origin: Path, monkeypatch: Any) -> Any:
"""Assemble the real graph with only the network boundaries faked.""" """Assemble the real graph with only the network boundaries faked."""
config = config_for(tmp_path) config = config_for(tmp_path)
# The proxy here doesn't exist: a call to it fails at once rather than
# waiting out an outage's backoff (L5).
monkeypatch.setattr("nisse_dev.llm.OUTAGE_BACKOFF_S", ())
monkeypatch.setattr( monkeypatch.setattr(
"nisse_dev.agent.GiteaClient.whoami", lambda self: "nisse-dev-testing" "nisse_dev.agent.GiteaClient.whoami", lambda self: "nisse-dev-testing"
-26
View File
@@ -478,29 +478,3 @@ def test_a_workspace_of_any_project_inside_is_refused(tmp_path: Path) -> None:
state_db=str(work / "state.db"), state_db=str(work / "state.db"),
projects__other={**SECOND, "workspace": str(work)}, projects__other={**SECOND, "workspace": str(work)},
) )
# --- roles and profiles (L9) -------------------------------------------------
def test_the_new_loops_roles_fall_back() -> None:
config = build()
assert config.roles.resolve("driver") == "claude-opus-5" # the planner's
assert config.roles.resolve("classifier") == "claude-haiku-4-5" # the expert's
named = build(roles__driver="qwen3.8-27b", roles__classifier="gemma-4-12b")
assert named.roles.resolve("driver") == "qwen3.8-27b"
assert named.roles.pinned()["classifier"] == "gemma-4-12b"
def test_capability_profiles_are_read() -> None:
config = build(
profiles={"qwen3.8-27b": {"tools": "reliable", "weaknesses": "speed"}}
)
profile = config.profiles["qwen3.8-27b"]
assert profile.describe() == "tool calls: reliable; weak at speed"
assert build().profiles == {}
def test_a_bad_profile_is_reported() -> None:
with pytest.raises(ConfigError, match="profiles.m.colour: unknown setting"):
build(profiles={"m": {"colour": "red"}})
-83
View File
@@ -14,7 +14,6 @@ from nisse_dev.canonical import (
from nisse_dev.envfile import load_env_file from nisse_dev.envfile import load_env_file
from nisse_dev.llm import ( from nisse_dev.llm import (
DEFAULT_MAX_OUTPUT_TOKENS, DEFAULT_MAX_OUTPUT_TOKENS,
OUTAGE_BACKOFF_S,
RATE_LIMIT_RETRY_S, RATE_LIMIT_RETRY_S,
AnthropicPassthroughClient, AnthropicPassthroughClient,
BudgetExhausted, BudgetExhausted,
@@ -198,10 +197,6 @@ def test_role_map_resolves_and_pins() -> None:
"planner": "anthropic/claude-opus-5", "planner": "anthropic/claude-opus-5",
"helper": "anthropic/claude-opus-5", "helper": "anthropic/claude-opus-5",
"expert": "anthropic/claude-opus-5", "expert": "anthropic/claude-opus-5",
# Unset, the new loop's roles fall back: driver to the planner's
# model, classifier to the expert's (L9).
"driver": "anthropic/claude-opus-5",
"classifier": "anthropic/claude-opus-5",
} }
@@ -433,81 +428,3 @@ def test_the_answering_model_is_recorded() -> None:
) )
client.complete(EVENTS, tools=None, model="local-fast") client.complete(EVENTS, tools=None, model="local-fast")
assert client.last_model == "gpt-test-1" assert client.last_model == "gpt-test-1"
# --- outages delay, never fail (L5) --------------------------------------------
def test_a_gateway_error_is_waited_out_with_backoff() -> None:
responses = [
httpx.Response(502, text="bad gateway"),
httpx.Response(503, text="unavailable"),
completion({"content": "back"}),
]
waited: list[float] = []
client = make_client(lambda request: responses.pop(0), sleep=waited.append)
[event] = client.complete(EVENTS, tools=None, model="local/qwen")
assert isinstance(event, AssistantText) and event.text == "back"
assert waited == list(OUTAGE_BACKOFF_S[:2])
# Waiting isn't the model's time.
assert client.last_rate_limit_wait_s == sum(OUTAGE_BACKOFF_S[:2])
def test_an_outage_that_outlasts_the_backoff_is_surfaced() -> None:
attempts: list[int] = []
waited: list[float] = []
def handler(request: httpx.Request) -> httpx.Response:
attempts.append(1)
return httpx.Response(502, text="bad gateway")
client = make_client(handler, sleep=waited.append)
with pytest.raises(UpstreamUnavailable, match="502"):
client.complete(EVENTS, tools=None, model="local/qwen")
assert len(attempts) == len(OUTAGE_BACKOFF_S) + 1
assert waited == list(OUTAGE_BACKOFF_S)
def test_a_dropped_connection_is_retried() -> None:
calls: list[int] = []
def handler(request: httpx.Request) -> httpx.Response:
calls.append(1)
if len(calls) == 1:
raise httpx.ConnectError("refused")
return completion({"content": "ok"})
[event] = make_client(handler).complete(EVENTS, tools=None, model="local/qwen")
assert isinstance(event, AssistantText) and len(calls) == 2
def test_a_500_or_a_timeout_is_not_retried() -> None:
# A 500 may be the request's fault; a timeout already had its full
# request_timeout_s.
for failure in (
lambda request: httpx.Response(500, text="boom"),
lambda request: (_ for _ in ()).throw(httpx.ReadTimeout("slow")),
):
calls: list[int] = []
def handler(
request: httpx.Request, failure=failure, calls=calls
) -> httpx.Response:
calls.append(1)
return failure(request)
with pytest.raises(LLMError):
make_client(handler).complete(EVENTS, tools=None, model="local/qwen")
assert len(calls) == 1
def test_a_budget_refusal_is_not_waited_on() -> None:
calls: list[int] = []
def handler(request: httpx.Request) -> httpx.Response:
calls.append(1)
return httpx.Response(503, text="Budget has been exceeded! Current cost: 5.1")
with pytest.raises(BudgetExhausted):
make_client(handler).complete(EVENTS, tools=None, model="local/qwen")
assert len(calls) == 1
+1 -3
View File
@@ -214,9 +214,7 @@ def test_go_starts_the_work_and_close_item_closes_the_issue(h: Harness) -> None:
def test_a_go_for_an_earlier_revision_is_not_applied(h: Harness) -> None: def test_a_go_for_an_earlier_revision_is_not_applied(h: Harness) -> None:
h.model.turns.append( h.model.turns.append([call("restate", text="revised: in a new module")])
[call("restate", text="revised: in a new module", changed="a new module")]
)
h.agent.intake() h.agent.intake()
h.say("put it in its own module") # changes, so the driver restates rev-2 h.say("put it in its own module") # changes, so the driver restates rev-2
assert loop_of(h.item()).rev == 2 assert loop_of(h.item()).rev == 2
+1 -1
View File
@@ -170,7 +170,7 @@ def test_a_plain_yes_files_the_work(world: dict[str, Any]) -> None:
quiet(world) quiet(world)
class Reader: class Reader:
def agrees(self, proposal: str, reply: str, model: str = "") -> bool: def agrees(self, proposal: str, reply: str) -> bool:
return reply == "sounds right" return reply == "sounds right"
model = ScriptedModel([PROPOSAL], [call("ask_human", question="anything else?")]) model = ScriptedModel([PROPOSAL], [call("ask_human", question="anything else?")])
-326
View File
@@ -1,326 +0,0 @@
"""Help for the driver on the new loop (M2.6, L9): roles pinned per item,
delegation on a branch off the milestone's (merged back on a handback,
discarded otherwise), the stuck trigger, ask_expert and its budget, and
briefings from capability profiles. Real git on the throwaway build host."""
import json
from typing import Any
from test_loop_milestones import (
RESTATE,
Forge,
ScriptedModel,
call,
git,
item,
loop_agent,
say,
)
from test_loop_milestones import world as world # noqa: PLC0414 -- the fixture
from nisse_dev.canonical import AssistantText, Fact, ToolCall, ToolResult
from nisse_dev.config import Profile
from nisse_dev.delegation import DELEGATION_INTENT, Caps
from nisse_dev.llm import RoleMap
from nisse_dev.loop.help import Helpers
from nisse_dev.loop.items import ANSWER, MERGE, loop_of
ROLES = RoleMap(executor="", planner="drv", helper="hlp", expert="exp", driver="drv")
class Spy:
"""A scripted model that records what each call was told."""
def __init__(self, *turns: list[Any]) -> None:
self.turns = list(turns)
self.calls: list[tuple[str, str]] = [] # (model, system)
def complete(self, events, tools, model, system=None) -> list[Any]:
self.calls.append((model, system or ""))
assert self.turns, f"{model} was called more often than scripted"
return self.turns.pop(0)
def hand_back(succeeded: bool = True) -> ToolCall:
return ToolCall(
call_id=f"hb-{succeeded}",
tool="hand_back",
arguments={
"changed": "wrote helped.py",
"now_true": "the helper's part is done",
"next": "finish the milestone",
"succeeded": succeeded,
},
)
def helpers(helper: Spy, expert: Spy | None = None, **kw: Any) -> Helpers:
return Helpers(
client=expert or Spy(), # type: ignore[arg-type]
roles=ROLES,
client_for=lambda model: helper, # type: ignore[arg-type,return-value]
caps=Caps(turns=5),
profiles={
"hlp": Profile(tools="reliable", strengths="tricky tests"),
"drv": Profile(tools="fair", weaknesses="long refactors"),
},
**kw,
)
def driver_results(world: dict[str, Any], tool: str) -> list[str]:
events = world["store"].transcript(item(world).id, "driver").read()
ids = {
e.event.call_id
for e in events
if isinstance(e.event, ToolCall) and e.event.tool == tool
}
return [
e.event.result
for e in events
if isinstance(e.event, ToolResult) and e.event.call_id in ids
]
def local_branches(world: dict[str, Any]) -> list[str]:
return world["repo"].local_branches()
def test_a_helpers_work_lands_on_the_milestone(world: dict[str, Any]) -> None:
forge: Forge = world["forge"]
helper = Spy(
[call("write_file", path="helped.py", content="H = 1\n")],
[hand_back()],
)
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="words"),
call("delegate", task="write helped.py with H = 1"),
call("finish_milestone", summary="with help", final="yes"),
],
)
agent, transport = loop_agent(world, model, helpers=helpers(helper))
agent.intake()
say(transport, world, "go")
[result] = driver_results(world, "delegate")
assert result.startswith("the helper finished:") and "helped.py" in result
assert helper.calls[0][0] == "hlp" # the helper role's model
# Briefed about the driver by its profile, never by name.
assert "tool calls: fair; weak at long refactors" in helper.calls[0][1]
[pr] = forge.prs
assert forge.files(pr) == ["helped.py"]
assert not [b for b in local_branches(world) if "-help-" in b]
assert loop_of(item(world)).wait == MERGE
def test_a_failed_helpers_work_is_discarded(world: dict[str, Any]) -> None:
helper = Spy(
[call("write_file", path="half.py", content="half\n")],
[hand_back(succeeded=False)],
)
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="words"),
call("write_file", path="mine.py", content="M = 1\n"),
call("delegate", task="write half.py"),
call("ask_human", question="the helper couldn't; what now?"),
],
)
agent, transport = loop_agent(world, model, helpers=helpers(helper))
agent.intake()
say(transport, world, "go")
[result] = driver_results(world, "delegate")
assert "didn't finish" in result and "discarded" in result
root = world["repo"].root
assert not (root / "half.py").exists()
assert (root / "mine.py").exists() # the driver's own work stays
assert world["repo"].current_branch() == loop_of(item(world)).branch
assert not [b for b in local_branches(world) if "-help-" in b]
assert not [
i for i in world["store"].pending_intents() if i.kind == DELEGATION_INTENT
]
def test_delegating_needs_a_milestone(world: dict[str, Any]) -> None:
model = ScriptedModel(
RESTATE,
[
call("delegate", task="anything"),
call("ask_human", question="ok?"),
],
)
agent, transport = loop_agent(world, model, helpers=helpers(Spy()))
agent.intake()
say(transport, world, "go")
assert driver_results(world, "delegate") == [
"start a milestone first: a helper works on its branch"
]
def test_a_stuck_driver_gets_a_helper_once_per_milestone(
world: dict[str, Any],
) -> None:
forge: Forge = world["forge"]
helper = Spy(
[call("write_file", path="fixed.py", content="F = 1\n")],
[hand_back()],
)
failing = [
call("run_command", command="false"),
call("run_command", command="false"),
]
model = ScriptedModel(
RESTATE,
[call("start_milestone", name="words"), *failing],
[call("finish_milestone", summary="fixed with help", final="yes")],
)
agent, transport = loop_agent(world, model, helpers=helpers(helper))
agent._driver._failure_limit = 2
agent.intake()
say(transport, world, "go")
assert any("so a helper is taking over" in m.text for m in transport.outbound)
assert "The driver got stuck" in str(helper.calls) # in the brief
[pr] = forge.prs
assert forge.files(pr) == ["fixed.py"]
assert loop_of(item(world)).helped_at == 1
def test_stuck_again_in_the_same_milestone_asks_the_operator(
world: dict[str, Any],
) -> None:
helper = Spy(
[call("write_file", path="tried.py", content="T = 1\n")],
[hand_back()],
)
failing = [
call("run_command", command="false"),
call("run_command", command="false"),
]
model = ScriptedModel(
RESTATE,
[call("start_milestone", name="words"), *failing],
failing, # after the helper: stuck again
)
agent, transport = loop_agent(world, model, helpers=helpers(helper))
agent._driver._failure_limit = 2
agent.intake()
say(transport, world, "go")
loop = loop_of(item(world))
assert loop.wait == ANSWER and "commands failed in a row" in loop.note
assert len(helper.calls) == 2 # one delegation: its two turns
def test_ask_expert_answers_within_its_budget(world: dict[str, Any]) -> None:
answer = {"answer": "use a dict", "confidence": "high", "caveats": ""}
expert = Spy(
[AssistantText(text=json.dumps(answer), model="exp")],
[AssistantText(text=json.dumps(answer), model="exp")],
)
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="words"),
call("ask_expert", question="dict or list?"),
call("ask_expert", question="and again?"),
call("ask_expert", question="once more?"),
call("ask_human", question="ok?"),
],
)
agent, transport = loop_agent(
world, model, helpers=helpers(Spy(), expert, expert_budget=2)
)
agent.intake()
say(transport, world, "go")
first, second, third = driver_results(world, "ask_expert")
assert first.startswith("use a dict") and second.startswith("use a dict")
assert "budget" in third
assert expert.calls[0][0] == "exp"
def test_roles_are_pinned_when_the_item_opens(world: dict[str, Any]) -> None:
spy = Spy(RESTATE)
agent, _ = loop_agent(world, spy, helpers=helpers(Spy())) # type: ignore[arg-type]
agent._roles = {**ROLES.pinned(), "driver": "pinned-driver"}
agent.intake()
assert spy.calls[0][0] == "pinned-driver"
pinned = item(world).data["roles"]
assert pinned["driver"] == "pinned-driver" and pinned["classifier"] == "exp"
def test_the_driver_is_told_about_its_helpers_by_profile(
world: dict[str, Any],
) -> None:
spy = Spy(RESTATE, [call("ask_human", question="ok?")])
agent, transport = loop_agent(world, spy, helpers=helpers(Spy())) # type: ignore[arg-type]
agent.intake()
say(transport, world, "go")
after_go = spy.calls[1][1]
assert "Asking for help is normal" in after_go
assert "tool calls: reliable; strong at tricky tests" in after_go
assert "hlp" not in after_go # never a model's name
def test_a_delegation_cut_off_by_a_crash_is_discarded_at_boot(
world: dict[str, Any],
) -> None:
model = ScriptedModel(
RESTATE,
[call("start_milestone", name="words"), call("ask_human", question="?")],
)
agent, transport = loop_agent(world, model, helpers=helpers(Spy()))
agent.intake()
say(transport, world, "go")
milestone = loop_of(item(world)).branch
repo = world["repo"]
repo.checkout_new_branch(f"{milestone}-help-1")
(repo.root / "partial.py").write_text("p\n")
repo.commit_all("partial")
world["store"].begin_intent(
DELEGATION_INTENT,
{"branch": f"{milestone}-help-1", "base": milestone},
cause="delegating",
work_item=item(world).id,
)
agent2, _ = loop_agent(world, ScriptedModel(), helpers=helpers(Spy()))
agent2.boot()
assert repo.current_branch() == milestone
assert not (repo.root / "partial.py").exists()
assert f"{milestone}-help-1" not in repo.local_branches()
assert not [
i for i in world["store"].pending_intents() if i.kind == DELEGATION_INTENT
]
facts = [
e.event.text
for e in world["store"].transcript(item(world).id, "orchestrator").read()
if isinstance(e.event, Fact)
]
assert any("cut off by a restart and discarded" in f for f in facts)
assert git
def test_a_helpers_progress_is_posted(world: dict[str, Any]) -> None:
helper = Spy(
[
AssistantText(text="Writing helped.py with the constant.", model="hlp"),
call("write_file", path="helped.py", content="H = 1\n"),
],
[hand_back()],
)
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="words"),
call("delegate", task="write helped.py"),
call("ask_human", question="done?"),
],
)
agent, transport = loop_agent(world, model, helpers=helpers(helper))
agent.intake()
say(transport, world, "go")
said = [m.text for m in transport.outbound]
assert "helper: Writing helped.py with the constant." in said
+5 -12
View File
@@ -32,7 +32,7 @@ class Reader:
self.yes = set(yes) self.yes = set(yes)
self.asked: list[tuple[str, str]] = [] self.asked: list[tuple[str, str]] = []
def agrees(self, proposal: str, reply: str, model: str = "") -> bool: def agrees(self, proposal: str, reply: str) -> bool:
self.asked.append((proposal, reply)) self.asked.append((proposal, reply))
return reply in self.yes return reply in self.yes
@@ -118,7 +118,7 @@ def test_a_plain_yes_is_a_go_ahead_said_so_first(tmp_path: Path) -> None:
def test_anything_else_goes_to_the_driver(tmp_path: Path) -> None: def test_anything_else_goes_to_the_driver(tmp_path: Path) -> None:
h = Harness( h = Harness(
tmp_path, tmp_path,
ScriptedModel(RESTATED, [call("restate", text="v2", changed="as you asked")]), ScriptedModel(RESTATED, [call("restate", text="v2")]),
FakeGitea(issue()), FakeGitea(issue()),
reader=Reader(), reader=Reader(),
) )
@@ -132,7 +132,7 @@ def test_a_question_never_reaches_the_reader(tmp_path: Path) -> None:
reader = Reader("sounds good?") reader = Reader("sounds good?")
h = Harness( h = Harness(
tmp_path, tmp_path,
ScriptedModel(RESTATED, [call("restate", text="v2", changed="as you asked")]), ScriptedModel(RESTATED, [call("restate", text="v2")]),
FakeGitea(issue()), FakeGitea(issue()),
reader=reader, reader=reader,
) )
@@ -145,9 +145,7 @@ def test_a_plain_yes_to_an_older_revision_is_not_a_go(tmp_path: Path) -> None:
reader = Reader("yes") reader = Reader("yes")
h = Harness( h = Harness(
tmp_path, tmp_path,
ScriptedModel( ScriptedModel(RESTATED, [call("restate", text="v2")], STARTED),
RESTATED, [call("restate", text="v2", changed="as you asked")], STARTED
),
FakeGitea(issue()), FakeGitea(issue()),
reader=reader, reader=reader,
) )
@@ -226,12 +224,7 @@ def test_a_reply_stops_the_bugs_countdown(tmp_path: Path) -> None:
def test_a_new_revision_of_a_bug_gets_its_own_countdown(tmp_path: Path) -> None: def test_a_new_revision_of_a_bug_gets_its_own_countdown(tmp_path: Path) -> None:
h, clock = bug( h, clock = bug(tmp_path, RESTATED, [call("restate", text="v2")], STARTED)
tmp_path,
RESTATED,
[call("restate", text="v2", changed="as you asked")],
STARTED,
)
h.agent.intake() h.agent.intake()
clock.now += 300 clock.now += 300
h.say("use the other file") h.say("use the other file")
+4 -5
View File
@@ -252,11 +252,11 @@ def make_world(
def loop_agent( def loop_agent(
world: dict[str, Any], model: ScriptedModel, **agent: Any world: dict[str, Any], model: ScriptedModel
) -> tuple[LoopAgent, InMemoryTransport]: ) -> tuple[LoopAgent, InMemoryTransport]:
transport = InMemoryTransport() transport = InMemoryTransport()
cancel = Cancel() cancel = Cancel()
loop = LoopAgent( agent = LoopAgent(
world["store"], world["store"],
transport, transport,
world["forge"], world["forge"],
@@ -271,10 +271,9 @@ def loop_agent(
cancel=cancel, cancel=cancel,
oplog=OpLog("nisse-dev", io.StringIO()), oplog=OpLog("nisse-dev", io.StringIO()),
milestones=world["milestones"], milestones=world["milestones"],
**agent,
) )
transport.on_message(loop.on_message) transport.on_message(agent.on_message)
return loop, transport return agent, transport
def item(world: dict[str, Any]) -> Any: def item(world: dict[str, Any]) -> Any:
+1 -43
View File
@@ -19,7 +19,6 @@ from test_loop_milestones import (
from test_loop_milestones import world as world # noqa: PLC0414 -- the fixture from test_loop_milestones import world as world # noqa: PLC0414 -- the fixture
from nisse_dev.canonical import ToolCall, ToolResult from nisse_dev.canonical import ToolCall, ToolResult
from nisse_dev.gitea import Issue
from nisse_dev.loop.items import MERGE, WAITING, loop_of from nisse_dev.loop.items import MERGE, WAITING, loop_of
@@ -95,7 +94,7 @@ def test_the_next_milestone_waits_for_the_merge(world: dict[str, Any]) -> None:
agent, transport = loop_agent(world, model) agent, transport = loop_agent(world, model)
agent.intake() agent.intake()
say(transport, world, "go") say(transport, world, "go")
say(transport, world, "carry on") say(transport, world, "how's it going?")
assert "hasn't merged yet" in result_of(world, "start_milestone") assert "hasn't merged yet" in result_of(world, "start_milestone")
assert (item(world).position, loop_of(item(world)).wait) == (WAITING, MERGE) assert (item(world).position, loop_of(item(world)).wait) == (WAITING, MERGE)
assert len(forge.prs) == 1 assert len(forge.prs) == 1
@@ -175,44 +174,3 @@ def test_an_unsaid_final_closes_the_issue(world: dict[str, Any]) -> None:
say(transport, world, "go") say(transport, world, "go")
[pr] = forge.prs [pr] = forge.prs
assert "Closes #14" in forge.prs[pr]["body"] assert "Closes #14" in forge.prs[pr]["body"]
def test_a_design_milestone_holds_documents_only(world: dict[str, Any]) -> None:
"""D1, D2: a `design` item agrees its design before any code; the
harness refuses a design milestone with code in it."""
forge: Forge = world["forge"]
forge.issues[0] = Issue(14, "Add count_words", "count words", "don", ("design",))
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="design"),
call("write_file", path="ARCHITECTURE.md", content="# arch\n"),
call("write_file", path="docs/plans/words.md", content="# plan\n"),
call("write_file", path="words.py", content="W = 1\n"), # too soon
call("finish_milestone", summary="design", final="no"),
],
[
call("run_command", command="rm words.py"),
call("finish_milestone", summary="design", final="no"),
],
milestone("words.py", final="yes"), # after the design merged
)
agent, transport = loop_agent(world, model)
agent.intake()
say(transport, world, "go")
[design] = forge.prs
assert sorted(forge.files(design)) == ["ARCHITECTURE.md", "docs/plans/words.md"]
refused = [
e.event.result
for e in world["store"].transcript(item(world).id, "driver").read()
if isinstance(e.event, ToolResult)
and "this milestone is the design" in e.event.result
]
assert refused and "words.py" in refused[0]
forge.merge(design)
agent.watch()
agent._schedule()
code = max(forge.prs)
assert code != design and forge.files(code) == ["words.py"]
assert loop_of(item(world)).design_merged
+6 -89
View File
@@ -23,13 +23,10 @@ from test_loop_projects import Router
from nisse_dev.cancel import Cancel from nisse_dev.cancel import Cancel
from nisse_dev.canonical import Fact, PlanSubmitted, ToolCall, UserMessage from nisse_dev.canonical import Fact, PlanSubmitted, ToolCall, UserMessage
from nisse_dev.delegation import DELEGATION_INTENT, Caps
from nisse_dev.execution import SSHHost from nisse_dev.execution import SSHHost
from nisse_dev.gitea import Comment, Issue from nisse_dev.gitea import Comment, Issue
from nisse_dev.llm import RoleMap
from nisse_dev.loop.agent import LoopAgent, Project from nisse_dev.loop.agent import LoopAgent, Project
from nisse_dev.loop.driver import Driver from nisse_dev.loop.driver import Driver
from nisse_dev.loop.help import Helpers
from nisse_dev.loop.items import ( from nisse_dev.loop.items import (
ANSWER, ANSWER,
BLOCKED, BLOCKED,
@@ -66,18 +63,11 @@ class PolicyModel:
self.n += 1 self.n += 1
offered = {t["function"]["name"] for t in tools} offered = {t["function"]["name"] for t in tools}
facts = [getattr(e.event, "text", "") for e in events] facts = [getattr(e.event, "text", "") for e in events]
# A `design` item writes documents until its design has merged.
design = "labelled design" in (system or "") and not any(
"the design has merged" in t for t in facts
)
path = f"docs/d{self.n}.md" if design else f"f{self.n}.py"
roll = self.rng.random() roll = self.rng.random()
if roll < 0.08: if roll < 0.08:
return [call("ask_human", question=f"question {self.n}?")] return [call("ask_human", question=f"question {self.n}?")]
if roll < 0.12: if roll < 0.12:
return [ToolCall(call_id=f"p{self.n}", tool="nonsense", arguments={})] return [ToolCall(call_id=f"p{self.n}", tool="nonsense", arguments={})]
if "reply" in offered and "propose_work" not in offered:
return [call("reply", text=f"because {self.n}")] # a question run
if "reply" in offered: # a conversation: talk, or propose work if "reply" in offered: # a conversation: talk, or propose work
if self.rng.random() < 0.1: if self.rng.random() < 0.1:
return [ return [
@@ -101,16 +91,14 @@ class PolicyModel:
if "restate" not in offered and "finish_milestone" not in offered: if "restate" not in offered and "finish_milestone" not in offered:
return [call("ask_human", question=f"talk first {self.n}?")] # discuss return [call("ask_human", question=f"talk first {self.n}?")] # discuss
if "finish_milestone" not in offered: if "finish_milestone" not in offered:
return [call("restate", text=f"restatement {self.n}", changed=f"c{self.n}")] return [call("restate", text=f"restatement {self.n}")]
if "delegate" in offered and self.rng.random() < 0.1:
return [call("delegate", task=f"help {self.n}")]
if "wait_for_review" in offered and self.rng.random() < 0.2: if "wait_for_review" in offered and self.rng.random() < 0.2:
return [call("wait_for_review", note=f"waiting {self.n}")] return [call("wait_for_review", note=f"waiting {self.n}")]
if self.rng.random() < 0.15: if self.rng.random() < 0.15:
# Half done, then a question: work a switch must park. # Half done, then a question: work a switch must park.
return [ return [
call("start_milestone", name=f"m{self.n}"), call("start_milestone", name=f"m{self.n}"),
call("write_file", path=path, content=f"x = {self.n}\n"), call("write_file", path=f"f{self.n}.py", content=f"x = {self.n}\n"),
call("ask_human", question=f"halfway {self.n}?"), call("ask_human", question=f"halfway {self.n}?"),
] ]
merged = any("merged" in t for t in facts[-6:]) merged = any("merged" in t for t in facts[-6:])
@@ -118,7 +106,7 @@ class PolicyModel:
return [call("close_item", summary=f"done {self.n}")] return [call("close_item", summary=f"done {self.n}")]
return [ return [
call("start_milestone", name=f"m{self.n}"), call("start_milestone", name=f"m{self.n}"),
call("write_file", path=path, content=f"x = {self.n}\n"), call("write_file", path=f"f{self.n}.py", content=f"x = {self.n}\n"),
call( call(
"finish_milestone", "finish_milestone",
summary=f"work {self.n}", summary=f"work {self.n}",
@@ -127,32 +115,6 @@ class PolicyModel:
] ]
class HelperModel:
"""A scripted helper: does some work, then hands back, succeeding or
not as the seed says."""
def __init__(self, rng: random.Random) -> None:
self.rng = rng
self.n = 0
def complete(self, events, tools, model, system=None) -> list[Any]:
assert_renderable(events)
self.n += 1
return [
call("write_file", path=f"h{self.n}.py", content=f"h = {self.n}\n"),
ToolCall(
call_id=f"hb{self.n}",
tool="hand_back",
arguments={
"changed": f"h{self.n}.py",
"now_true": "helped",
"next": "go on",
"succeeded": self.rng.random() < 0.6,
},
),
]
class Walk: class Walk:
"""``worlds``: project name -> its world (forge, bare repo, clone, """``worlds``: project name -> its world (forge, bare repo, clone,
milestones), all over one store.""" milestones), all over one store."""
@@ -207,18 +169,13 @@ class Walk:
cancel=cancel, cancel=cancel,
oplog=OpLog("nisse-dev", _Null()), oplog=OpLog("nisse-dev", _Null()),
reader=self, # type: ignore[arg-type] reader=self, # type: ignore[arg-type]
helpers=Helpers(
client=HelperModel(self.rng), # type: ignore[arg-type]
roles=RoleMap(executor="", planner="m", helper="h", expert="x"),
caps=Caps(turns=4),
),
bug_grace_s=600, bug_grace_s=600,
clock=lambda: self.now, clock=lambda: self.now,
) )
transport.on_message(agent.on_message) transport.on_message(agent.on_message)
return agent, transport return agent, transport
def agrees(self, proposal: str, reply: str, model: str = "") -> bool: def agrees(self, proposal: str, reply: str) -> bool:
return self.reads_yes(proposal, reply) return self.reads_yes(proposal, reply)
def items(self) -> list[Any]: def items(self) -> list[Any]:
@@ -250,7 +207,7 @@ class Walk:
forge: Forge = self.worlds[project]["forge"] forge: Forge = self.worlds[project]["forge"]
if item and loop: if item and loop:
weights.update({"go": 3, "answer": 2, "abandon": 1, "go-stale": 1}) weights.update({"go": 3, "answer": 2, "abandon": 1, "go-stale": 1})
weights.update({"plain-yes": 2, "wait": 2, "question": 2}) weights.update({"plain-yes": 2, "wait": 2})
if loop.new_project: if loop.new_project:
weights.update({"done": 2}) weights.update({"done": 2})
if item.scope != "general": # conversations aren't named by number if item.scope != "general": # conversations aren't named by number
@@ -266,28 +223,6 @@ class Walk:
elif action == "done" and loop: elif action == "done" and loop:
# The operator says a new project is ready; no entry was added. # The operator says a new project is ready; no entry was added.
self.transport.receive(DON, "done", thread=loop.thread) self.transport.receive(DON, "done", thread=loop.thread)
elif action == "question" and loop:
# L5: a question never changes the item it's asked about.
before = (item.position, loop.wait, loop.rev, loop.pr, loop.branch)
self.transport.receive(DON, f"why {len(self.steps)}?", thread=loop.thread)
after_item = self.world["store"].work_item(item.id)
after = loop_of(after_item)
if (
item.scope != "general"
and item.position == WAITING
and loop.wait in (GO_AHEAD, ANSWER, MERGE)
):
now = (
after_item.position,
after.wait,
after.rev,
after.pr,
after.branch,
)
assert now == before, (
f"questions_change_nothing: {before} -> {now}: "
f"seed {self.seed}, steps {self.steps}"
)
elif action == "chat": elif action == "chat":
# A top-level message: a conversation of its own (L1, L5). # A top-level message: a conversation of its own (L1, L5).
self.transport.receive(DON, f"thinking aloud {len(self.steps)}") self.transport.receive(DON, f"thinking aloud {len(self.steps)}")
@@ -464,22 +399,7 @@ class Walk:
assert not (loop.prepared or loop.milestone or loop.pr), ( assert not (loop.prepared or loop.milestone or loop.pr), (
f"conversations_do_no_work: {item.id}: {where}" f"conversations_do_no_work: {item.id}: {where}"
) )
# L9: a delegation never outlives its step: merged back or discarded.
assert not [
i for i in store.pending_intents() if i.kind == DELEGATION_INTENT
], f"delegations_settle (intent): {where}"
for name, w in self.worlds.items(): for name, w in self.worlds.items():
helping = [
b
for b in [
*w["repo"].local_branches(),
*_git(
w["bare"], "for-each-ref", "--format=%(refname:short)"
).split(),
]
if "-help-" in b
]
assert not helping, f"delegations_settle (branch {helping}): {where}"
self._records_have_one_item(name, w, where) self._records_have_one_item(name, w, where)
self._own_repository_only(name, w, where) self._own_repository_only(name, w, where)
self._parked_work_safe(w, name, where) self._parked_work_safe(w, name, where)
@@ -594,10 +514,7 @@ class Walk:
written = { written = {
str(e.event.arguments.get("path")) str(e.event.arguments.get("path"))
for e in self.world["store"].transcript(item.id, "driver").read() for e in self.world["store"].transcript(item.id, "driver").read()
# The driver's own: a failed helper's are discarded on purpose. if isinstance(e.event, ToolCall) and e.event.tool == "write_file"
if isinstance(e.event, ToolCall)
and e.event.tool == "write_file"
and e.actor == "driver"
} }
here = current.startswith(f"{item.id}-") here = current.startswith(f"{item.id}-")
for path in written: for path in written:
-163
View File
@@ -1,163 +0,0 @@
"""Conversation on the new loop (M2.7, L5, #42, #77): questions answered
without changing the item, revisions that say what changed, progress in
prose at a readable rate, raw tool activity on request, and a note for a
message that can't be read yet."""
from pathlib import Path
from typing import Any
from test_loop import DON, RESTATED, FakeGitea, Harness, ScriptedModel, call, issue
from nisse_dev.canonical import AssistantText
from nisse_dev.loop.items import ANSWER, GO_AHEAD, WAITING, loop_of
class Clock:
def __init__(self) -> None:
self.now = 1000.0
def __call__(self) -> float:
return self.now
def harness(tmp_path: Path, *turns: list[Any], **kw: Any) -> Harness:
return Harness(tmp_path, ScriptedModel(*turns), FakeGitea(issue()), **kw)
# --- questions never change state (#77) -------------------------------------
def test_a_question_about_a_restatement_leaves_it_up(tmp_path: Path) -> None:
h = harness(
tmp_path,
RESTATED,
[call("reply", text="Because text.py already holds the string helpers.")],
[call("ask_human", question="started; anything else?")],
)
h.agent.intake()
h.say("Why text.py?")
loop = loop_of(h.item())
assert (h.item().position, loop.wait, loop.rev) == (WAITING, GO_AHEAD, 1)
assert not loop.question and not loop.started
assert h.posted()[-1] == "Because text.py already holds the string helpers."
h.say("go") # still bound to rev-1
assert loop_of(h.item()).started
def test_a_question_back_keeps_the_agents_own_question_open(tmp_path: Path) -> None:
h = harness(
tmp_path,
[call("ask_human", question="One function or two?")],
[call("reply", text="One handles words, two would split lines too.")],
)
h.agent.intake()
h.say("What would the second one do?")
loop = loop_of(h.item())
assert (loop.wait, loop.note) == (ANSWER, "One function or two?")
def test_a_question_that_isnt_answered_still_goes_back(tmp_path: Path) -> None:
h = harness(
tmp_path,
RESTATED,
[call("restate", text="sneaky v2", changed="x")], # not offered
[AssistantText(text="hmm", model="m")],
[AssistantText(text="hmm", model="m")], # prose twice: the run fails
)
h.agent.intake()
h.say("why?")
loop = loop_of(h.item())
# restate isn't offered in a question run: no new revision.
assert (loop.wait, loop.rev) == (GO_AHEAD, 1)
assert h.posted()[-1].startswith("I couldn't answer that")
# --- explanations ------------------------------------------------------------
def test_a_revision_says_what_changed(tmp_path: Path) -> None:
h = harness(
tmp_path,
RESTATED,
[call("restate", text="count_words in words.py", changed="")],
[
call(
"restate",
text="count_words in words.py",
changed="a module of its own, as you asked",
)
],
)
h.agent.intake()
h.say("put it in its own module")
loop = loop_of(h.item())
assert loop.rev == 2
assert "Changed from rev-1: a module of its own, as you asked" in h.posted()[-1]
# --- progress and details (#42) ----------------------------------------------
def test_progress_is_posted_at_a_readable_rate(tmp_path: Path) -> None:
clock = Clock()
class Ticking(ScriptedModel):
def complete(self, events, tools, model, system=None): # type: ignore[no-untyped-def]
clock.now += 30 # each turn takes a while
return super().complete(events, tools, model, system)
h = Harness(
tmp_path,
Ticking(
[
AssistantText(text="Looking at text.py first.", model="m"),
call("list_dir", path="."),
],
[
AssistantText(text="Reading the tests too.", model="m"),
call("list_dir", path="tests"),
],
[
AssistantText(text="x" * 1000, model="m"),
call("list_dir", path="."),
],
[call("restate", text="I'll add count_words.", changed="")],
),
FakeGitea(issue()),
clock=clock,
)
h.agent.intake()
said = h.posted()
assert "Looking at text.py first." in said
assert "Reading the tests too." not in said # 30 s later: folded away
long = [p for p in said if p.startswith("xxx")]
assert long and len(long[0]) < 420 and long[0].endswith(" …") # 60 s on
def test_details_shows_the_last_tool_calls(tmp_path: Path) -> None:
h = harness(
tmp_path,
[
call("list_dir", path="src"),
call("read_file", path="text.py"),
call("restate", text="I'll add count_words.", changed=""),
],
)
h.agent.intake()
h.say("details")
shown = h.posted()[-1]
assert shown.startswith("The last tool calls on #14:")
assert "list_dir(src)" in shown and "read_file(text.py)" in shown
assert loop_of(h.item()).wait == GO_AHEAD # nothing changed
def test_a_message_mid_run_is_acknowledged(tmp_path: Path) -> None:
h = harness(tmp_path, RESTATED)
h.agent.intake()
h.agent._running = True # as if a driver run were under way
h.agent._active_thread = "elsewhere"
h.agent._active_name = "#15"
h.transport.receive(DON, "one more thing", thread=h.thread())
assert "Noted; I'm in the middle of #15" in h.posted()[-1]
h.transport.receive(DON, "and another", thread="elsewhere")
assert h.posted()[-1] == "Noted; I'll read it when this turn ends."
+22 -54
View File
@@ -1,19 +1,13 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Bring-up runs of nisse-dev on this development VM (Development Plan M0.2). # Bring-up runs of nisse-dev on this development VM (Development Plan M0.2).
# #
# The agent runs as its container image (M1.5), built from the code under # The agent runs as root, as it will on its own VM in production (D7): it may
# test, as a transient systemd unit, so it outlives the shell that started # install what it needs. It touches only test resources (the sandbox repo, the
# it. It touches only test resources (an operator-made test repository, the # test Matrix room, a budget-capped LiteLLM key), from the run's config. It
# test Matrix room, a budget-capped LiteLLM key), from the run's config. # runs as a transient systemd unit, so it outlives the shell that started it.
# #
# $E2E is mounted at the same path inside the container, so the config's # scripts/bringup.sh start [REF] set the last run aside, export REF (default
# paths (state_db, keys) mean the same there; for that it runs as root here, # HEAD) and start the agent
# since the keys are root's. In production the image runs as its own user
# with a state volume (Dockerfile).
#
# scripts/bringup.sh start [REF] set the last run aside, build REF (default
# HEAD) into an image and start the agent
# scripts/bringup.sh plan [REF] print what start would build and run
# scripts/bringup.sh stop stop it cleanly (SIGTERM) # scripts/bringup.sh stop stop it cleanly (SIGTERM)
# scripts/bringup.sh status the unit, and the agent's state # scripts/bringup.sh status the unit, and the agent's state
# scripts/bringup.sh logs follow the agent's log # scripts/bringup.sh logs follow the agent's log
@@ -22,7 +16,7 @@
# #
# Layout, under $E2E (default /root/nisse-e2e): # Layout, under $E2E (default /root/nisse-e2e):
# nisse-dev.yaml, .env the run config and its secrets (made by hand once) # nisse-dev.yaml, .env the run config and its secrets (made by hand once)
# code/ the exported code under test, built into the image # code/ the exported code under test, with its own .venv
# state.db the agent's state (its workspace is on the build VM) # state.db the agent's state (its workspace is on the build VM)
# ssh/ the agent's key for the build VM, and push keys # ssh/ the agent's key for the build VM, and push keys
# run-info, collected/ what was run, and the output of `collect` # run-info, collected/ what was run, and the output of `collect`
@@ -33,28 +27,11 @@ E2E=${NISSE_E2E:-/root/nisse-e2e}
SRC=${NISSE_SRC:-/root/nisse} # repository to export code from SRC=${NISSE_SRC:-/root/nisse} # repository to export code from
CODE=$E2E/code CODE=$E2E/code
UNIT=nisse-bringup UNIT=nisse-bringup
IMAGE=nisse-dev:bringup UV=${UV:-$(command -v uv || echo /snap/bin/uv)}
die() { echo "bringup: $*" >&2; exit 1; } die() { echo "bringup: $*" >&2; exit 1; }
running() { systemctl is-active --quiet "$UNIT"; } running() { systemctl is-active --quiet "$UNIT"; }
# The container the unit runs: host networking (Matrix, Gitea, LiteLLM as
# here), $E2E at its own path, the config and secrets from it.
docker_run() {
echo docker run --rm --name "$UNIT" --network host --user 0 \
-v "$E2E:$E2E" \
-e NISSE_CONFIG="$E2E/nisse-dev.yaml" -e NISSE_ENV="$E2E/.env" \
"$IMAGE"
}
cmd_plan() {
local ref=${1:-HEAD} sha
sha=$(git -C "$SRC" rev-parse --verify "$ref^{commit}") || die "unknown ref $ref"
echo "git -C $SRC archive $sha | tar -x -C $CODE"
echo "docker build -t $IMAGE $CODE"
echo "systemd-run --unit=$UNIT ... $(docker_run)"
}
cmd_start() { cmd_start() {
local ref=${1:-HEAD} sha local ref=${1:-HEAD} sha
[ -f "$E2E/nisse-dev.yaml" ] && [ -f "$E2E/.env" ] || die "no config in $E2E" [ -f "$E2E/nisse-dev.yaml" ] && [ -f "$E2E/.env" ] || die "no config in $E2E"
@@ -81,16 +58,16 @@ cmd_start() {
rm -rf "$CODE" rm -rf "$CODE"
mkdir -p "$CODE" mkdir -p "$CODE"
git -C "$SRC" archive "$sha" | tar -x -C "$CODE" git -C "$SRC" archive "$sha" | tar -x -C "$CODE"
docker build --quiet -t "$IMAGE" "$CODE" > /dev/null (cd "$CODE" && "$UV" sync --quiet --all-packages)
printf 'ref: %s\ncommit: %s\nstarted: %s\n' "$ref" "$sha" "$(date -Is)" > "$E2E/run-info" printf 'ref: %s\ncommit: %s\nstarted: %s\n' "$ref" "$sha" "$(date -Is)" > "$E2E/run-info"
systemctl reset-failed "$UNIT" 2>/dev/null || true systemctl reset-failed "$UNIT" 2>/dev/null || true
# SIGTERM reaches the agent through docker run's signal proxy; it shuts
# down cleanly (main.py).
# shellcheck disable=SC2046
systemd-run --quiet --unit="$UNIT" --collect \ systemd-run --quiet --unit="$UNIT" --collect \
--property=WorkingDirectory="$CODE" \
--property=KillSignal=SIGTERM --property=TimeoutStopSec=120 \ --property=KillSignal=SIGTERM --property=TimeoutStopSec=120 \
$(docker_run) --setenv=NISSE_CONFIG="$E2E/nisse-dev.yaml" \
--setenv=NISSE_ENV="$E2E/.env" \
"$CODE/.venv/bin/nisse-dev"
echo "started $UNIT at ${sha:0:8} ($ref)" echo "started $UNIT at ${sha:0:8} ($ref)"
} }
@@ -101,17 +78,15 @@ remote_aside() {
where=$(python3 - "$E2E/nisse-dev.yaml" <<'PY' where=$(python3 - "$E2E/nisse-dev.yaml" <<'PY'
import sys, yaml import sys, yaml
c = yaml.safe_load(open(sys.argv[1])) c = yaml.safe_load(open(sys.argv[1]))
for p in c["projects"].values(): # each project's workspace, on its build VM (D8) b = c["build_host"]
b = p["build_host"] print(b.get("user", "root"), b["address"], b.get("port", 22), b["key"], c["workspace"])
print(b.get("user", "root"), b["address"], b.get("port", 22), b["key"], p["workspace"])
PY PY
) || die "cannot read the projects' build VMs and workspaces from the config" ) || die "cannot read build_host and workspace from the config"
while read -r user address port key workspace; do read -r user address port key workspace <<< "$where"
ssh -n -i "$key" -p "$port" -o BatchMode=yes -o StrictHostKeyChecking=accept-new \ ssh -i "$key" -p "$port" -o BatchMode=yes -o StrictHostKeyChecking=accept-new \
-o UserKnownHostsFile="$E2E/ssh/known_hosts" "$user@$address" \ -o UserKnownHostsFile="$E2E/ssh/known_hosts" "$user@$address" \
"if [ -e '$workspace' ]; then mv '$workspace' '$workspace.$stamp' && echo 'build VM: set aside $workspace.$stamp'; fi" \ "if [ -e '$workspace' ]; then mv '$workspace' '$workspace.$stamp' && echo 'build VM: set aside $workspace.$stamp'; fi" \
|| die "cannot reach the build VM $address" || die "cannot reach the build VM $address"
done <<< "$where"
} }
cmd_stop() { cmd_stop() {
@@ -123,35 +98,28 @@ cmd_stop() {
cmd_status() { cmd_status() {
if running; then echo "unit: running"; else echo "unit: not running"; fi if running; then echo "unit: running"; else echo "unit: not running"; fi
[ -f "$E2E/run-info" ] && sed 's/^/ /' "$E2E/run-info" [ -f "$E2E/run-info" ] && sed 's/^/ /' "$E2E/run-info"
if [ -f "$E2E/state.db" ]; then if [ -f "$E2E/state.db" ] && [ -x "$CODE/.venv/bin/python" ]; then
dump "$E2E/state.db" | head -n 3 "$CODE/.venv/bin/python" -m nisse_dev.dump "$E2E/state.db" | head -n 3
fi fi
} }
cmd_logs() { journalctl -u "$UNIT" -f -o cat; } cmd_logs() { journalctl -u "$UNIT" -f -o cat; }
# The state store, read with the image's own code.
dump() {
docker run --rm --user 0 -v "$E2E:$E2E" --entrypoint python "$IMAGE" \
-m nisse_dev.dump "$@"
}
cmd_collect() { cmd_collect() {
[ -f "$E2E/run-info" ] || die "no run to collect" [ -f "$E2E/run-info" ] || die "no run to collect"
local out=$E2E/collected since local out=$E2E/collected since
since=$(sed -n 's/^started: //p' "$E2E/run-info") since=$(sed -n 's/^started: //p' "$E2E/run-info")
mkdir -p "$out" mkdir -p "$out"
journalctl -u "$UNIT" --since "$since" -o cat > "$out/agent.log" journalctl -u "$UNIT" --since "$since" -o cat > "$out/agent.log"
dump --full "$E2E/state.db" > "$out/transcript.txt" "$CODE/.venv/bin/python" -m nisse_dev.dump --full "$E2E/state.db" > "$out/transcript.txt"
echo "collected into $out" echo "collected into $out"
} }
case "${1:-}" in case "${1:-}" in
start) shift; cmd_start "$@" ;; start) shift; cmd_start "$@" ;;
plan) shift; cmd_plan "$@" ;;
stop) cmd_stop ;; stop) cmd_stop ;;
status) cmd_status ;; status) cmd_status ;;
logs) cmd_logs ;; logs) cmd_logs ;;
collect) cmd_collect ;; collect) cmd_collect ;;
*) sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;; *) sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
esac esac
+4 -44
View File
@@ -1,33 +1,16 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# The end-to-end test (Development Plan M1.6): the whole agent against real # The end-to-end test (Development Plan M1.6): the whole agent against real
# services, driven by a scripted operator. CI runs it on every PR (with # services, driven by a scripted operator. Run after each milestone; it
# --local); locally, run only the tests you're writing (-k) or diagnosing # spends model tokens and takes minutes, so it is not part of check.sh or CI.
# (--lf).
# #
# scripts/e2e.sh on the bring-up build VM (configured below) # scripts/e2e.sh on the bring-up build VM (configured below)
# scripts/e2e.sh --local on a throwaway local sshd instead # scripts/e2e.sh --local on a throwaway local sshd instead
# scripts/e2e.sh --image REF inside an image's `test` stage (Dockerfile),
# on its own sshd: checks a release as built
# #
# Uses tests/integration/nisse-test.yaml and its secrets (.env at the repo # Uses tests/integration/nisse-test.yaml and its secrets (.env at the repo
# root), like the integration tests. Every role runs on a local model # root), like the integration tests.
# (tests/integration/e2e_support.py; NISSE_E2E_MODEL overrides it), so tell
# the operator before a full run: it keeps the RTX 4090 busy for a while.
set -euo pipefail set -euo pipefail
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."
if [ "${1:-}" = "--image" ]; then
image=${2:?--image needs an image reference}
shift 2
runs=${NISSE_E2E_RUNS:-/root/nisse-e2e/test-runs}
mkdir -p "$runs"
secrets=()
[ -f .env ] && secrets=(--env-file .env)
# Host networking: the suite reaches Matrix, Gitea and LiteLLM as here.
exec docker run --rm --network host "${secrets[@]}" \
-e NISSE_E2E_RUNS=/runs -v "$runs:/runs" \
"$image" scripts/e2e.sh --local "$@"
fi
if [ "${1:-}" = "--local" ]; then if [ "${1:-}" = "--local" ]; then
shift shift
else else
@@ -36,27 +19,4 @@ else
fi fi
# Each run's state store is kept here, for nisse_dev.dump after a failure. # Each run's state store is kept here, for nisse_dev.dump after a failure.
export NISSE_E2E_RUNS=${NISSE_E2E_RUNS:-/root/nisse-e2e/test-runs} export NISSE_E2E_RUNS=${NISSE_E2E_RUNS:-/root/nisse-e2e/test-runs}
exec uv run pytest -m e2e -rs tests/integration/test_end_to_end.py tests/integration/test_end_to_end_loop.py "$@"
# What the run spent on the test key, read from LiteLLM (spend only). The
# end-to-end tests run on a local model, so anything above $0 means a call
# went to a paid one: an override, or a proxy fallback.
spend() {
(set -a; [ -f .env ] && . ./.env; set +a
uv run python -c '
import os, httpx
r = httpx.get("https://litellm.dongar.net/key/info", timeout=30,
headers={"Authorization": "Bearer " + os.environ["NISSE_LLM_TOKEN"]})
i = r.json()["info"]
print(i["spend"], i.get("max_budget"))' 2>/dev/null) || echo "? ?"
}
read -r before budget < <(spend)
status=0
uv run pytest -m e2e -rs tests/integration/test_end_to_end.py tests/integration/test_end_to_end_loop.py "$@" || status=$?
read -r after _ < <(spend)
python3 -c "
b, a, cap = '$before', '$after', '$budget'
try:
print(f'e2e spend: \${float(a) - float(b):.4f} this run; \${float(a):.2f} of \${float(cap):.2f} today')
except ValueError:
print('e2e spend: unknown (could not read the key)')"
exit "$status"
-11
View File
@@ -11,17 +11,6 @@ from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from nisse_dev.gitea import GiteaClient, GiteaError from nisse_dev.gitea import GiteaClient, GiteaError
from nisse_dev.llm import RoleMap
# The end-to-end tests run every role on a local model, which costs nothing
# per call; NISSE_E2E_MODEL overrides it (e.g. claude-haiku-4-5 for a quick,
# paid run).
E2E_MODEL = "qwen3.8-27b"
def e2e_roles() -> RoleMap:
model = os.environ.get("NISSE_E2E_MODEL", "").strip() or E2E_MODEL
return RoleMap(executor=model, planner=model, helper=model, expert=model)
def task(run: str) -> str: def task(run: str) -> str:
+1 -2
View File
@@ -25,7 +25,7 @@ from pathlib import Path
from typing import Any from typing import Any
import pytest import pytest
from e2e_support import e2e_roles, fresh_repo, operator_gitea from e2e_support import fresh_repo, operator_gitea
from e2e_support import task as make_task from e2e_support import task as make_task
from nisse_dev.agent import Agent, build_agent from nisse_dev.agent import Agent, build_agent
@@ -84,7 +84,6 @@ def agent(
workspace=Path(f"/tmp/nisse-e2e-{run}/sandbox"), workspace=Path(f"/tmp/nisse-e2e-{run}/sandbox"),
build_host=build_host(ssh_host), build_host=build_host(ssh_host),
), ),
roles=e2e_roles(),
state_db=tmp_path / "state.db", state_db=tmp_path / "state.db",
) )
transport = InMemoryTransport() transport = InMemoryTransport()
+1 -34
View File
@@ -22,7 +22,7 @@ from pathlib import Path
from typing import Any from typing import Any
import pytest import pytest
from e2e_support import dependent_task, e2e_roles, fresh_repo, operator_gitea, task from e2e_support import dependent_task, fresh_repo, operator_gitea, task
from nisse_dev.config import AgentConfig, LoopTuning from nisse_dev.config import AgentConfig, LoopTuning
from nisse_dev.execution import SSHHost from nisse_dev.execution import SSHHost
@@ -75,7 +75,6 @@ def running(
build_host=build_host(ssh_host), build_host=build_host(ssh_host),
), ),
loop="new", loop="new",
roles=e2e_roles(),
state_db=keep / "state.db", state_db=keep / "state.db",
# Short, so a bug's grace period runs out within the test. # Short, so a bug's grace period runs out within the test.
tuning=replace(live_config.tuning, loop=LoopTuning(bug_grace_s=BUG_GRACE_S)), tuning=replace(live_config.tuning, loop=LoopTuning(bug_grace_s=BUG_GRACE_S)),
@@ -634,35 +633,3 @@ def test_a_conversation_becomes_a_new_project(
] ]
assert "ARCHITECTURE.md" in files, files assert "ARCHITECTURE.md" in files, files
assert any(f.startswith("docs/plans/") for f in files), files assert any(f.startswith("docs/plans/") for f in files), files
def test_a_question_leaves_the_restatement_up(
running: tuple[RunningLoop, InMemoryTransport, str],
) -> None:
"""M2.7: a question while a restatement is up is answered, and changes
nothing: the same revision stays up, and a go binds to it."""
agent, transport, run = running
operator = Operator(agent, transport)
repo = agent.config.gitea_repo
as_operator = operator_gitea(agent.config.gitea_url)
issues: list[int] = []
try:
issues.append(as_operator.create_issue(repo, f"vowels_{run}", task(run)))
as_operator._request(
"PATCH",
f"/repos/{repo}/issues/{issues[0]}",
json={"assignees": [agent.account]},
)
operator.tick()
item = operator.drive_until(GO_AHEAD, issues[0])
rev = loop_of(item).rev
said = len(transport.outbound)
operator.say("Why put it in that module rather than a new one?", issues[0])
answer = [m.text for m in transport.outbound[said:]]
assert answer and not answer[-1].startswith("rev-"), answer
loop = loop_of(operator.item(issues[0]))
assert (loop.wait, loop.rev) == (GO_AHEAD, rev), loop
operator.say("go", issues[0])
assert loop_of(operator.item(issues[0])).started
finally:
operator.tidy(issues, [])