Compare commits

...
24 Commits
Author SHA1 Message Date
dev-agent 07a2eb9d10 Merge pull request 'Wait out LLM outages with backoff (L5)' (#116) from llm-retry into main
CI / e2e (push) Skipped
CI / checks (push) Skipped
Release / build-and-push (release) Successful in 19s
2026-10-07 22:29:40 +00:00
dev-agent cd9eb3fd76 Merge pull request 'The agent as a container image (M1.5)' (#115) from image into main
CI / checks (push) Successful in 5m44s
CI / e2e (push) Skipped
2026-10-07 21:48:43 +00:00
dongarandClaude Opus 5.5 ad8a07fcb7 Wait out LLM outages with backoff (L5: outages delay, never fail)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 6m14s
CI / e2e (pull_request) Successful in 30m8s
A 502, 503 or 504 from the LiteLLM proxy, or a dropped connection, is
retried after 5, 15 and 45 seconds before the call fails. Twice on
2026-10-07 a short burst of 502s stopped work outright: the integration
tier at 16:39 and an e2e conversation at 20:43, where the item waited on
the operator after one refused call.

Not retried: a 500 (it may be the request's fault), a timeout (it already
had its request_timeout_s), or a budget refusal. Rate limits keep their
single 20-second retry. The time spent waiting is reported with the rate
limit's (last_rate_limit_wait_s), so it isn't counted as the model's.

Tests: an outage waited out, one that outlasts the backoff, a dropped
connection, a 500 and a timeout not retried, a budget refusal not waited
on. The offline agent in test_agent_build skips the backoff (its proxy
doesn't exist).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 21:11:45 +00:00
dongarandClaude Opus 5.5 6af54790a9 CI: no image build on PRs; images are built only on release
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 12m10s
CI / e2e (pull_request) Successful in 34m0s
The build-only image job on every PR goes: images are built when a
release is published (release.yml), so PRs don't wait on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 21:00:21 +00:00
dongarandClaude Opus 5.5 2766cf1e8e Release: the dev organization's registry, pushed by registry-push
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 6m28s
CI / image (pull_request) Successful in 21s
CI / e2e (pull_request) Canceled after 1m27s
The repository moved to the dev organization: images go to
gitea.dongar.net/dev/nisse-dev, pushed by the registry-push bot with the
organization's REGISTRY_TOKEN.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 20:24:51 +00:00
dongarandClaude Opus 5.5 4d42040a34 Release: build on a published release, as in inlaw_outlaw
- on: release (published), not on any v* tag: only a deliberate release
  builds an image; the version is its tag without the v, plus latest.
- docker/login-action, setup-buildx and build-push-action, as the runner
  already serves them for inlaw_outlaw; the version as a build argument
  and image label.
- Comments on why the registry needs a personal token (the job token's
  empty package scope) and that packages belong to the repository owner.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 20:14:56 +00:00
dongarandClaude Opus 5.5 0660fa3c51 The agent as a container image (M1.5)
CI / checks (push) Skipped
CI / e2e (pull_request) Failing after 49m48s
CI / checks (pull_request) Successful in 10m0s
CI / image (pull_request) Successful in 27s
- Dockerfile: built with uv into /opt/nisse (installed, not editable),
  run on python:3.12-slim with openssh-client as user nisse; stateless,
  with state in a /var/lib/nisse-dev volume, config mounted at
  /etc/nisse-dev/nisse-dev.yaml, secrets as environment variables, keys
  mounted read-only. 202 MB. Its `test` stage adds the tests and their
  tools over the installed package (UV_NO_SYNC, so `uv run` never swaps
  in the source tree); the default target is the agent.
- scripts/e2e.sh --image REF runs the suite inside an image's test stage,
  on its own sshd.
- CI builds both images on every PR (nothing pushed); release.yml pushes
  gitea.dongar.net/dongar/nisse-dev:<tag> and :latest on a v* tag, with
  the REGISTRY_TOKEN secret.
- scripts/bringup.sh builds the code under test into an image and runs it
  in the systemd unit (root, $E2E mounted at its own path); `plan` prints
  what it would run; the build-VM set-aside reads the projects: config.

Checked: the image refuses to start without a config, runs as uid 10001;
one e2e test passed from the test stage (2m45s, $0).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 19:50:59 +00:00
dev-agent 2d6aa98f87 Merge pull request 'Conversation on the new loop (M2.7)' (#114) from conversation into main
CI / checks (push) Successful in 7m0s
CI / e2e (push) Skipped
2026-10-07 19:20:35 +00:00
dongarandClaude Opus 5.5 8994b3fb4b Conversation on the new loop (M2.7)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 13m26s
CI / e2e (pull_request) Successful in 38m44s
- Questions never change state (L5, #77): a message ending in "?" to a
  waiting project item starts a question run (reply and look-only tools,
  an instruction to change nothing), and the item goes back to the very
  wait it was in: a restatement up stays up and a go binds to it; the
  agent's own question stays open.
- Explanations: a restatement gives its reasons and the main alternative;
  a revision says what changed (restate(changed), sent back without it);
  a milestone's summary says what was done, verified and left open.
- Progress in prose (#42): the driver's and a helper's own words for a
  turn are posted, trimmed, at most once a minute per item; `details`
  shows the last tool calls; a message arriving mid-run is acknowledged at
  once ("Noted; ...").
- CI: the e2e job's comments and the plan no longer claim a newer push
  cancels the run it supersedes (Gitea here doesn't); the concurrency
  block goes.

Tests: conversation unit tests (questions in a go-ahead and an answer wait,
an unanswerable question, revisions, progress rate and trimming, details,
mid-run notes), the helper's progress, the walk asking questions in every
state with a questions_change_nothing property (routing questions the old
way fails 32 of 48 seeds), and an e2e question while a restatement is up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 18:01:51 +00:00
dev-agent 5967d022b6 Merge pull request 'Roles, capability profiles and help on the new loop (M2.6)' (#113) from roles into main
CI / checks (push) Successful in 6m6s
CI / e2e (push) Skipped
2026-10-07 17:40:45 +00:00
dongarandClaude Opus 5.5 ba001a9cec CI: run the e2e job only after the checks pass
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 5m48s
CI / e2e (pull_request) Successful in 21m52s
A PR that's already red doesn't spend ~25 minutes of the RTX 4090.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 16:40:49 +00:00
dongarandClaude Opus 5.5 59fd00ce8c Roles, capability profiles and help on the new loop (M2.6)
CI / checks (push) Skipped
CI / checks (pull_request) Failing after 5m23s
CI / e2e (pull_request) Canceled after 21m42s
- roles.driver (unset: the planner's model) and roles.classifier (unset:
  the expert's); the mapping is pinned when an item opens, and the driver
  and the go-ahead reader run on the item's own (L9).
- profiles: in configuration (tools, strengths, weaknesses), used in
  briefings: the driver's instructions say what its helper and expert are
  good at, the helper's brief who reads its handback; never a model name.
- delegate(task) and ask_expert(question) for the driver. A helper works
  on a branch off the milestone's; on a handback its work comes back as
  uncommitted changes, otherwise it is discarded, so harness branches
  never hold commits only the clone has. ask_expert keeps its budget.
- Triggers: failure_limit failed commands in a row (Driver's new "stuck"
  outcome) or the turn cap, mid-milestone, call a helper in once per
  milestone; if it fails too, the item waits on the operator. A
  delegation cut off by a crash is discarded at boot.
- Evaluation capture: delegations and ask_expert as before, and the
  reader's verdicts as classifier pairs.

Tests: help tests (merge-back, discard, needs a milestone, the stuck
trigger and its once-per-milestone limit, the expert and its budget,
pinning, briefings by profile, boot discarding a cut-off delegation),
config tests for roles and profiles, and the walk delegating with a
scripted helper that sometimes fails, with a delegations_settle property.
The walk found helper commits stranded on a merged PR's branch, fixed by
returning a helper's work uncommitted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 16:33:26 +00:00
dev-agent 04c352b30c Merge pull request 'New projects from conversation (M2.5, PR B)' (#108) from new-projects into main
CI / e2e (push) Skipped
CI / checks (push) Successful in 5m6s
2026-10-07 16:25:41 +00:00
dongarandClaude Opus 5.5 88c7d2e8c9 A design milestone holds documents only, enforced
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 5m4s
CI / e2e (pull_request) Successful in 24m22s
Found by the first local e2e run: qwen3.8-27b put a new project's code in
the same PR as its ARCHITECTURE.md, with no build plan. D1 says a `design`
item agrees its design before any code, and that rested on wording only.

- Until a PR of a `design` item has merged, finish_milestone is sent back
  if the milestone changes anything but ARCHITECTURE.md and docs/, naming
  the paths to take out. The merge is recorded (design_merged) and code
  may follow.
- The label's instruction and the new project's brief say so plainly:
  documents first, in a PR of their own; code after it has merged.

Tests: a design milestone with code is sent back and only the documents
are published, then code after the merge; the walk's model follows the
rule.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 15:39:00 +00:00
dongarandClaude Opus 5.5 ad2d090143 New projects from conversation (M2.5, PR B)
- propose_project(name, purpose, decisions) in a conversation goes up for
  a go-ahead (checked: a slug, not an existing project, no trailing
  question).
- On go, the provisioning checklist (D1): a repository with a README,
  protection on its default branch, the agent as a collaborator, a build
  VM, a deploy key, and the project entry as YAML with <...> to fill in.
- On "done", the agent reads its configuration again and adds the new
  project live (build_loop_agent(load=...); only additions), then checks
  what was provided (D4, verified, not assumed): permissions and
  protection as it sees them, then on a throwaway branch with an empty
  commit, a branch push must succeed and a push to the default branch and
  merging its own PR must be refused. The check's PR is closed and its
  branch deleted. Failures are said per item; nothing starts.
- Checked: the tracking issue (the distilled decisions, carrying the new
  item's ID), and the project's item, agreed and labelled `design`, whose
  first milestone is ARCHITECTURE.md and the build plan. A conversation
  waiting on provisioning doesn't close as idle.
- The walk replays exactly from its seed (item IDs too), and event IDs
  stay unique across simulated restarts, as a server's are.
- The test config holds output to 8192 tokens, under the test key's
  per-request cost cap.

Tests: new-project unit tests (checklist and its YAML parse as config, done
before the entry, what isn't set up, a checked project's record and first
PR, an existing name), protection enforced by a hook on the fake forge,
the walk proposing projects and saying done, and an e2e new project in a
repository the test operator provisions from the agent's checklist.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 15:39:00 +00:00
dongar 9a86ccb2a4 Merge pull request 'End-to-end tests on a local model, and in CI on every PR' (#110) from e2e-local-model into main
CI / e2e (push) Skipped
CI / checks (push) Successful in 5m26s
Reviewed-on: dongar/nisse#110
Reviewed-by: Don Garrett <dgarrett@acm.org>
2026-10-07 15:38:45 +00:00
dongarandClaude Opus 5.5 6db80dfb4b End-to-end tests on a local model, and in CI on every PR
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 5m26s
CI / e2e (pull_request) Successful in 17m27s
- Every e2e role runs on qwen3.8-27b on the RTX 4090 (e2e_roles();
  NISSE_E2E_MODEL overrides, e.g. claude-haiku-4-5). The first full run
  passed 7 of 8 in 26 minutes for $0.0000; the eighth was a real finding,
  fixed with the new-project PR.
- scripts/e2e.sh reports a run's spend on the test key and the day's total
  against its budget: above $0 on the local model means a paid fallback.
- CI runs the suite as its own job on pull requests (not pushes to main),
  on a throwaway sshd (--local), cancelling a run a newer push supersedes,
  so the checks job still reports in minutes.
- The plan says so: e2e on every PR; locally only -k / --lf.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-07 15:14:43 +00:00
dev-agent b3bf6d5a6e Merge pull request 'A repository per test run instead of the shared sandbox' (#107) from fresh-test-repos into main
CI / checks (push) Successful in 5m30s
2026-10-06 05:26:00 +00:00
dongarandClaude Opus 5.5 679ace1fd8 A repository per test run instead of the shared sandbox
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 5m17s
Every live test that touches a repository gets one of its own from
e2e_support.fresh_repo, made by the test operator as an operator makes a
project's (D4, D8): seeded from tests/integration/seed/ (the sandbox's
package, without past runs' leftovers), the agent's account a collaborator
with write access, main protected (pushes only by the owner, an approval to
merge), and a deploy key generated for the run. It is deleted afterwards
(NISSE_E2E_KEEP_REPOS keeps it); scripts/e2e-cleanup.sh deletes leftovers.

- Both e2e fixtures use it; per-run base branches are gone. The merge
  helper approves first, as the protection now requires.
- test_gitea_live uses it, so CI needs NISSE_OPERATOR_TOKEN and no longer
  writes the sandbox deploy key (NISSE_PUSH_KEY).
- A live test of the helper: seeded, shared with the agent, protected
  against it, deleted.
- nisse-test.yaml's project entry is a placeholder; pytest and ruff skip
  the seed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-06 05:10:27 +00:00
dev-agent 6c962e1ed9 Merge pull request 'Conversations on the new loop (M2.5, PR A)' (#106) from loop-conversations into main
CI / checks (push) Successful in 5m5s
2026-10-06 04:33:05 +00:00
dongarandClaude Opus 5.5 3f54896f65 Conversations on the new loop (M2.5, PR A)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 5m14s
- A top-level message that isn't a command is a conversation: an item in
  the `general` scope (L1), threaded under the operator's message, with
  `reply`, `ask_human` and `propose_work` and no workspace or execution.
  `general` is reserved as a project name.
- propose_work(project, title, summary) goes up as a revision for a
  go-ahead (or a plain yes). On go, under a file_issue intent (L8), the
  record issue is opened in that project, assigned to the agent and
  carrying the new item's ID; its item starts agreed, with the summary as
  its brief. Intake skips any issue carrying an ID (D1); boot adopts a
  filed issue or puts the proposal up again.
- A conversation closes after tuning.loop.general_idle_s (default a day);
  a reply in its thread then starts a new conversation that continues it.

Tests: conversation unit tests (threading, filing, the marker, an unknown
project, changes, idle close and continuation, a plain yes, both crash
cases, commands at top level); the walk with conversations, proposals and
idle time, and records_have_one_item / conversations_do_no_work; an e2e
conversation that becomes filed work.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-06 04:26:40 +00:00
dev-agent d70d43a8cc Merge pull request 'Projects as configuration entries (M1.4)' (#105) from projects into main
CI / checks (push) Successful in 5m14s
2026-10-06 03:42:06 +00:00
dongarandClaude Opus 5.5 6aa1439a7f Entry-point test: a config with no projects is named in the startup errors
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 5m28s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-06 02:36:06 +00:00
dongarandClaude Opus 5.5 064d9bf554 Projects as configuration entries (M1.4, D8)
CI / checks (push) Skipped
CI / checks (pull_request) Failing after 5m7s
- Config: `projects: {name: entry}` with repo, base_branch, workspace,
  build_host, push credential, labels (defaults added to every issue),
  secrets by ${VAR} reference with visibility (commands | harness), and
  egress (recorded, not enforced). Every problem in every entry is
  reported at once; an inline secret, a bad visibility, a repeated repo or
  a non-slug name is an error.
- The old loop runs exactly one project (only_project) and refuses more.
- The new loop is per project: each its own clone, workspace (with its
  commands-visible secrets, redacted on return), push credential and
  milestones; intake polls every project's repository; items are scoped
  to their project for every run, publish and PR watch. Issues are named
  project#14 when there is more than one project, and a bare #14 that
  matches two is asked about. Switching projects parks the one worked on
  last (all others after a boot).
- A restatement that ends in a question is sent back to the driver, not
  posted: the e2e driver kept asking "Is this the right interpretation?",
  which a plain yes can't answer.

Tests: config entries and errors, two-project assembly, two-project loop
tests (intake, own workspaces and repos, #14 in both, do-next, label
defaults, parking across projects), the property walk over two projects
with a new own_repository_only property (an item using the first
project's repo fails 15 of 16 seeds), and the restatement check.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-06 01:12:57 +00:00
53 changed files with 4823 additions and 471 deletions

No files matched your search

+9
View File
@@ -0,0 +1,9 @@
.git
.env
**/.venv
**/__pycache__
**/.pytest_cache
**/.ruff_cache
_apalache-out
tmp
bench
+43 -9
View File
@@ -42,15 +42,6 @@ jobs:
- name: Unit tests - name: Unit tests
run: uv run pytest -m "not integration and not e2e" run: uv run pytest -m "not integration and not e2e"
- name: Push key
# The sandbox's deploy key, where nisse-test.yaml expects it.
env:
NISSE_PUSH_KEY: ${{ secrets.NISSE_PUSH_KEY }}
run: |
mkdir -p /tmp/nisse-test
printf '%s\n' "$NISSE_PUSH_KEY" > /tmp/nisse-test/push_key
chmod 600 /tmp/nisse-test/push_key
- name: Integration tests - name: Integration tests
# Everything but the secrets is in tests/integration/nisse-test.yaml, # Everything but the secrets is in tests/integration/nisse-test.yaml,
# which refers to these by name. A missing secret fails the live # which refers to these by name. A missing secret fails the live
@@ -59,4 +50,47 @@ jobs:
NISSE_LLM_TOKEN: ${{ secrets.NISSE_LLM_TOKEN }} NISSE_LLM_TOKEN: ${{ secrets.NISSE_LLM_TOKEN }}
NISSE_MATRIX_TOKEN: ${{ secrets.NISSE_MATRIX_TOKEN }} NISSE_MATRIX_TOKEN: ${{ secrets.NISSE_MATRIX_TOKEN }}
NISSE_GITEA_TOKEN: ${{ secrets.NISSE_GITEA_TOKEN }} NISSE_GITEA_TOKEN: ${{ secrets.NISSE_GITEA_TOKEN }}
# The test operator, who makes each run's repository (and its deploy
# key) and deletes it afterwards.
NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }}
run: uv run pytest -m "integration and not e2e" -rs run: uv run pytest -m "integration and not e2e" -rs
e2e:
# The end-to-end suite (scripts/e2e.sh): the whole agent against real
# services, every role on a local model, so it spends nothing but keeps
# the RTX 4090 busy for ~25 minutes. On PRs only, and only after the
# checks pass: a PR that's red already doesn't spend the GPU. There is
# one runner and Gitea doesn't honour cancel-in-progress, so a run a
# newer push supersedes holds the runner until it's cancelled by hand.
# Locally, run only what you're writing (-k) or diagnosing (--lf).
if: github.event_name != 'push'
needs: checks
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
- name: Install uv
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Sync workspace
run: uv sync --all-packages
- name: Install sshd
# The build VM is a throwaway sshd here (scripts/e2e.sh --local).
run: |
SUDO=$(command -v sudo || true)
$SUDO apt-get update -qq
$SUDO apt-get install -y -qq openssh-server openssh-client > /dev/null
$SUDO mkdir -p /run/sshd
- name: End-to-end tests
env:
NISSE_LLM_TOKEN: ${{ secrets.NISSE_LLM_TOKEN }}
NISSE_MATRIX_TOKEN: ${{ secrets.NISSE_MATRIX_TOKEN }}
NISSE_GITEA_TOKEN: ${{ secrets.NISSE_GITEA_TOKEN }}
NISSE_OPERATOR_TOKEN: ${{ secrets.NISSE_OPERATOR_TOKEN }}
NISSE_E2E_RUNS: /tmp/nisse-e2e-runs
run: scripts/e2e.sh --local -v
+63
View File
@@ -0,0 +1,63 @@
name: Release
# A published release builds the agent's image and pushes it to Gitea's
# registry (Development Plan M1.5, X4): gitea.dongar.net/dev/nisse-dev
# tagged with the version (the release tag without its v) and latest. Cut a
# release in Gitea (Releases → New release, tag vX.Y.Z); a tag pushed for any
# other reason builds nothing.
#
# The registry needs a personal access token with write:package
# (REGISTRY_TOKEN, an organization secret of `dev`, belonging to the
# registry-push bot), never the built-in job token: Gitea's registry
# resolves an Actions job token to an empty package scope (go-gitea/gitea
# #23642 and others). Packages belong to the repository's owner, which is
# why this repository is the `dev` organization's: the bot writes packages
# through the org's registry-push team. Never on pull_request: registry
# credentials stay off that trigger.
on:
release:
types: [published]
env:
REGISTRY: gitea.dongar.net
REGISTRY_USER: registry-push # the token's account
IMAGE: gitea.dongar.net/dev/nisse-dev
jobs:
build-and-push:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Derive the image version from the release's tag
id: vars
env:
TAG: ${{ github.event.release.tag_name }}
run: echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
- name: Log in to Gitea's container registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ env.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and push the agent
uses: docker/build-push-action@v5
with:
context: .
target: agent
push: true
build-args: |
VERSION=${{ steps.vars.outputs.version }}
labels: |
org.opencontainers.image.version=${{ steps.vars.outputs.version }}
org.opencontainers.image.revision=${{ github.sha }}
tags: |
${{ env.IMAGE }}:${{ steps.vars.outputs.version }}
${{ env.IMAGE }}:latest
+68
View File
@@ -0,0 +1,68 @@
# The nisse-dev agent as a container image (Development Plan M1.5, X4).
#
# Stateless: the state store and SSH known-hosts live in a volume
# (/var/lib/nisse-dev); configuration is mounted read-only
# (/etc/nisse-dev/nisse-dev.yaml); secrets come as environment variables
# (docker --env-file); SSH and deploy keys are mounted read-only wherever the
# configuration says. The agent runs no model-chosen command here (L13):
# those go to build VMs over SSH.
#
# docker build -t nisse-dev . the agent
# docker build --target test -t nisse-dev:test . plus the tests, for
# scripts/e2e.sh --image
FROM python:3.12-slim AS build
COPY --from=ghcr.io/astral-sh/uv:0.12.10 /uv /usr/local/bin/uv
ENV UV_PROJECT_ENVIRONMENT=/opt/nisse UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
WORKDIR /src
COPY pyproject.toml uv.lock ./
COPY nisse-core nisse-core
COPY nisse-dev nisse-dev
COPY nisse-ops nisse-ops
# The agent and what it needs, installed (not editable): the source tree
# isn't part of the image.
RUN uv sync --frozen --no-dev --package nisse-dev --no-editable
FROM python:3.12-slim AS runtime
ARG VERSION=dev
LABEL org.opencontainers.image.title=nisse-dev org.opencontainers.image.version=$VERSION
RUN apt-get update \
&& apt-get install -y --no-install-recommends openssh-client \
&& rm -rf /var/lib/apt/lists/* \
&& useradd --create-home --uid 10001 nisse \
&& mkdir -p /var/lib/nisse-dev /etc/nisse-dev \
&& chown nisse /var/lib/nisse-dev
COPY --from=build /opt/nisse /opt/nisse
ENV PATH=/opt/nisse/bin:$PATH \
NISSE_CONFIG=/etc/nisse-dev/nisse-dev.yaml \
NISSE_ENV=/run/nisse-dev/.env
VOLUME /var/lib/nisse-dev
USER nisse
ENTRYPOINT ["nisse-dev"]
# The image's own package plus the tests, so the end-to-end suite runs
# against what was built, not the source tree.
FROM runtime AS test
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends git openssh-server \
&& rm -rf /var/lib/apt/lists/* \
&& mkdir -p /run/sshd
COPY --from=build /usr/local/bin/uv /usr/local/bin/uv
WORKDIR /src
COPY pyproject.toml uv.lock ./
COPY nisse-core nisse-core
COPY nisse-dev nisse-dev
COPY nisse-ops nisse-ops
COPY tests tests
COPY scripts scripts
COPY conftest.py ./
ENV UV_PROJECT_ENVIRONMENT=/opt/nisse
# Test tools too, all installed (not editable), and `uv run` never re-syncs:
# the tests import the package as built, not the source tree beside them.
RUN uv sync --frozen --all-packages --no-editable --inexact
ENV UV_NO_SYNC=1
ENTRYPOINT []
# The default target: the agent itself.
FROM runtime AS agent
+91 -15
View File
@@ -31,7 +31,7 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
| **L6** Context | partial | Context fitting, per-role views and orchestrator facts (M1.2); no scope or agent memory | | **L6** Context | partial | Context fitting, per-role views and orchestrator facts (M1.2); no scope or agent memory |
| **L7** Canonical log | partial | Work item and actor on every event, `fact` (M1.1, M1.2); no `classifier_verdict`, `knowledge_retrieved` | | **L7** Canonical log | partial | Work item and actor on every event, `fact` (M1.1, M1.2); no `classifier_verdict`, `knowledge_retrieved` |
| **L8** Durability | mostly done | WAL store, intents, boot reconciliation, PR adoption, comment redelivery; to move onto the new loop | | **L8** Durability | mostly done | WAL store, intents, boot reconciliation, PR adoption, comment redelivery; to move onto the new loop |
| **L9** Models and roles | partial | LiteLLM, delegation, `ask_expert`, eval capture; roles are still planner/executor, no capability profiles | | **L9** Models and roles | mostly done | LiteLLM, eval capture; on the new loop: `driver`/`helper`/`expert`/`classifier` pinned per item, capability profiles in briefings, delegation and `ask_expert` with triggers (M2.6) |
| **L10** Knowledge vault | none | | | **L10** Knowledge vault | none | |
| **L11** Injection | partial | Operator allowlists on every channel; no classifier | | **L11** Injection | partial | Operator allowlists on every channel; no classifier |
| **L12** Secrets, logging | partial | Structured state-change log; secrets in `.env`; no per-operation handout or redaction | | **L12** Secrets, logging | partial | Structured state-change log; secrets in `.env`; no per-operation handout or redaction |
@@ -39,7 +39,7 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
| **D1–D2** Workflow | partial | On the new loop: issue intake, restatement and go-ahead (M2.1), labels (M2.3); no documents in the repo | | **D1–D2** Workflow | partial | On the new loop: issue intake, restatement and go-ahead (M2.1), labels (M2.3); no documents in the repo |
| **D3–D4** Workspace, git | partial | Idempotent publishing; no fresh workspace (#62), `git add -A` (#80) | | **D3–D4** Workspace, git | partial | Idempotent publishing; no fresh workspace (#62), `git add -A` (#80) |
| **D5** Review | partial | PR conversation comments only (#69), watch stall (#70) | | **D5** Review | partial | PR conversation comments only (#69), watch stall (#70) |
| **D7–D8** Deployment, projects | none | Runs from a venv as a systemd unit on this box; one project in config | | **D7–D8** Deployment, projects | partial | Runs from a venv as a systemd unit on this box; project entries in config (M1.4); egress not enforced |
| **O**, **X** | none | No container image yet | | **O**, **X** | none | No container image yet |
| `specs/` | done for now | `orchestrator.qnt` (current loop), `loop.qnt` (the architecture, #91) | | `specs/` | done for now | `orchestrator.qnt` (current loop), `loop.qnt` (the architecture, #91) |
@@ -54,8 +54,14 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
- **Models by role today:** executor `qwen3.8-27b`, planner and helper - **Models by role today:** executor `qwen3.8-27b`, planner and helper
`claude-opus-5-5`, expert `claude-haiku-4-5`. Phase 2 moves to driver and helpers. `claude-opus-5-5`, expert `claude-haiku-4-5`. Phase 2 moves to driver and helpers.
- **Bring-up** runs on this development VM (`scripts/bringup.sh`), against test - **Bring-up** runs on this development VM (`scripts/bringup.sh`), against test
resources: the sandbox repository, the test Matrix room, a budget-capped LiteLLM resources: a test repository, the test Matrix room, a budget-capped LiteLLM key;
key; configuration in `/root/nisse-e2e/`. configuration in `/root/nisse-e2e/`.
- **Test repositories are made per run** by the test operator
(`nisse-testing-operator`, `NISSE_OPERATOR_TOKEN`), as an operator provides a
project's: seeded from `tests/integration/seed/`, the agent a collaborator,
`main` protected, a deploy key of its own; deleted afterwards
(`scripts/e2e-cleanup.sh` for leftovers). They replaced the shared sandbox
repository, whose state accumulated across runs.
- **A build VM for bring-up** is provided by the operator by hand until Terraform - **A build VM for bring-up** is provided by the operator by hand until Terraform
(Phase 6); the agent reaches it over SSH with a key from `/root/nisse-e2e/ssh/`. (Phase 6); the agent reaches it over SSH with a key from `/root/nisse-e2e/ssh/`.
@@ -67,9 +73,12 @@ Only `nisse-dev` has real code; `nisse-core` and `nisse-ops` are placeholders.
`CLAUDE.md`. A milestone marked **needs discussion** starts with a design `CLAUDE.md`. A milestone marked **needs discussion** starts with a design
discussion first. discussion first.
- **Every change comes with tests,** through `scripts/check.sh`. - **Every change comes with tests,** through `scripts/check.sh`.
- **The end-to-end test follows each milestone** (M1.6): the real agent, real models, - **The end-to-end test runs on every PR** (M1.6): the real agent, every role on a
the Gitea sandbox and a real build host, driven by a scripted operator. It runs local model (so it costs nothing per call), a repository of the run's own and a
after each milestone, not on every push, since it spends model tokens. Manual real build host, driven by a scripted operator. CI runs it as its own job on pull
requests, after the checks pass. There is one runner, and a run a newer push
supersedes has to be cancelled by hand. Locally only the tests being written
(`-k`) or diagnosed (`--lf`); `scripts/e2e.sh` reports what a run spent. Manual
live runs are for the new loop's first runs and phase ends. Findings are filed as live runs are for the new loop's first runs and phase ends. Findings are filed as
issues and either fixed before moving on or deliberately deferred. issues and either fixed before moving on or deliberately deferred.
- **The Quint models move with the loop:** `orchestrator.qnt` while `control.py` - **The Quint models move with the loop:** `orchestrator.qnt` while `control.py`
@@ -184,31 +193,98 @@ between milestones (D2, D4). End-to-end: a two-milestone feature.
- Deferred: a milestone with no changes that wasn't expected to change code is - Deferred: a milestone with no changes that wasn't expected to change code is
still raised with the operator (D4). still raised with the operator (D4).
**M2.5 Conversations and new projects.** *Needs discussion;* the design was agreed **M2.5 Conversations and new projects.** **Done** (#106 and this PR). *Needs discussion;* the design was agreed
in this PR (`general` scope, L1, L5; the new-project sequence, D1). Work started in in this PR (`general` scope, L1, L5; the new-project sequence, D1). Work started in
conversation, record issues carrying their item's ID, the new-project flow and its conversation, record issues carrying their item's ID, the new-project flow and its
project entry (D1, D8, #29). After M1.4. End-to-end: a small new project, in a project entry (D1, D8, #29). After M1.4. End-to-end: a small new project, in a
repository the test operator creates under its own account for the run and repository the test operator creates under its own account for the run and
deletes afterwards. deletes afterwards.
- **PR A, conversations — done (this PR).**
- A top-level message that isn't a command is a `general` item, threaded under
it; the driver gets `reply`, `ask_human` and `propose_work`, with no workspace.
- Agreed work (a go-ahead, or a plain yes, on the proposal) is filed under an
intent as a record issue carrying the new item's ID. Its item starts agreed,
with no second restatement. Intake skips any issue carrying an ID, and boot
settles a filing cut off by a crash.
- A conversation closes after `tuning.loop.general_idle_s` without a message; a
reply in its thread then starts a new one that continues it.
- **PR B, new projects — done (this PR).**
- `propose_project(name, purpose, decisions)` goes up for a go-ahead.
- On go, the checklist: repository, protection, collaborator, build VM, deploy
key, and the project entry as YAML to fill in.
- On "done", the agent reads its configuration again and adds the new project
live (only additions). It then checks the access: the repository's
permissions and protection as the agent sees them, then in practice on a
throwaway branch with an empty commit (a branch push must succeed, a push to
the default branch and merging its own PR must be refused); the branch and PR
are removed. Anything that fails is said, and nothing starts.
- When all is well: the tracking issue with the distilled decisions, and the
project's item, agreed and labelled `design`, whose first milestone is
`ARCHITECTURE.md` and the build plan. A conversation waiting on provisioning
doesn't close as idle.
**M2.6 Roles and capability profiles.** *Pre-approved.* `driver`, `helper`, **M2.6 Roles and capability profiles.** *Pre-approved.* **Done** (this PR).
`expert`, `classifier` (L9); escalation triggers; delegation and `ask_expert` on the `driver`, `helper`, `expert`, `classifier` (L9); escalation triggers; delegation and
new loop. `ask_expert` on the new loop.
- `roles.driver` (unset: the planner's model) and `roles.classifier` (unset: the
expert's), pinned per item; the driver and the go-ahead reader run on the item's.
- `profiles:` in configuration (tools, strengths, weaknesses), used in briefings:
the driver is told what its helper and expert are good at, the helper who will
read its handback; never a model's name.
- `delegate(task)` and `ask_expert(question)` for the driver. A helper works on a
branch off the milestone's; its work comes back as uncommitted changes on a
handback and is discarded otherwise, so harness branches never hold commits only
the clone has (the walk found a fix stranded that way when a PR merged mid-help).
- Triggers: `failure_limit` failed commands in a row, or the turn cap, mid-
milestone, call a helper in once per milestone; if it fails too, the item waits
on the operator as before. A delegation cut off by a crash is discarded at boot.
- Evaluation capture: delegations, `ask_expert` and the reader's verdicts.
- Deferred: the context-won't-fit trigger (L6); the action classifier (M3.2).
**M2.7 Conversation.** *Pre-approved.* Explanations, progress in prose, asking **M2.7 Conversation.** *Pre-approved.* **Done** (this PR). Explanations, progress
mid-work (L5, #42, #77). in prose, asking mid-work (L5, #42, #77).
- Questions never change state: a message ending in `?` to a waiting project item
is answered in a question run (`reply` and look-only tools), and the item goes
back to the very wait it was in, so a restatement up stays up and a go binds to
it.
- A restatement gives its reasons and the main alternative; a revision says what
changed (`restate(changed)`, sent back without it).
- Progress: the driver's and a helper's own words for a turn are posted, trimmed,
at most once a minute per item; `details` shows the last tool calls. A message
arriving mid-run is acknowledged at once.
- Deferred: streaming (#42); the question property in `specs/loop.qnt` (the walk
checks it against the code).
**M1.4 Projects** (D8). *Pre-approved.* **M1.4 Projects** (D8). *Pre-approved.* **Done** (this PR).
- The project entry in configuration: repository, default branch, build VM, label - The project entry in configuration: repository, default branch, build VM, label
defaults, secrets by name with their visibility, egress additions. defaults, secrets by name with their visibility, egress additions.
- Acceptance: the bring-up configuration is one project entry; a second project - Acceptance: the bring-up configuration is one project entry; a second project
needs only a second entry. needs only a second entry.
- `projects: {name: entry}`. The old loop runs exactly one project and refuses
more. On the new loop each project has its own clone, workspace, push credential
and milestones. Issues are named `project#14` when two projects could both have
`#14`, and a bare `#14` that matches two is asked about. Switching projects parks
the one worked on last.
- Secrets are by `${VAR}` reference only, `commands` or `harness`. `commands`
secrets reach `run_command` as environment variables, redacted from output;
`harness` secrets are held for later. Egress is recorded, not enforced (D7).
- Deferred: config reload without a restart (with M2.5's entry step if wanted);
the trust dial's configuration (M3.3).
**M1.5 The agent as a container image.** *Pre-approved.* **M1.5 The agent as a container image.** *Pre-approved.* **Built** (this PR);
accepted once a tag's image passes the e2e suite.
- A Dockerfile for `nisse-dev`, stateless with a volume for state; a Gitea Actions - A Dockerfile for `nisse-dev`, stateless with a volume for state; a Gitea Actions
workflow that builds it and pushes to Gitea's registry on each release tag (X4). workflow that builds it and pushes to Gitea's registry on each release tag (X4).
- `bringup.sh` runs the image instead of a venv. - `bringup.sh` runs the image instead of a venv.
- Acceptance: a tagged release produces an image; the end-to-end test runs from it. - Acceptance: a tagged release produces an image; the end-to-end test runs from it.
- The image runs as its own user; state in `/var/lib/nisse-dev`, configuration
mounted at `/etc/nisse-dev/nisse-dev.yaml`, secrets as environment variables,
keys mounted read-only. Its `test` stage carries the tests over the installed
package, for `scripts/e2e.sh --image REF`. Images are built only on release:
`release.yml` pushes `gitea.dongar.net/dev/nisse-dev:<version>` and `:latest`
on a published release (`REGISTRY_TOKEN`, the `dev` organization's).
- The live bring-up's switch to the image waits for the next live test (it needs
its own test repository too); `bringup.sh plan` shows what it will run.
**M2.8 Switch over.** *Pre-approved, after acceptance.* `nisse-dev` runs the new **M2.8 Switch over.** *Pre-approved, after acceptance.* `nisse-dev` runs the new
loop; `control.py`, its tests and `orchestrator.qnt` are removed. loop; `control.py`, its tests and `orchestrator.qnt` are removed.
+10
View File
@@ -526,3 +526,13 @@ thread stops the countdown, and `go` starts at once.
- **Deciding:** about the time it takes you to notice a message and read it - **Deciding:** about the time it takes you to notice a message and read it
when you're around. A wrong start costs only time, since the fix still comes when you're around. A wrong start costs only time, since the fix still comes
as a PR you review. as a PR you review.
### `tuning.loop.general_idle_s` — default 86400
How long a `general` conversation (one not about a project, architecture L1)
stays open without a message. It then closes with a note in its log; a later
reply in its thread starts a new conversation linked to it.
- **Too low:** a conversation you come back to after lunch has to be picked up
again from its summary.
- **Too high:** idle conversations pile up in `status` and in the queue.
+36 -6
View File
@@ -9,15 +9,26 @@
# silently doing nothing. # silently doing nothing.
state_db: ~/nisse-demo-workspace/state.db state_db: ~/nisse-demo-workspace/state.db
workspace: /root/work/demo # the clone, on the build host
project: demo
# loop: new # the new conversational loop (M2.1); default old (control.py) # loop: new # the new conversational loop (M2.1); default old (control.py)
build_host: # L13: every command runs here, over SSH # D8: one entry per project. A second project is a second entry (the old loop
# runs exactly one). Changed by PR like the rest of this file.
projects:
demo:
repo: owner/demo-repo
base_branch: main
workspace: /root/work/demo # the clone, on the build host
build_host: # L13: every command runs here, over SSH
address: build-demo.example address: build-demo.example
key: ~/.ssh/nisse-dev # the agent's key for the build host key: ~/.ssh/nisse-dev # the agent's key for the build host
# user: root # user: root
# port: 22 # port: 22
push_key: ~/.ssh/push-demo-repo # a deploy key with write access (D4)
# labels: [discuss] # added to every issue's own (D1)
# secrets: # by name; values only by reference (D7)
# DATABASE_URL: {value: "${DEMO_DATABASE_URL}", visible: commands}
# DEPLOY_TOKEN: {value: "${DEMO_DEPLOY_TOKEN}", visible: harness}
# egress: [pypi.org] # beyond the default; recorded, not yet enforced
llm: # L9: LiteLLM is the only LLM destination llm: # L9: LiteLLM is the only LLM destination
base_url: http://litellm.example:4000 base_url: http://litellm.example:4000
@@ -34,6 +45,27 @@ roles: # L9: role -> LiteLLM model alias
planner: claude-opus-5-5 # high-end judgement: plans a weaker executor follows planner: claude-opus-5-5 # high-end judgement: plans a weaker executor follows
helper: claude-opus-5-5 # takes over what the executor got stuck on helper: claude-opus-5-5 # takes over what the executor got stuck on
expert: claude-haiku-4-5 # one question at a time, no tools: cheap is enough expert: claude-haiku-4-5 # one question at a time, no tools: cheap is enough
# The new loop's roles (loop: new): unset, driver is the planner's model and
# classifier the expert's. Each work item pins the mapping when it opens.
# driver: claude-opus-5-5 # drives the loop: talks, restates, works
# classifier: claude-haiku-4-5 # reads plain-language go-aheads (later: actions, L4)
# Capability profiles (L9), from benchmarks: what one role is told about
# another (never a model's name). Optional; a model without one is briefed as
# "no profile recorded".
profiles:
qwen3.8-27b:
tools: reliable (37/40 on the executor benchmark, PR #55)
strengths: careful multi-step edits that follow a clear brief
weaknesses: speed (minutes per hard task) and open-ended design
claude-haiku-4-5:
tools: reliable
strengths: quick, well-scoped changes and short answers
weaknesses: long multi-file work (15/20 on the harder benchmark tasks)
claude-opus-5-5:
tools: reliable
strengths: design, debugging, and work other models got stuck on
weaknesses: cost; use it for what needs it
matrix: # L5 matrix: # L5
homeserver: https://matrix.example homeserver: https://matrix.example
@@ -45,10 +77,7 @@ matrix: # L5
gitea: # D4, D5 gitea: # D4, D5
url: https://gitea.example url: https://gitea.example
token: ${NISSE_GITEA_TOKEN} token: ${NISSE_GITEA_TOKEN}
repo: owner/demo-repo
operators: [your-gitea-login] operators: [your-gitea-login]
base_branch: main
push_key: ~/.ssh/push-demo-repo # a deploy key with write access (D4)
# What each knob does, and how to choose a value: docs/Tuning-Guide.md. # What each knob does, and how to choose a value: docs/Tuning-Guide.md.
# These are the defaults; the section can be left out entirely. The file is # These are the defaults; the section can be left out entirely. The file is
@@ -71,3 +100,4 @@ tuning:
safety_margin: 0.1 # fraction of the window held back safety_margin: 0.1 # fraction of the window held back
loop: # the new loop (loop: new) loop: # the new loop (loop: new)
bug_grace_s: 600 # a `bug` restatement proceeds after this long unanswered bug_grace_s: 600 # a `bug` restatement proceeds after this long unanswered
general_idle_s: 86400 # a conversation closes after this long without a message
+22 -17
View File
@@ -14,7 +14,7 @@ from dataclasses import dataclass
from nisse_dev.approvals import PlanBoard from nisse_dev.approvals import PlanBoard
from nisse_dev.cancel import Cancel from nisse_dev.cancel import Cancel
from nisse_dev.catalog import ModelCatalog, select_client from nisse_dev.catalog import ModelCatalog, select_client
from nisse_dev.config import AgentConfig from nisse_dev.config import AgentConfig, Project
from nisse_dev.context import BudgetedClient from nisse_dev.context import BudgetedClient
from nisse_dev.control import Orchestrator from nisse_dev.control import Orchestrator
from nisse_dev.delegation import Caps, Delegation from nisse_dev.delegation import Caps, Delegation
@@ -165,30 +165,34 @@ def make_llm_clients(
return normalized, passthrough return normalized, passthrough
def build_host(config: AgentConfig) -> SSHHost: def build_host(config: AgentConfig, project: Project | None = None) -> SSHHost:
"""The tool VM the workspace lives on (L13). Its known-hosts file and """A project's build VM (L13; the only project's, if none is named). Its
connection socket live beside the state store.""" known-hosts file and connection socket live beside the state store."""
host = (project or config.only_project()).build_host
return SSHHost( return SSHHost(
config.build_host.address, host.address,
key=config.build_host.key, key=host.key,
user=config.build_host.user, user=host.user,
port=config.build_host.port, port=host.port,
state_dir=config.state_db.parent / "ssh", state_dir=config.state_db.parent / "ssh",
) )
def push_credential(config: AgentConfig) -> PushCredential | None: def push_credential(
"""What pushes go by: a push-only account's token over HTTPS, or a config: AgentConfig, project: Project | None = None
deploy key over SSH (D4).""" ) -> PushCredential | None:
if config.push_user and config.push_token: """What a project's pushes go by: a push-only account's token over
return HTTPSToken(config.push_user, config.push_token) HTTPS, or a deploy key over SSH (D4)."""
if config.push_key is None: project = project or config.only_project()
if project.push_user and project.push_token:
return HTTPSToken(project.push_user, project.push_token)
if project.push_key is None:
return None return None
try: try:
return SSHKey(config.push_key.read_text()) return SSHKey(project.push_key.read_text())
except OSError as exc: except OSError as exc:
raise RuntimeError( raise RuntimeError(
f"cannot read the push key {config.push_key}: {exc}" f"cannot read the push key {project.push_key}: {exc}"
) from exc ) from exc
@@ -205,6 +209,7 @@ def prepare_workspace(
account: str, account: str,
remote: str, remote: str,
credential: PushCredential | None, credential: PushCredential | None,
project: Project | None = None,
) -> GitRepo: ) -> GitRepo:
"""Ensure the workspace on the build VM is a clone of the project repo. """Ensure the workspace on the build VM is a clone of the project repo.
@@ -213,7 +218,7 @@ def prepare_workspace(
Cloning here rather than when a work item starts is the single-project Cloning here rather than when a work item starts is the single-project
shortcut that M1.4 and D3's fresh workspace replace. shortcut that M1.4 and D3's fresh workspace replace.
""" """
repo = GitRepo(host, config.workspace) repo = GitRepo(host, (project or config.only_project()).workspace)
if not repo.exists(): if not repo.exists():
repo.clone(remote, credential=credential) repo.clone(remote, credential=credential)
repo.configure_identity(account, f"{account}@users.noreply.local") repo.configure_identity(account, f"{account}@users.noreply.local")
+301 -57
View File
@@ -17,7 +17,7 @@ silently ignored would look exactly like a knob that has no effect.
import os import os
import re import re
from collections.abc import Mapping from collections.abc import Mapping
from dataclasses import dataclass, field, fields from dataclasses import dataclass, field, fields, replace
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
@@ -30,8 +30,6 @@ DEFAULT_CONFIG_FILE = "nisse-dev.yaml"
REQUIRED_KEYS = ( REQUIRED_KEYS = (
"state_db", "state_db",
"workspace",
"project",
"llm.base_url", "llm.base_url",
"llm.token", "llm.token",
"roles.executor", "roles.executor",
@@ -45,22 +43,34 @@ REQUIRED_KEYS = (
"matrix.operators", "matrix.operators",
"gitea.url", "gitea.url",
"gitea.token", "gitea.token",
"gitea.repo",
"gitea.operators", "gitea.operators",
"build_host.address",
"build_host.key",
) )
OPTIONAL_KEYS = ( OPTIONAL_KEYS = (
"roles.driver",
"roles.classifier",
"llm.max_output_tokens", "llm.max_output_tokens",
"llm.request_timeout_s", "llm.request_timeout_s",
"gitea.base_branch",
"gitea.push_key",
"gitea.push_user",
"gitea.push_token",
"build_host.user",
"build_host.port",
"loop", "loop",
) )
# A project entry (D8), under `projects: {name: {...}}`.
PROJECT_REQUIRED = ("repo", "workspace", "build_host.address", "build_host.key")
PROJECT_OPTIONAL = (
"base_branch",
"push_key",
"push_user",
"push_token",
"build_host.user",
"build_host.port",
"labels",
"secrets",
"egress",
)
# Who may see a project secret (D7, D8): the model's commands get it as an
# environment variable, redacted from what comes back; harness-only ones are
# never in a command the model chose.
VISIBILITY = ("commands", "harness")
_PROJECT_NAME = re.compile(r"[a-z0-9][a-z0-9-]*")
GENERAL = "general" # the scope of conversation not about a project (L1)
_REFERENCE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}") _REFERENCE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}")
@@ -109,6 +119,8 @@ class LoopTuning:
# A restatement on a `bug` issue proceeds after this long without a # A restatement on a `bug` issue proceeds after this long without a
# reply (D1). # reply (D1).
bug_grace_s: float = 600.0 bug_grace_s: float = 600.0
# A `general` conversation closes after this long without a message (L1).
general_idle_s: float = 86400.0
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -225,6 +237,139 @@ def _read_file(path: Path, errors: list[str]) -> dict[str, Any]:
return loaded return loaded
def _parse_projects(
raw: Any, env: Mapping[str, str], errors: list[str]
) -> tuple["Project", ...]:
"""`projects:`, a mapping of name to entry (D8). Every problem in every
entry is reported, like the rest of the file."""
if not isinstance(raw, dict) or not raw:
errors.append("projects: must name at least one project")
return ()
projects: list[Project] = []
repos: dict[str, str] = {}
for name, entry in raw.items():
where = f"projects.{name}"
if not isinstance(name, str) or not _PROJECT_NAME.fullmatch(name):
errors.append(f"{where}: a project name is lowercase letters, digits, -")
continue
if name == GENERAL:
errors.append(f"{where}: `{GENERAL}` is the scope of conversations (L1)")
continue
if not isinstance(entry, dict):
errors.append(f"{where}: must be a mapping")
continue
project = _parse_project(name, entry, env, errors)
if project is None:
continue
if project.repo in repos:
errors.append(
f"{where}.repo: {project.repo} is also project {repos[project.repo]}'s"
)
repos[project.repo] = name
projects.append(project)
return tuple(projects)
def _parse_project(
name: str, entry: Mapping[str, Any], env: Mapping[str, str], errors: list[str]
) -> "Project | None":
where = f"projects.{name}"
# labels, secrets and egress are structured; the rest flattens.
flat = _flatten({k: v for k, v in entry.items() if k not in ("secrets",)})
known = set(PROJECT_REQUIRED) | set(PROJECT_OPTIONAL)
for key in sorted(set(flat) - known):
errors.append(f"{where}.{key}: unknown setting")
values = {
key: _resolve(value, f"{where}.{key}", env, errors)
for key, value in flat.items()
if key in known
}
missing = [
key
for key in PROJECT_REQUIRED
if values.get(key) is None
or (isinstance(values[key], str) and not values[key].strip())
]
if missing:
errors.append(f"{where}: unset or empty — " + ", ".join(missing))
return None
port = values.get("build_host.port", 22)
try:
port = int(port)
if not 0 < port < 65536:
raise ValueError
except (TypeError, ValueError):
errors.append(f"{where}.build_host.port: must be a TCP port number")
push_user = values.get("push_user")
push_token = values.get("push_token")
push_key = values.get("push_key")
if bool(push_user) != bool(push_token):
errors.append(f"{where}: push_user and push_token: set both or neither")
if push_token and push_key:
errors.append(f"{where}: set a push token or a push key, not both")
labels = _strings(values.get("labels", []), f"{where}.labels", errors)
egress = _strings(values.get("egress", []), f"{where}.egress", errors)
secrets = _parse_secrets(entry.get("secrets"), where, env, errors)
return Project(
name=name,
repo=str(values["repo"]),
workspace=Path(values["workspace"]).expanduser(),
build_host=BuildHost(
address=str(values["build_host.address"]),
key=Path(values["build_host.key"]).expanduser(),
user=str(values.get("build_host.user") or "root"),
port=port if isinstance(port, int) else 22,
),
base_branch=str(values.get("base_branch") or "main"),
push_key=Path(push_key).expanduser() if push_key else None,
push_user=str(push_user) if push_user else None,
push_token=str(push_token) if push_token else None,
labels=labels,
secrets=secrets,
egress=egress,
)
def _strings(raw: Any, where: str, errors: list[str]) -> tuple[str, ...]:
if not isinstance(raw, list) or not all(
isinstance(x, str) and x.strip() for x in raw
):
errors.append(f"{where}: must be a list of names")
return ()
return tuple(x.strip() for x in raw)
def _parse_secrets(
raw: Any, where: str, env: Mapping[str, str], errors: list[str]
) -> dict[str, "Secret"]:
"""name → {value: ${VAR}, visible: commands | harness}. A value must be
a reference: secrets live in secret stores, never in this file."""
if raw is None:
return {}
if not isinstance(raw, dict):
errors.append(f"{where}.secrets: must be a mapping of name to secret")
return {}
secrets: dict[str, Secret] = {}
for name, spec in raw.items():
at = f"{where}.secrets.{name}"
if not isinstance(name, str) or not re.fullmatch(r"[A-Z_][A-Z0-9_]*", name):
errors.append(f"{at}: a secret's name is an environment variable name")
continue
if not isinstance(spec, dict) or set(spec) - {"value", "visible"}:
errors.append(f"{at}: give value and visible, nothing else")
continue
visible = spec.get("visible")
if visible not in VISIBILITY:
errors.append(f"{at}.visible: must be one of {', '.join(VISIBILITY)}")
continue
value = spec.get("value")
if not isinstance(value, str) or not _REFERENCE.fullmatch(value.strip()):
errors.append(f"{at}.value: must be a ${{VAR}} reference, never inline")
continue
secrets[name] = Secret(_resolve(value.strip(), at, env, errors), visible)
return secrets
@dataclass(frozen=True) @dataclass(frozen=True)
class BuildHost: class BuildHost:
"""The tool VM the workspace lives on (L13, D3). ``key`` is the path to """The tool VM the workspace lives on (L13, D3). ``key`` is the path to
@@ -236,11 +381,84 @@ class BuildHost:
port: int = 22 port: int = 22
@dataclass(frozen=True)
class Profile:
"""A model's capability profile (L9), from benchmarks: what one role is
told about another, never the model's name."""
tools: str = "" # how reliably it keeps the tool-call contract
strengths: str = ""
weaknesses: str = ""
def describe(self) -> str:
parts = [
f"tool calls: {self.tools}" if self.tools else "",
f"strong at {self.strengths}" if self.strengths else "",
f"weak at {self.weaknesses}" if self.weaknesses else "",
]
return "; ".join(p for p in parts if p) or "no profile recorded"
def _parse_profiles(raw: Any, errors: list[str]) -> dict[str, "Profile"]:
"""`profiles:`, model alias -> {tools, strengths, weaknesses}."""
if raw is None:
return {}
if not isinstance(raw, dict):
errors.append("profiles: must be a mapping of model alias to profile")
return {}
profiles: dict[str, Profile] = {}
for model, spec in raw.items():
where = f"profiles.{model}"
if not isinstance(spec, dict):
errors.append(f"{where}: must be a mapping")
continue
unknown = set(spec) - {"tools", "strengths", "weaknesses"}
if unknown:
errors.append(f"{where}.{min(unknown)}: unknown setting")
continue
if not all(isinstance(v, str) for v in spec.values()):
errors.append(f"{where}: values are text")
continue
profiles[str(model)] = Profile(**{k: v.strip() for k, v in spec.items()})
return profiles
@dataclass(frozen=True)
class Secret:
value: str = field(repr=False)
visible: str = "harness" # one of VISIBILITY
@dataclass(frozen=True)
class Project:
"""A project entry (D8): what makes a project known to the agent, and
everything per-project the rest of the agent refers to."""
name: str
repo: str
workspace: Path # on the build VM
build_host: BuildHost
base_branch: str = "main"
# What may push to the project repository, handed to each push on the
# build VM and to nothing else (D4): a push-only account's HTTPS token
# (push_user, push_token), or the path to a deploy key (push_key).
push_key: Path | None = None
push_user: str | None = None
push_token: str | None = field(default=None, repr=False)
labels: tuple[str, ...] = () # added to every issue's own (D1)
secrets: Mapping[str, Secret] = field(default_factory=dict)
egress: tuple[str, ...] = () # beyond the default; recorded, not yet enforced
def command_secrets(self) -> dict[str, str]:
"""The secrets the model's commands may see, by name."""
return {
name: s.value for name, s in self.secrets.items() if s.visible == "commands"
}
@dataclass(frozen=True) @dataclass(frozen=True)
class AgentConfig: class AgentConfig:
state_db: Path state_db: Path
workspace: Path
project: str
llm_base_url: str llm_base_url: str
# Credentials never appear in a repr: a failing test or a log line # Credentials never appear in a repr: a failing test or a log line
# would print them. # would print them.
@@ -250,16 +468,9 @@ class AgentConfig:
operators: frozenset[str] operators: frozenset[str]
gitea_url: str gitea_url: str
gitea_token: str = field(repr=False) gitea_token: str = field(repr=False)
gitea_repo: str
gitea_operators: frozenset[str] gitea_operators: frozenset[str]
build_host: BuildHost projects: tuple[Project, ...]
base_branch: str = "main" profiles: Mapping[str, Profile] = field(default_factory=dict)
# What may push to the project repository, handed to each push on the
# build VM and to nothing else (D4): a push-only account's HTTPS token
# (push_user, push_token), or the path to a deploy key (push_key).
push_key: Path | None = None
push_user: str | None = None
push_token: str | None = field(default=None, repr=False)
max_output_tokens: int = DEFAULT_MAX_OUTPUT_TOKENS max_output_tokens: int = DEFAULT_MAX_OUTPUT_TOKENS
request_timeout_s: float = DEFAULT_REQUEST_TIMEOUT_S request_timeout_s: float = DEFAULT_REQUEST_TIMEOUT_S
tuning: Tuning = field(default_factory=Tuning) tuning: Tuning = field(default_factory=Tuning)
@@ -268,6 +479,55 @@ class AgentConfig:
loop: str = "old" loop: str = "old"
config_file: Path | None = None config_file: Path | None = None
# -- one project, for the old loop ----------------------------------------
# control.py (until M2.8) knows one project; these read it, and refuse a
# configuration with more.
def only_project(self) -> Project:
if len(self.projects) != 1:
names = ", ".join(p.name for p in self.projects)
raise ConfigError(
f"cannot start: the old loop runs exactly one project, but "
f"{len(self.projects)} are configured ({names}); use `loop: new`"
)
return self.projects[0]
def with_project(self, **changes: Any) -> "AgentConfig":
"""This configuration with its one project changed (for tests)."""
return replace(self, projects=(replace(self.only_project(), **changes),))
@property
def project(self) -> str:
return self.only_project().name
@property
def gitea_repo(self) -> str:
return self.only_project().repo
@property
def workspace(self) -> Path:
return self.only_project().workspace
@property
def build_host(self) -> BuildHost:
return self.only_project().build_host
@property
def base_branch(self) -> str:
return self.only_project().base_branch
@property
def push_key(self) -> Path | None:
return self.only_project().push_key
@property
def push_user(self) -> str | None:
return self.only_project().push_user
@property
def push_token(self) -> str | None:
return self.only_project().push_token
@classmethod @classmethod
def load(cls, env: Mapping[str, str] = os.environ) -> "AgentConfig": def load(cls, env: Mapping[str, str] = os.environ) -> "AgentConfig":
"""The file named by NISSE_CONFIG, or ./nisse-dev.yaml.""" """The file named by NISSE_CONFIG, or ./nisse-dev.yaml."""
@@ -304,7 +564,11 @@ class AgentConfig:
config_file: Path | None, config_file: Path | None,
) -> "AgentConfig": ) -> "AgentConfig":
tuning = _parse_tuning(tree.get("tuning"), errors) tuning = _parse_tuning(tree.get("tuning"), errors)
flat = _flatten(tree) projects = _parse_projects(tree.get("projects"), env, errors)
profiles = _parse_profiles(tree.get("profiles"), errors)
flat = _flatten(
{k: v for k, v in tree.items() if k not in ("projects", "profiles")}
)
flat.pop("tuning", None) flat.pop("tuning", None)
known = set(REQUIRED_KEYS) | set(OPTIONAL_KEYS) known = set(REQUIRED_KEYS) | set(OPTIONAL_KEYS)
@@ -343,30 +607,14 @@ class AgentConfig:
raise ValueError raise ValueError
except (TypeError, ValueError): except (TypeError, ValueError):
errors.append("llm.request_timeout_s: must be a positive number of seconds") errors.append("llm.request_timeout_s: must be a positive number of seconds")
port = values.get("build_host.port", 22)
try:
port = int(port)
if not 0 < port < 65536:
raise ValueError
except (TypeError, ValueError):
errors.append("build_host.port: must be a TCP port number")
loop = str(values.get("loop") or "old") loop = str(values.get("loop") or "old")
if loop not in ("old", "new"): if loop not in ("old", "new"):
errors.append("loop: must be old or new") errors.append("loop: must be old or new")
push_user = values.get("gitea.push_user")
push_token = values.get("gitea.push_token")
if bool(push_user) != bool(push_token):
errors.append("gitea.push_user and gitea.push_token: set both or neither")
if push_token and values.get("gitea.push_key"):
errors.append("gitea: set a push token or a push key, not both")
if errors: if errors:
raise ConfigError("cannot start: " + "; ".join(errors)) raise ConfigError("cannot start: " + "; ".join(errors))
push_key = values.get("gitea.push_key")
config = cls( config = cls(
state_db=Path(values["state_db"]).expanduser(), state_db=Path(values["state_db"]).expanduser(),
workspace=Path(values["workspace"]).expanduser(),
project=str(values["project"]),
llm_base_url=str(values["llm.base_url"]), llm_base_url=str(values["llm.base_url"]),
llm_token=str(values["llm.token"]), llm_token=str(values["llm.token"]),
roles=RoleMap( roles=RoleMap(
@@ -374,6 +622,8 @@ class AgentConfig:
planner=str(values["roles.planner"]), planner=str(values["roles.planner"]),
helper=str(values["roles.helper"]), helper=str(values["roles.helper"]),
expert=str(values["roles.expert"]), expert=str(values["roles.expert"]),
driver=str(values.get("roles.driver") or ""),
classifier=str(values.get("roles.classifier") or ""),
), ),
matrix=MatrixConfig( matrix=MatrixConfig(
homeserver=str(values["matrix.homeserver"]).rstrip("/"), homeserver=str(values["matrix.homeserver"]).rstrip("/"),
@@ -384,18 +634,9 @@ class AgentConfig:
operators=_people(values["matrix.operators"]), operators=_people(values["matrix.operators"]),
gitea_url=str(values["gitea.url"]), gitea_url=str(values["gitea.url"]),
gitea_token=str(values["gitea.token"]), gitea_token=str(values["gitea.token"]),
gitea_repo=str(values["gitea.repo"]),
gitea_operators=_people(values["gitea.operators"]), gitea_operators=_people(values["gitea.operators"]),
build_host=BuildHost( projects=projects,
address=str(values["build_host.address"]), profiles=profiles,
key=Path(values["build_host.key"]).expanduser(),
user=str(values.get("build_host.user") or "root"),
port=port,
),
push_key=Path(push_key).expanduser() if push_key else None,
push_user=str(push_user) if push_user else None,
push_token=str(push_token) if push_token else None,
base_branch=str(values.get("gitea.base_branch") or "main"),
max_output_tokens=max_output, max_output_tokens=max_output,
request_timeout_s=request_timeout, request_timeout_s=request_timeout,
tuning=tuning, tuning=tuning,
@@ -416,13 +657,14 @@ class AgentConfig:
from and how the agent is deployed, which is not the project's from and how the agent is deployed, which is not the project's
business either. business either.
""" """
workspace = self.workspace.expanduser().resolve()
candidates = [ candidates = [
("state_db", self.state_db), ("state_db", self.state_db),
("NISSE_ENV", Path(env.get("NISSE_ENV", ".env"))), ("NISSE_ENV", Path(env.get("NISSE_ENV", ".env"))),
] ]
if self.config_file is not None: if self.config_file is not None:
candidates.append(("NISSE_CONFIG", self.config_file)) candidates.append(("NISSE_CONFIG", self.config_file))
for project in self.projects:
workspace = project.workspace.expanduser().resolve()
offenders = [] offenders = []
for label, path in candidates: for label, path in candidates:
resolved = path.expanduser().resolve() resolved = path.expanduser().resolve()
@@ -431,16 +673,18 @@ class AgentConfig:
if offenders: if offenders:
raise ConfigError( raise ConfigError(
"refusing to start: these sit inside the workspace clone at " "refusing to start: these sit inside the workspace clone at "
f"{workspace} and would be committed and pushed to the project " f"{workspace} (project {project.name}) and would be "
"repository — " + ", ".join(offenders) "committed and pushed to the project repository — "
+ ", ".join(offenders)
) )
def redacted(self) -> dict[str, str]: def redacted(self) -> dict[str, str]:
"""What the agent may safely announce about how it is configured.""" """What the agent may safely announce about how it is configured."""
return { return {
"project": self.project, "projects": ", ".join(
"workspace": f"{self.build_host.address}:{self.workspace}", f"{p.name} ({p.repo}, {p.build_host.address}:{p.workspace})"
"repo": self.gitea_repo, for p in self.projects
),
"planner": self.roles.planner, "planner": self.roles.planner,
"executor": self.roles.executor, "executor": self.roles.executor,
"helper": self.roles.helper, "helper": self.roles.helper,
+16 -3
View File
@@ -54,8 +54,8 @@ Stay inside the sub-task you were given. It comes from a plan a human already
approved; widening it is not yours to do. If the sub-task turns out to be approved; widening it is not yours to do. If the sub-task turns out to be
wrong or impossible, say so in the handback rather than substituting your own. wrong or impossible, say so in the handback rather than substituting your own.
When you are done, call hand_back exactly once. Write it for a materially When you are done, call hand_back exactly once. Write it for {reader},
weaker model that cannot see your reasoning and will act on your note alone: which cannot see your reasoning and will act on your note alone:
- what changed - what changed
- what is now true that was not before - what is now true that was not before
@@ -172,10 +172,17 @@ class Delegation:
caps: Caps | None = None, caps: Caps | None = None,
now: Callable[[], float] = time.monotonic, now: Callable[[], float] = time.monotonic,
cancel: Cancel | None = None, cancel: Cancel | None = None,
reader: str = "",
on_turn: Callable[[str], None] | None = None,
) -> None: ) -> None:
self._client = client self._client = client
self._store = store self._store = store
self._workspace = workspace self._workspace = workspace
# Who acts on the handback, from its capability profile (L9): never
# a model's name.
self._reader = reader or "a materially weaker model"
# The delegate's own text for each turn, for progress (#42).
self._on_turn = on_turn
self._git = git_repo self._git = git_repo
self._model = model self._model = model
self._backing_model = backing_model self._backing_model = backing_model
@@ -323,7 +330,8 @@ class Delegation:
self, transcript: Transcript, brief: str, branch: str, started_at: float self, transcript: Transcript, brief: str, branch: str, started_at: float
) -> DelegationResult: ) -> DelegationResult:
system = ( system = (
f"{SYSTEM_PROMPT}\n\n{self._workspace.describe()}" f"{SYSTEM_PROMPT.format(reader=self._reader)}\n\n"
f"{self._workspace.describe()}"
f"\n\nYour sub-task:\n{brief}" f"\n\nYour sub-task:\n{brief}"
) )
touched: set[str] = set() touched: set[str] = set()
@@ -395,6 +403,11 @@ class Delegation:
transcript.append(event) transcript.append(event)
if isinstance(event, ToolCall): if isinstance(event, ToolCall):
calls.append(event) calls.append(event)
said = " ".join(
e.text.strip() for e in produced if isinstance(e, AssistantText)
).strip()
if said and self._on_turn is not None:
self._on_turn(said)
if not calls: if not calls:
# A delegate that stops without handing back has not reported, # A delegate that stops without handing back has not reported,
+7
View File
@@ -255,6 +255,13 @@ class GitRepo:
self._git("commit", "-q", "-m", message) self._git("commit", "-q", "-m", message)
return self.head_sha() return self.head_sha()
def changed_paths(self, since: str) -> list[str]:
"""Every path the work changes relative to ``since``, committed or
not, new files included; build artifacts aren't work (#80)."""
tracked = self._git("diff", "--name-only", since).splitlines()
new = self._git("ls-files", "--others", "--exclude-standard").splitlines()
return sorted({p for p in [*tracked, *new] if p})
def left_out(self) -> list[str]: def left_out(self) -> list[str]:
"""Changes not in the last commit: build artifacts the harness left """Changes not in the last commit: build artifacts the harness left
out, to be reported rather than silently dropped (D4).""" out, to be reported rather than silently dropped (D4)."""
+18
View File
@@ -261,6 +261,24 @@ class GiteaClient:
who = (event.get("user") or {}).get("login") who = (event.get("user") or {}).get("login")
return who return who
def repository(self, repo: str) -> dict[str, Any]:
"""The repository as this account sees it, permissions included."""
return dict(self._request("GET", f"/repos/{repo}"))
def branch(self, repo: str, name: str) -> dict[str, Any]:
"""A branch as this account sees it: protected, user_can_push."""
return dict(self._request("GET", f"/repos/{repo}/branches/{name}"))
def merge_pull_request(self, repo: str, index: int) -> None:
self._request(
"POST", f"/repos/{repo}/pulls/{index}/merge", json={"Do": "merge"}
)
def assign_issue(self, repo: str, index: int, assignee: str) -> None:
self._request(
"PATCH", f"/repos/{repo}/issues/{index}", json={"assignees": [assignee]}
)
def close_issue(self, repo: str, index: int) -> None: def close_issue(self, repo: str, index: int) -> None:
self._request( self._request(
"PATCH", f"/repos/{repo}/issues/{index}", json={"state": "closed"} "PATCH", f"/repos/{repo}/issues/{index}", json={"state": "closed"}
+58 -13
View File
@@ -12,7 +12,7 @@ config change cannot split a task across two planners.
import json import json
import time import time
from dataclasses import asdict, dataclass from dataclasses import dataclass
from typing import Any, ClassVar, Protocol from typing import Any, ClassVar, Protocol
import httpx import httpx
@@ -39,6 +39,13 @@ class LLMError(RuntimeError):
RATE_LIMIT_STATUS = 429 RATE_LIMIT_STATUS = 429
RATE_LIMIT_RETRY_S = 20.0 RATE_LIMIT_RETRY_S = 20.0
# The proxy, or what's behind it, briefly gone: a 502/503/504 or a dropped
# connection. Outages delay, never fail (L5): waited out with backoff before
# the work is stopped. Not a 500 (it may be the request) and not a timeout
# (a hung model has had its request_timeout_s already).
OUTAGE_STATUSES = frozenset({502, 503, 504})
OUTAGE_BACKOFF_S = (5.0, 15.0, 45.0)
# Generous by default. Output is billed by what is produced, not by the # Generous by default. Output is billed by what is produced, not by the
# ceiling, and a ceiling too low does not fail loudly — it truncates a tool # ceiling, and a ceiling too low does not fail loudly — it truncates a tool
# call mid-argument and hands back something that parses. Measured: Opus # call mid-argument and hands back something that parses. Measured: Opus
@@ -107,21 +114,40 @@ class LLMClient(Protocol):
@dataclass(frozen=True) @dataclass(frozen=True)
class RoleMap: class RoleMap:
executor: str executor: str # the old loop's (control.py), until M2.8
planner: str planner: str # the old loop's, until M2.8
helper: str helper: str # delegations (L9)
expert: str expert: str # ask_expert (L9)
driver: str = "" # drives the new loop; unset, the planner's model
classifier: str = "" # reads go-aheads now, actions later (L4); unset, expert's
ROLES: ClassVar[tuple[str, ...]] = ("executor", "planner", "helper", "expert") ROLES: ClassVar[tuple[str, ...]] = (
"executor",
"planner",
"helper",
"expert",
"driver",
"classifier",
)
_FALLBACK: ClassVar[dict[str, str]] = {"driver": "planner", "classifier": "expert"}
def resolve(self, role: str) -> str: def resolve(self, role: str) -> str:
if role not in self.ROLES: if role not in self.ROLES:
raise LLMError(f"unknown role: {role!r}") raise LLMError(f"unknown role: {role!r}")
return str(getattr(self, role)) model = str(getattr(self, role))
if not model and role in self._FALLBACK:
return self.resolve(self._FALLBACK[role])
return model
def pinned(self) -> dict[str, str]: def pinned(self) -> dict[str, str]:
"""The full resolved mapping, to record at task start (L9).""" """The full resolved mapping, to record at task start (L9)."""
return asdict(self) return {role: self.resolve(role) for role in self.ROLES}
@classmethod
def from_pinned(cls, pinned: dict[str, str], default: "RoleMap") -> "RoleMap":
"""A work item's pinned mapping; a role it predates comes from
``default``."""
return cls(**{r: pinned.get(r) or default.resolve(r) for r in cls.ROLES})
class LiteLLMClient: class LiteLLMClient:
@@ -362,20 +388,39 @@ class AnthropicPassthroughClient:
def _post_with_rate_limit_retry( def _post_with_rate_limit_retry(
http: httpx.Client, path: str, body: dict[str, Any], sleep: Any http: httpx.Client, path: str, body: dict[str, Any], sleep: Any
) -> tuple[httpx.Response, float]: ) -> tuple[httpx.Response, float]:
"""The response, and how long was spent waiting out a rate limit.""" """The response, and how long was spent waiting: out a rate limit (once)
or an outage (with backoff), not on the model."""
waited = 0.0 waited = 0.0
for attempt in (1, 2): rate_limited = False
outages = iter(OUTAGE_BACKOFF_S)
while True:
try: try:
response = http.post(path, json=body) response = http.post(path, json=body)
except httpx.TimeoutException as exc:
raise UpstreamUnavailable(f"LLM request failed: {exc}") from exc
except httpx.HTTPError as exc: except httpx.HTTPError as exc:
# Could not reach the proxy at all: nothing about the request was # Could not reach the proxy at all: nothing about the request was
# wrong, so callers holding work in progress should keep it. # wrong, so callers holding work in progress should keep it.
pause = next(outages, None)
if pause is None:
raise UpstreamUnavailable(f"LLM request failed: {exc}") from exc raise UpstreamUnavailable(f"LLM request failed: {exc}") from exc
if response.status_code != RATE_LIMIT_STATUS or attempt == 2: sleep(pause)
return response, waited waited += pause
continue
if response.status_code == RATE_LIMIT_STATUS and not rate_limited:
rate_limited = True
sleep(RATE_LIMIT_RETRY_S) sleep(RATE_LIMIT_RETRY_S)
waited += RATE_LIMIT_RETRY_S waited += RATE_LIMIT_RETRY_S
raise AssertionError("unreachable") continue
if response.status_code in OUTAGE_STATUSES and not _looks_like_budget_refusal(
response.text
):
pause = next(outages, None)
if pause is not None:
sleep(pause)
waited += pause
continue
return response, waited
# Model families we have a native renderer adapter for. L9: delegations take # Model families we have a native renderer adapter for. L9: delegations take
File diff suppressed because it is too large. Load diff
+92 -32
View File
@@ -2,6 +2,7 @@
LoopAgent in place of the Orchestrator. Chosen with `loop: new` in config.""" LoopAgent in place of the Orchestrator. Chosen with `loop: new` in config."""
import threading import threading
from collections.abc import Callable
from dataclasses import dataclass from dataclasses import dataclass
from nisse_dev.agent import ( from nisse_dev.agent import (
@@ -14,10 +15,14 @@ from nisse_dev.agent import (
remote_url, remote_url,
) )
from nisse_dev.cancel import Cancel from nisse_dev.cancel import Cancel
from nisse_dev.config import AgentConfig from nisse_dev.catalog import select_client
from nisse_dev.config import AgentConfig, ConfigError
from nisse_dev.config import Project as ProjectEntry
from nisse_dev.delegation import Caps
from nisse_dev.gitea import GiteaClient from nisse_dev.gitea import GiteaClient
from nisse_dev.loop.agent import LoopAgent from nisse_dev.loop.agent import LoopAgent, Project
from nisse_dev.loop.driver import Driver from nisse_dev.loop.driver import Driver
from nisse_dev.loop.help import Helpers
from nisse_dev.loop.milestones import Milestones from nisse_dev.loop.milestones import Milestones
from nisse_dev.loop.reader import GoAheadReader from nisse_dev.loop.reader import GoAheadReader
from nisse_dev.matrix import MatrixTransport from nisse_dev.matrix import MatrixTransport
@@ -41,7 +46,8 @@ class RunningLoop:
def start(self) -> None: def start(self) -> None:
self.loop.start() self.loop.start()
self.transport.post( self.transport.post(
f"nisse-dev up as {self.account} (new loop) on {self.config.gitea_repo}; " f"nisse-dev up as {self.account} (new loop) on "
f"{', '.join(p.repo for p in self.config.projects)}; "
f"{self.loop_status()}" f"{self.loop_status()}"
) )
@@ -63,7 +69,48 @@ class RunningLoop:
self._stopped.wait() self._stopped.wait()
def build_loop_agent(config: AgentConfig) -> RunningLoop: def project_runtime(
config: AgentConfig,
entry: ProjectEntry,
store: StateStore,
gitea: GiteaClient,
account: str,
) -> Project:
"""One project's part of the loop: its clone on its build VM, workspace
(with the secrets its commands may see, redacted on return), push
credential and milestones."""
host = build_host(config, entry)
credential = push_credential(config, entry)
remote = remote_url(gitea, entry.repo, credential)
git = prepare_workspace(host, config, account, remote, credential, entry)
return Project(
name=entry.name,
repo=entry.repo,
labels=entry.labels,
workspace=Workspace(
host,
entry.workspace,
timeout_s=config.tuning.tools.command_timeout_s,
max_output_chars=config.tuning.tools.max_output_chars,
secrets=entry.command_secrets(),
),
milestones=Milestones(
store,
git,
gitea,
repo=entry.repo,
remote=remote,
credential=credential,
base=entry.base_branch,
),
)
def build_loop_agent(
config: AgentConfig, load: Callable[[], AgentConfig] | None = None
) -> RunningLoop:
"""``load`` reads the configuration again, for a new project's entry
(D8); by default from the file this one came from."""
oplog = OpLog("nisse-dev") oplog = OpLog("nisse-dev")
worker = SerialWorker( worker = SerialWorker(
on_error=lambda exc: oplog.emit( on_error=lambda exc: oplog.emit(
@@ -75,54 +122,58 @@ def build_loop_agent(config: AgentConfig) -> RunningLoop:
catalog = discover_models(config, oplog) catalog = discover_models(config, oplog)
gitea = GiteaClient(config.gitea_url, config.gitea_token) gitea = GiteaClient(config.gitea_url, config.gitea_token)
account = gitea.whoami() account = gitea.whoami()
host = build_host(config) # D8: each project its own build VM, clone, push credential and milestones.
credential = push_credential(config) projects = [
remote = remote_url(gitea, config.gitea_repo, credential) project_runtime(config, entry, store, gitea, account)
git = prepare_workspace(host, config, account, remote, credential) for entry in config.projects
workspace = Workspace( ]
host, normalized, passthrough = make_llm_clients(config, catalog)
config.workspace,
timeout_s=config.tuning.tools.command_timeout_s,
max_output_chars=config.tuning.tools.max_output_chars,
)
normalized, _passthrough = make_llm_clients(config, catalog)
cancel = Cancel() cancel = Cancel()
# The driver is the planner's model for now (M2.6 brings the driver role). # The configured driver; each work item runs on the one it pinned (L9).
driver = Driver( driver = Driver(
normalized, normalized,
config.roles.resolve("planner"), config.roles.resolve("driver"),
max_turns=config.tuning.executor.max_turns, max_turns=config.tuning.executor.max_turns,
cancel=cancel, cancel=cancel,
failure_limit=config.tuning.executor.failure_limit,
)
tuning = config.tuning
helpers = Helpers(
client=normalized,
roles=config.roles,
client_for=lambda model: select_client(model, normalized, passthrough, catalog),
caps=Caps(
turns=tuning.delegation.max_turns,
tokens=tuning.delegation.max_tokens,
wall_clock_s=tuning.delegation.wall_clock_s,
),
expert_budget=tuning.expert.consult_budget,
profiles=config.profiles,
) )
transport = MatrixTransport(config.matrix) transport = MatrixTransport(config.matrix)
loop = LoopAgent( loop = LoopAgent(
store, store,
transport, transport,
gitea, gitea,
repo=config.gitea_repo, projects=projects,
project=config.project,
account=account, account=account,
operators=config.operators, operators=config.operators,
gitea_operators=config.gitea_operators, gitea_operators=config.gitea_operators,
driver=driver, driver=driver,
workspace=workspace,
worker=worker, worker=worker,
cancel=cancel, cancel=cancel,
oplog=oplog, oplog=oplog,
roles=config.roles.pinned(), roles=config.roles.pinned(),
# The driver's model reads plain-language go-aheads until the # The classifier role reads plain-language go-aheads (L5, L9).
# classifier role arrives (M2.6). reader=GoAheadReader(normalized, config.roles.resolve("classifier")),
reader=GoAheadReader(normalized, config.roles.resolve("planner")), helpers=helpers,
reload=lambda known: [
project_runtime(config, entry, store, gitea, account)
for entry in (load or _from_file(config))().projects
if entry.name not in known
],
bug_grace_s=config.tuning.loop.bug_grace_s, bug_grace_s=config.tuning.loop.bug_grace_s,
milestones=Milestones( general_idle_s=config.tuning.loop.general_idle_s,
store,
git,
gitea,
repo=config.gitea_repo,
remote=remote,
credential=credential,
base=config.base_branch,
),
) )
return RunningLoop( return RunningLoop(
config=config, config=config,
@@ -134,3 +185,12 @@ def build_loop_agent(config: AgentConfig) -> RunningLoop:
worker=worker, worker=worker,
_stopped=threading.Event(), _stopped=threading.Event(),
) )
def _from_file(config: AgentConfig) -> Callable[[], AgentConfig]:
def load() -> AgentConfig:
if config.config_file is None:
raise ConfigError("this agent wasn't started from a config file")
return AgentConfig.from_file(config.config_file)
return load
+51 -4
View File
@@ -12,7 +12,7 @@ from dataclasses import dataclass
from typing import Any, Literal from typing import Any, Literal
from nisse_dev.cancel import Cancel, skip_calls from nisse_dev.cancel import Cancel, skip_calls
from nisse_dev.canonical import ToolCall, ToolResult from nisse_dev.canonical import AssistantText, ToolCall, ToolResult
from nisse_dev.llm import LLMClient, LLMError from nisse_dev.llm import LLMClient, LLMError
from nisse_dev.loop.tools import WAITING, names from nisse_dev.loop.tools import WAITING, names
from nisse_dev.state import Transcript from nisse_dev.state import Transcript
@@ -29,7 +29,8 @@ _PROSE_NUDGE = (
@dataclass(frozen=True) @dataclass(frozen=True)
class RunOutcome: class RunOutcome:
kind: Literal["wait", "capped", "stopped", "failed"] # stuck: failed commands in a row hit the failure limit (L9's trigger)
kind: Literal["wait", "capped", "stuck", "stopped", "failed"]
tool: str = "" # the waiting tool called tool: str = "" # the waiting tool called
arguments: dict[str, Any] | None = None arguments: dict[str, Any] | None = None
note: str = "" note: str = ""
@@ -43,10 +44,12 @@ class Driver:
*, *,
max_turns: int = DEFAULT_MAX_TURNS, max_turns: int = DEFAULT_MAX_TURNS,
cancel: Cancel | None = None, cancel: Cancel | None = None,
failure_limit: int = 0, # 0: no limit
) -> None: ) -> None:
self._client = client self._client = client
self._model = model self._model = model
self._max_turns = max_turns self._max_turns = max_turns
self._failure_limit = failure_limit
self._cancel = cancel or Cancel() self._cancel = cancel or Cancel()
def run( def run(
@@ -58,10 +61,18 @@ class Driver:
workspace: Workspace | None, workspace: Workspace | None,
handlers: Mapping[str, Callable[[dict[str, Any]], tuple[str, bool]]] handlers: Mapping[str, Callable[[dict[str, Any]], tuple[str, bool]]]
| None = None, | None = None,
checks: Mapping[str, Callable[[dict[str, Any]], str | None]] | None = None,
model: str | None = None,
on_turn: Callable[[str], None] | None = None,
) -> RunOutcome: ) -> RunOutcome:
"""``handlers`` answer control tools that don't end the run (e.g. """``handlers`` answer control tools that don't end the run (e.g.
start_milestone): each returns a result and whether it succeeded.""" start_milestone): each returns a result and whether it succeeded.
``checks`` vet a waiting tool's arguments: a problem is the call's
failed result and the run goes on, so the model puts it right."""
handlers = handlers or {} handlers = handlers or {}
checks = checks or {}
model = model or self._model # the work item's pinned driver (L9)
failures: list[str] = [] # the failed commands in a row, latest last
offered = names(tools) offered = names(tools)
nudged = False nudged = False
for turn in range(self._max_turns): for turn in range(self._max_turns):
@@ -70,7 +81,7 @@ class Driver:
return RunOutcome("stopped", note=f"before turn {turn + 1}") return RunOutcome("stopped", note=f"before turn {turn + 1}")
try: try:
produced = self._client.complete( produced = self._client.complete(
transcript.view(), tools=tools, model=self._model, system=system transcript.view(), tools=tools, model=model, system=system
) )
except LLMError as exc: except LLMError as exc:
return RunOutcome("failed", note=f"the driver's call failed: {exc}") return RunOutcome("failed", note=f"the driver's call failed: {exc}")
@@ -79,6 +90,12 @@ class Driver:
transcript.append(event) transcript.append(event)
if isinstance(event, ToolCall): if isinstance(event, ToolCall):
calls.append(event) calls.append(event)
# The model's own words for the turn: its progress, in prose (#42).
said = " ".join(
e.text.strip() for e in produced if isinstance(e, AssistantText)
).strip()
if said and calls and on_turn is not None:
on_turn(said)
if not calls: if not calls:
if nudged: if nudged:
return RunOutcome("failed", note="kept replying in prose") return RunOutcome("failed", note="kept replying in prose")
@@ -89,6 +106,19 @@ class Driver:
if self._cancel.requested: if self._cancel.requested:
skip_calls(transcript, calls[index:]) skip_calls(transcript, calls[index:])
return RunOutcome("stopped", note=f"during turn {turn + 1}") return RunOutcome("stopped", note=f"during turn {turn + 1}")
problem = (
checks[call.tool](dict(call.arguments))
if call.tool in checks and call.tool in offered
else None
)
if problem is not None:
transcript.append(
ToolResult(call_id=call.call_id, result=problem, success=False)
)
skip_calls(
transcript, calls[index + 1 :], "not run: fix the call above"
)
break
if call.tool in WAITING and call.tool in offered: if call.tool in WAITING and call.tool in offered:
transcript.append( transcript.append(
ToolResult( ToolResult(
@@ -115,4 +145,21 @@ class Driver:
transcript.append( transcript.append(
ToolResult(call_id=call.call_id, result=result, success=ok) ToolResult(call_id=call.call_id, result=result, success=ok)
) )
if call.tool == "run_command":
failures = [] if ok else [*failures, _brief(call, result)]
if self._failure_limit and len(failures) >= self._failure_limit:
skip_calls(
transcript, calls[index + 1 :], "not run: you're stuck"
)
return RunOutcome(
"stuck",
note=f"{len(failures)} commands failed in a row: "
+ "; ".join(failures),
)
return RunOutcome("capped", note=f"hit the {self._max_turns}-turn cap") return RunOutcome("capped", note=f"hit the {self._max_turns}-turn cap")
def _brief(call: ToolCall, result: str) -> str:
"""A failed command and the end of what it said, for a helper's brief."""
command = str(call.arguments.get("command", ""))[:120]
return f"`{command}`: {result.strip()[-300:]}"
+73
View File
@@ -0,0 +1,73 @@
"""Help for the driver (L9): a helper for a sub-task (delegation), an expert
for one question, both by role, briefed from capability profiles.
The driver asks when it chooses to (its instructions treat asking as
normal), and the harness asks for it on observable triggers: commands failing
in a row, or the turn cap. Delegation runs on a branch off the milestone's,
merged back on a handback and discarded otherwise (L9: never build on a
half-edited tree).
"""
from collections.abc import Callable, Mapping
from dataclasses import dataclass, field
from nisse_dev.config import Profile
from nisse_dev.delegation import Caps
from nisse_dev.expert import DEFAULT_BUDGET
from nisse_dev.llm import LLMClient, RoleMap
from nisse_dev.loop.tools import _tool
DELEGATE = _tool(
"delegate",
"Hand a sub-task of the current milestone to a helper, who works on the "
"milestone's branch with the same tools and reports back what changed, "
"what is now true and what next. Use it when the sub-task is beyond what "
"you can do reliably, or you are going round in circles. Needs a "
"milestone started.",
task="The sub-task, self-contained: what to do, where, and how to know it's done.",
)
ASK_EXPERT = _tool(
"ask_expert",
"Ask an expert one self-contained question: a design trade-off, how a "
"library behaves. It can't see the repository. The budget per item is "
"small.",
question="The question, with whatever context it needs.",
)
@dataclass(frozen=True)
class Helpers:
client: LLMClient # for the expert
roles: RoleMap # the configured mapping; each item pins its own (L9)
# The helper's client by model (vendor passthrough where it fits, L9);
# unset, ``client``.
client_for: Callable[[str], LLMClient] | None = None
caps: Caps = field(default_factory=Caps)
expert_budget: int = DEFAULT_BUDGET
profiles: Mapping[str, Profile] = field(default_factory=dict)
def delegate_client(self, model: str) -> LLMClient:
return self.client_for(model) if self.client_for else self.client
def profile(self, model: str) -> str:
"""What one role is told about another: its capability profile,
never its name (L9)."""
found = self.profiles.get(model)
return found.describe() if found else "no profile recorded"
def briefing(self, roles: RoleMap) -> str:
"""The driver's instructions on asking for help."""
return (
"Asking for help is normal, not a failure. delegate hands a "
"sub-task to a helper ("
+ self.profile(roles.resolve("helper"))
+ "); ask_expert asks one question of an expert ("
+ self.profile(roles.resolve("expert"))
+ "). If commands keep failing, a helper is called in for you."
)
def reader(self, roles: RoleMap) -> str:
"""Who acts on a helper's handback: the driver, by its profile."""
return "the driver, a model with this profile: " + self.profile(
roles.resolve("driver")
)
+34 -9
View File
@@ -20,6 +20,8 @@ CLOSED = "closed"
GO_AHEAD = "go-ahead" # a go on the restatement up (L3, D1) GO_AHEAD = "go-ahead" # a go on the restatement up (L3, D1)
ANSWER = "answer" # an answer to a question ANSWER = "answer" # an answer to a question
MERGE = "merge" # the PR's review and merge (D4, D5) MERGE = "merge" # the PR's review and merge (D4, D5)
CHAT = "conversation" # a conversation waits for the operator to say more (L5)
PROVISION = "provisioning" # a new project waits for what it needs (D1, D8)
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -43,6 +45,24 @@ class Loop:
labels: tuple[str, ...] = () # the issue's labels at intake: they steer (D1) labels: tuple[str, ...] = () # the issue's labels at intake: they steer (D1)
grace_until: float = 0.0 # a `bug` restatement proceeds then, unanswered grace_until: float = 0.0 # a `bug` restatement proceeds then, unanswered
talked: bool = False # the operator has spoken in the item's thread talked: bool = False # the operator has spoken in the item's thread
# A conversation (`general`, L1): when it last heard or said anything,
# the conversation it continues, the work it turned into, and the work
# proposed for a go-ahead.
idle_since: float = 0.0
follows: str = ""
became: tuple[str, ...] = ()
proposal_project: str = ""
proposal_title: str = ""
proposal_kind: str = "" # "work" on a project, or a new "project"
new_project: str = "" # a new project being provisioned (D1)
design_merged: bool = False # a `design` item's design PR has merged (D1, D2)
helps: int = 0 # delegations so far, for their branch names (L9)
helped_at: int = 0 # the milestone a helper was last called in for, unasked
# A question from the operator is answered without changing the item
# (L5): the wait it interrupted, to go back to.
question: bool = False
back_to: str = ""
back_note: str = ""
# Where the operator last spoke (D5: answer where asked): "" for the # Where the operator last spoke (D5: answer where asked): "" for the
# Matrix thread, or the PR it came from, with the inline location and a # Matrix thread, or the PR it came from, with the inline location and a
# little of what they said, for quoting. # little of what they said, for quoting.
@@ -59,6 +79,7 @@ def loop_of(item: WorkItem) -> Loop:
data["notes"] = tuple(data.get("notes", ())) data["notes"] = tuple(data.get("notes", ()))
data["blocked_by"] = tuple(data.get("blocked_by", ())) data["blocked_by"] = tuple(data.get("blocked_by", ()))
data["labels"] = tuple(data.get("labels", ())) data["labels"] = tuple(data.get("labels", ()))
data["became"] = tuple(data.get("became", ()))
return Loop(**data) return Loop(**data)
@@ -75,17 +96,21 @@ def save(
) )
def item_for_thread(store: StateStore, thread: str) -> WorkItem | None: def item_for_thread(
for item in store.work_items("open"): store: StateStore, thread: str, status: str | None = "open"
if loop_of(item).thread == thread: ) -> WorkItem | None:
return item """The item whose conversation is ``thread``; the latest, when a closed
return None conversation's thread was picked up again."""
found = [i for i in store.work_items(status) if loop_of(i).thread == thread]
return found[-1] if found else None
def item_for_issue(store: StateStore, issue: int) -> WorkItem | None: def item_for_issue(
"""Any item, open or closed, recorded by this issue: intake never takes store: StateStore, issue: int, project: str | None = None
an issue twice (D1).""" ) -> WorkItem | None:
"""Any item, open or closed, recorded by this issue (in ``project``,
when given): intake never takes an issue twice (D1)."""
for item in store.work_items(): for item in store.work_items():
if loop_of(item).issue == issue: if loop_of(item).issue == issue and project in (None, item.scope):
return item return item
return None return None
@@ -7,7 +7,9 @@ Publishing is idempotent under an intent (L8): an open PR for the branch is
adopted, and a PR the operator closed is an answer, never reopened. adopted, and a PR the operator closed is an answer, never reopened.
""" """
import contextlib
import time import time
import uuid
from dataclasses import dataclass from dataclasses import dataclass
from typing import Literal from typing import Literal
@@ -159,6 +161,88 @@ class Milestones:
if git.remote_sha(self._remote, parked, credential=self._credential): if git.remote_sha(self._remote, parked, credential=self._credential):
git.delete_remote_branch(self._remote, parked, credential=self._credential) git.delete_remote_branch(self._remote, parked, credential=self._credential)
@property
def git(self) -> GitRepo:
"""The clone, for a delegation's branch (L9)."""
return self._git
def changed(self) -> list[str]:
"""What the current milestone changes, from the default branch."""
return self._git.changed_paths(f"origin/{self._base}")
# -- a new project's checks (D1, D4) -----------------------------------------
def verify_access(self, account: str) -> list[str]:
"""What the operator provided, checked rather than assumed: the
agent may push a branch with its push credential, may not push to
the default branch, and may not merge its own PR without approval.
A throwaway branch with an empty commit carries the checks; its PR
is closed and the branch deleted. Returns what failed."""
git, base, failures = self._git, self._base, []
try:
repo = self._gitea.repository(self._repo)
if not (repo.get("permissions") or {}).get("push"):
failures.append(f"{account} isn't a collaborator with write access")
branch = self._gitea.branch(self._repo, base)
if not branch.get("protected") or branch.get("user_can_push"):
failures.append(f"{base} isn't protected against my pushes")
except GiteaError as exc:
return [f"I can't see the repository {self._repo}: {exc}"]
check = f"nisse-check-{uuid.uuid4().hex[:6]}"
try:
git.fetch(credential=self._credential)
git.reset_to(check, f"origin/{base}")
git._git("commit", "-q", "--allow-empty", "-m", "nisse-dev: access check")
except GitError as exc:
return [*failures, f"the workspace isn't usable: {exc}"]
try:
git.push(self._remote, check, credential=self._credential)
except GitError as exc:
failures.append(f"my push credential can't push a branch: {exc}")
self._back_to_base(check)
return failures
try:
git.push(self._remote, base, credential=self._credential)
failures.append(
f"I could push to {base}, which protection should refuse; an "
f"empty commit, 'nisse-dev: access check', is on {base} now"
)
except GitError:
pass # refused, as it should be
pr = None
try:
pr = self._gitea.open_pull_request(
self._repo,
head=check,
base=base,
title="nisse-dev access check (never merged)",
body="Checks that merging needs an approval; closed straight away.",
)
try:
self._gitea.merge_pull_request(self._repo, pr.number)
failures.append(
f"I could merge my own PR #{pr.number} without an approval"
)
except GiteaError:
pass # refused, as it should be
except GiteaError as exc:
failures.append(f"I couldn't open a PR to check merging: {exc}")
finally:
if pr is not None:
with contextlib.suppress(GiteaError):
self._gitea.close_pull_request(self._repo, pr.number)
with contextlib.suppress(GitError):
git.delete_remote_branch(
self._remote, check, credential=self._credential
)
self._back_to_base(check)
return failures
def _back_to_base(self, branch: str) -> None:
with contextlib.suppress(GitError):
self._git.reset_to(self._base, f"origin/{self._base}")
self._git._git("branch", "-q", "-D", branch)
def _ahead_of_origin(self) -> bool: def _ahead_of_origin(self) -> bool:
current = self._git.current_branch() current = self._git.current_branch()
try: try:
+3 -2
View File
@@ -50,7 +50,8 @@ class GoAheadReader:
self._client = client self._client = client
self._model = model self._model = model
def agrees(self, proposal: str, reply: str) -> bool: def agrees(self, proposal: str, reply: str, model: str = "") -> bool:
"""``model``: the work item's pinned classifier (L9)."""
if is_question(reply): if is_question(reply):
return False return False
prompt = f"The proposal:\n\n{proposal}\n\nThe operator's reply:\n\n{reply}" prompt = f"The proposal:\n\n{proposal}\n\nThe operator's reply:\n\n{reply}"
@@ -65,7 +66,7 @@ class GoAheadReader:
) )
], ],
tools=[VERDICT], tools=[VERDICT],
model=self._model, model=model or self._model,
system=SYSTEM, system=SYSTEM,
) )
except LLMError: except LLMError:
+48 -3
View File
@@ -25,9 +25,12 @@ def _tool(tool: str, description: str, /, **params: str) -> dict[str, Any]:
RESTATE = _tool( RESTATE = _tool(
"restate", "restate",
"Put your understanding of the request to the operator, in their item's " "Put your understanding of the request to the operator, in their item's "
"thread: what you take it to mean, how it fits the project, and your " "thread: what you take it to mean, how it fits the project, your first "
"first step. Ends your turn; work starts when they agree.", "step, and why that way (and the main alternative you considered). Ends "
"your turn; work starts when they agree.",
text="Your restatement, in plain language.", text="Your restatement, in plain language.",
changed="What changed from your previous restatement, and why; leave it "
"empty for the first.",
) )
ASK_HUMAN = _tool( ASK_HUMAN = _tool(
"ask_human", "ask_human",
@@ -66,8 +69,50 @@ WAIT_FOR_REVIEW = _tool(
note="What you're waiting on, for the operator.", note="What you're waiting on, for the operator.",
) )
# A conversation's tools (L1, L5): it talks, and may propose work.
REPLY = _tool(
"reply",
"Answer the operator in the conversation's thread. Ends your turn; you "
"continue when they say more.",
text="Your answer, in plain language.",
)
PROPOSE_WORK = _tool(
"propose_work",
"The conversation has turned into work on one of the projects: propose "
"an issue for it. The operator agrees to it as to a restatement; then the "
"issue is opened, and work on it starts as its own item. Ends your turn.",
project="The project's name, from the list in your instructions.",
title="A short title for the issue.",
summary="What was decided, distilled: the request, what it means, and the "
"first step. Not the conversation; it is the issue's body and the work's "
"whole brief.",
)
PROPOSE_PROJECT = _tool(
"propose_project",
"The conversation has become a new project: propose it. The operator "
"agrees to it as to a restatement; then you list what it needs set up "
"(a repository, a build VM, the project's entry), and once it is ready "
"and checked, the project's architecture is your first piece of work. "
"Ends your turn.",
name="The project's name: lowercase letters, digits and -.",
purpose="What it is for, in a paragraph.",
decisions="The decisions setting it up depends on: language and toolchain "
"(which pick the build VM), services, secrets, network access beyond the "
"default; and anything else agreed so far.",
)
WAITING = frozenset( WAITING = frozenset(
{"restate", "ask_human", "close_item", "finish_milestone", "wait_for_review"} {
"restate",
"ask_human",
"close_item",
"finish_milestone",
"wait_for_review",
"reply",
"propose_work",
"propose_project",
}
) )
+67 -7
View File
@@ -9,7 +9,7 @@ from typing import Any
import httpx import httpx
import pytest import pytest
from test_config import SECRETS, tree from test_config import SECRETS, P, tree
from nisse_dev.agent import build_agent, discover_models, prepare_workspace from nisse_dev.agent import build_agent, discover_models, prepare_workspace
from nisse_dev.catalog import ModelCatalog, UnknownModel from nisse_dev.catalog import ModelCatalog, UnknownModel
@@ -51,11 +51,13 @@ def config_for(tmp_path: Path, **changes: str) -> AgentConfig:
return AgentConfig.from_tree( return AgentConfig.from_tree(
tree( tree(
state_db=str(tmp_path / "state.db"), state_db=str(tmp_path / "state.db"),
workspace=str(tmp_path / "work"), **{
build_host__address=HOST.address, f"{P}workspace": str(tmp_path / "work"),
build_host__user=HOST.user, f"{P}build_host__address": HOST.address,
build_host__port=str(HOST.port), f"{P}build_host__user": HOST.user,
build_host__key=str(HOST._key), f"{P}build_host__port": str(HOST.port),
f"{P}build_host__key": str(HOST._key),
},
**changes, **changes,
), ),
SECRETS, SECRETS,
@@ -141,6 +143,9 @@ class FakeTransport:
def build_offline(tmp_path: Path, origin: Path, monkeypatch: Any) -> Any: def build_offline(tmp_path: Path, origin: Path, monkeypatch: Any) -> Any:
"""Assemble the real graph with only the network boundaries faked.""" """Assemble the real graph with only the network boundaries faked."""
config = config_for(tmp_path) config = config_for(tmp_path)
# The proxy here doesn't exist: a call to it fails at once rather than
# waiting out an outage's backoff (L5).
monkeypatch.setattr("nisse_dev.llm.OUTAGE_BACKOFF_S", ())
monkeypatch.setattr( monkeypatch.setattr(
"nisse_dev.agent.GiteaClient.whoami", lambda self: "nisse-dev-testing" "nisse_dev.agent.GiteaClient.whoami", lambda self: "nisse-dev-testing"
@@ -397,8 +402,63 @@ def test_the_new_loop_assembles(tmp_path: Path, origin: Path, monkeypatch) -> No
running = build_loop_agent(config) running = build_loop_agent(config)
try: try:
assert running.account == "nisse-dev-testing" assert running.account == "nisse-dev-testing"
assert running.loop._repo == config.gitea_repo [project] = running.loop._projects.values()
assert project.repo == config.gitea_repo
assert (config.workspace / "README.md").exists() # cloned on the build host assert (config.workspace / "README.md").exists() # cloned on the build host
finally: finally:
running.worker.call(running.store.close) running.worker.call(running.store.close)
running.worker.stop(timeout=5) running.worker.stop(timeout=5)
def test_each_project_gets_its_own_clone_and_secrets(
tmp_path: Path, origin: Path, monkeypatch
) -> None:
"""D8: a second project is only a second entry."""
from nisse_dev.loop.build import build_loop_agent
second = tmp_path / "second"
config = config_for(
tmp_path,
loop="new",
projects__other={
"repo": "don/other",
"workspace": str(tmp_path / "other-work"),
"build_host": {
"address": HOST.address,
"user": HOST.user,
"port": str(HOST.port),
"key": str(HOST._key),
},
"labels": ["discuss"],
"secrets": {
"API_KEY": {"value": "${TEST_LLM_TOKEN}", "visible": "commands"}
},
},
)
git("clone", "-q", "--bare", str(origin), str(second))
origins = {"don/demo": str(origin), "don/other": str(second)}
monkeypatch.setattr(
"nisse_dev.loop.build.GiteaClient.whoami", lambda self: "nisse-dev-testing"
)
monkeypatch.setattr(
"nisse_dev.loop.build.GiteaClient.clone_url", lambda self, repo: origins[repo]
)
monkeypatch.setattr(
"nisse_dev.agent.ModelCatalog.discover",
classmethod(lambda cls, *a, **kw: TEST_CATALOG),
)
monkeypatch.setattr(
"nisse_dev.loop.build.MatrixTransport", lambda cfg: FakeTransport()
)
running = build_loop_agent(config)
try:
projects = running.loop._projects
assert set(projects) == {"greet", "other"}
assert (tmp_path / "work" / "README.md").exists()
assert (tmp_path / "other-work" / "README.md").exists()
assert projects["other"].labels == ("discuss",)
assert projects["other"].workspace._secrets == {"API_KEY": "sk-x"}
assert projects["greet"].workspace._secrets == {}
finally:
running.worker.call(running.store.close)
running.worker.stop(timeout=5)
+155 -27
View File
@@ -10,8 +10,13 @@ from nisse_dev.llm import DEFAULT_REQUEST_TIMEOUT_S
# A complete config as the YAML file would give it, with secrets by reference. # A complete config as the YAML file would give it, with secrets by reference.
COMPLETE: dict[str, Any] = { COMPLETE: dict[str, Any] = {
"state_db": "~/demo/state.db", "state_db": "~/demo/state.db",
"projects": {
"greet": {
"repo": "don/demo",
"workspace": "~/demo/work", "workspace": "~/demo/work",
"project": "greet", "build_host": {"address": "build.test", "key": "~/.ssh/nisse-dev"},
}
},
"llm": {"base_url": "https://litellm.test", "token": "${TEST_LLM_TOKEN}"}, "llm": {"base_url": "https://litellm.test", "token": "${TEST_LLM_TOKEN}"},
"roles": { "roles": {
"executor": "claude-haiku-4-5", "executor": "claude-haiku-4-5",
@@ -29,11 +34,10 @@ COMPLETE: dict[str, Any] = {
"gitea": { "gitea": {
"url": "https://gitea.test", "url": "https://gitea.test",
"token": "${TEST_GITEA_TOKEN}", "token": "${TEST_GITEA_TOKEN}",
"repo": "don/demo",
"operators": ["don"], "operators": ["don"],
}, },
"build_host": {"address": "build.test", "key": "~/.ssh/nisse-dev"},
} }
P = "projects__greet__" # the one project's keys, for tree()
SECRETS = { SECRETS = {
"TEST_LLM_TOKEN": "sk-x", "TEST_LLM_TOKEN": "sk-x",
"TEST_MATRIX_TOKEN": "syt_x", "TEST_MATRIX_TOKEN": "syt_x",
@@ -83,15 +87,15 @@ def test_every_missing_setting_is_reported_at_once() -> None:
# Failing at first use would surface a missing token twenty minutes into # Failing at first use would surface a missing token twenty minutes into
# a milestone, on the call that mattered. # a milestone, on the call that mattered.
with pytest.raises(ConfigError) as excinfo: with pytest.raises(ConfigError) as excinfo:
build(matrix__token=None, gitea__repo=None) build(matrix__token=None, **{f"{P}repo": None})
message = str(excinfo.value) message = str(excinfo.value)
assert "matrix.token" in message assert "matrix.token" in message
assert "gitea.repo" in message assert "projects.greet: unset or empty — repo" in message
def test_empty_is_treated_as_missing() -> None: def test_empty_is_treated_as_missing() -> None:
with pytest.raises(ConfigError, match="gitea.repo"): with pytest.raises(ConfigError, match="projects.greet: unset or empty — repo"):
build(gitea__repo=" ") build(**{f"{P}repo": " "})
def test_a_reference_to_an_unset_variable_is_an_error() -> None: def test_a_reference_to_an_unset_variable_is_an_error() -> None:
@@ -115,7 +119,7 @@ def test_an_operator_list_of_only_blanks_is_missing() -> None:
def test_base_branch_is_configurable() -> None: def test_base_branch_is_configurable() -> None:
assert build(gitea__base_branch="trunk").base_branch == "trunk" assert build(**{f"{P}base_branch": "trunk"}).base_branch == "trunk"
def test_max_output_tokens_is_configurable_and_checked() -> None: def test_max_output_tokens_is_configurable_and_checked() -> None:
@@ -148,7 +152,7 @@ def test_redacted_summary_carries_no_secrets() -> None:
flat = " ".join(summary.values()) flat = " ".join(summary.values())
for secret in SECRETS.values(): for secret in SECRETS.values():
assert secret not in flat assert secret not in flat
assert summary["project"] == "greet" assert "greet (don/demo, build.test:" in summary["projects"]
def test_tuning_is_read() -> None: def test_tuning_is_read() -> None:
@@ -178,7 +182,7 @@ def test_a_partly_given_tuning_section_keeps_the_other_defaults() -> None:
def test_every_problem_is_reported_at_once() -> None: def test_every_problem_is_reported_at_once() -> None:
# A misspelt knob that was silently ignored would look exactly like a knob # A misspelt knob that was silently ignored would look exactly like a knob
# with no effect, so unknown keys are errors. # with no effect, so unknown keys are errors.
bad = tree(project=None, llm_extra=1) bad = tree(llm_extra=1, **{f"{P}workspace": None})
bad["tuning"] = { bad["tuning"] = {
"executor": {"max_turn": 40, "failure_limit": 0}, "executor": {"max_turn": 40, "failure_limit": 0},
"expert": {"consult_budget": True}, "expert": {"consult_budget": True},
@@ -188,7 +192,7 @@ def test_every_problem_is_reported_at_once() -> None:
AgentConfig.from_tree(bad, SECRETS) AgentConfig.from_tree(bad, SECRETS)
message = str(excinfo.value) message = str(excinfo.value)
for expected in ( for expected in (
"project", "projects.greet: unset or empty — workspace",
"llm_extra: unknown setting", "llm_extra: unknown setting",
"tuning.executor.max_turn: unknown setting", "tuning.executor.max_turn: unknown setting",
"tuning.executor.failure_limit: must be a positive number", "tuning.executor.failure_limit: must be a positive number",
@@ -207,8 +211,13 @@ def test_a_whole_number_is_required_where_one_is_meant() -> None:
FILE = """ FILE = """
state_db: ~/demo/state.db state_db: ~/demo/state.db
workspace: ~/demo/work projects:
project: greet greet:
repo: don/demo
workspace: ~/demo/work
build_host:
address: build.test
key: ~/.ssh/nisse-dev
llm: {base_url: "https://litellm.test", token: "${TEST_LLM_TOKEN}"} llm: {base_url: "https://litellm.test", token: "${TEST_LLM_TOKEN}"}
roles: roles:
executor: local-fast executor: local-fast
@@ -224,11 +233,7 @@ matrix:
gitea: gitea:
url: https://gitea.test url: https://gitea.test
token: ${TEST_GITEA_TOKEN} token: ${TEST_GITEA_TOKEN}
repo: don/demo
operators: [don] operators: [don]
build_host:
address: build.test
key: ~/.ssh/nisse-dev
""" """
@@ -297,7 +302,7 @@ def test_state_db_inside_the_clone_is_refused(tmp_path: Path) -> None:
# own state alongside the project. # own state alongside the project.
work = tmp_path / "project" work = tmp_path / "project"
with pytest.raises(ConfigError, match="inside the workspace"): with pytest.raises(ConfigError, match="inside the workspace"):
build(workspace=str(work), state_db=str(work / "state.db")) build(state_db=str(work / "state.db"), **{f"{P}workspace": str(work)})
def test_env_file_inside_the_clone_is_refused(tmp_path: Path) -> None: def test_env_file_inside_the_clone_is_refused(tmp_path: Path) -> None:
@@ -306,7 +311,7 @@ def test_env_file_inside_the_clone_is_refused(tmp_path: Path) -> None:
work = tmp_path / "project" work = tmp_path / "project"
with pytest.raises(ConfigError, match="NISSE_ENV"): with pytest.raises(ConfigError, match="NISSE_ENV"):
AgentConfig.from_tree( AgentConfig.from_tree(
tree(workspace=str(work), state_db=str(tmp_path / "state.db")), tree(state_db=str(tmp_path / "state.db"), **{f"{P}workspace": str(work)}),
{**SECRETS, "NISSE_ENV": str(work / ".env")}, {**SECRETS, "NISSE_ENV": str(work / ".env")},
) )
@@ -315,13 +320,17 @@ def test_a_config_file_inside_the_clone_is_refused(tmp_path: Path) -> None:
work = tmp_path / "project" work = tmp_path / "project"
work.mkdir() work.mkdir()
text = FILE.replace("workspace: ~/demo/work", f"workspace: {work}") text = FILE.replace("workspace: ~/demo/work", f"workspace: {work}")
assert text != FILE
with pytest.raises(ConfigError, match="NISSE_CONFIG"): with pytest.raises(ConfigError, match="NISSE_CONFIG"):
AgentConfig.from_file(write(work, text), SECRETS) AgentConfig.from_file(write(work, text), SECRETS)
def test_sibling_paths_are_fine(tmp_path: Path) -> None: def test_sibling_paths_are_fine(tmp_path: Path) -> None:
config = AgentConfig.from_tree( config = AgentConfig.from_tree(
tree(workspace=str(tmp_path / "project"), state_db=str(tmp_path / "state.db")), tree(
state_db=str(tmp_path / "state.db"),
**{f"{P}workspace": str(tmp_path / "project")},
),
{**SECRETS, "NISSE_ENV": str(tmp_path / ".env")}, {**SECRETS, "NISSE_ENV": str(tmp_path / ".env")},
) )
assert config.workspace.name == "project" assert config.workspace.name == "project"
@@ -342,30 +351,36 @@ def test_the_build_host_is_required_and_defaults_to_root_on_22() -> None:
assert config.build_host.port == 22 assert config.build_host.port == 22
assert config.build_host.key.is_absolute() assert config.build_host.key.is_absolute()
with pytest.raises(ConfigError, match="build_host.address"): with pytest.raises(ConfigError, match="build_host.address"):
build(build_host__address=None) build(**{f"{P}build_host__address": None})
def test_a_bad_build_host_port_is_reported() -> None: def test_a_bad_build_host_port_is_reported() -> None:
with pytest.raises(ConfigError, match="build_host.port"): with pytest.raises(ConfigError, match="build_host.port"):
build(build_host__port="ssh") build(**{f"{P}build_host__port": "ssh"})
def test_the_push_key_is_optional_and_a_path() -> None: def test_the_push_key_is_optional_and_a_path() -> None:
assert build().push_key is None assert build().push_key is None
assert build(gitea__push_key="~/k").push_key == Path("~/k").expanduser() assert build(**{f"{P}push_key": "~/k"}).push_key == Path("~/k").expanduser()
def test_a_push_token_needs_its_user_and_excludes_a_key() -> None: def test_a_push_token_needs_its_user_and_excludes_a_key() -> None:
config = build(gitea__push_user="pusher", gitea__push_token="${TEST_GITEA_TOKEN}") token = {f"{P}push_user": "pusher", f"{P}push_token": "${TEST_GITEA_TOKEN}"}
config = build(**token)
assert (config.push_user, config.push_token) == ("pusher", "gt") assert (config.push_user, config.push_token) == ("pusher", "gt")
with pytest.raises(ConfigError, match="set both or neither"): with pytest.raises(ConfigError, match="set both or neither"):
build(gitea__push_user="pusher") build(**{f"{P}push_user": "pusher"})
with pytest.raises(ConfigError, match="not both"): with pytest.raises(ConfigError, match="not both"):
build(gitea__push_user="p", gitea__push_token="t", gitea__push_key="~/k") build(**token, **{f"{P}push_key": "~/k"})
def test_no_credential_appears_in_the_config_repr() -> None: def test_no_credential_appears_in_the_config_repr() -> None:
config = build(gitea__push_user="pusher", gitea__push_token="${TEST_GITEA_TOKEN}") config = build(
**{f"{P}push_user": "pusher", f"{P}push_token": "${TEST_GITEA_TOKEN}"},
**{
f"{P}secrets": {"DB": {"value": "${TEST_LLM_TOKEN}", "visible": "commands"}}
},
)
text = repr(config) text = repr(config)
for secret in SECRETS.values(): for secret in SECRETS.values():
assert secret not in text assert secret not in text
@@ -376,3 +391,116 @@ def test_the_loop_defaults_to_old_and_takes_new() -> None:
assert build(loop="new").loop == "new" assert build(loop="new").loop == "new"
with pytest.raises(ConfigError, match="loop: must be old or new"): with pytest.raises(ConfigError, match="loop: must be old or new"):
build(loop="newer") build(loop="newer")
# --- projects (D8) -----------------------------------------------------------
SECOND = {
"repo": "don/other",
"workspace": "~/demo/other",
"build_host": {"address": "build2.test", "key": "~/.ssh/nisse-dev"},
}
def test_a_second_project_is_a_second_entry() -> None:
config = build(projects__other=SECOND)
assert [p.name for p in config.projects] == ["greet", "other"]
other = config.projects[1]
assert (other.repo, other.build_host.address) == ("don/other", "build2.test")
assert other.base_branch == "main" and other.labels == () and other.secrets == {}
def test_the_old_loop_refuses_more_than_one_project() -> None:
config = build(projects__other=SECOND)
with pytest.raises(ConfigError, match="exactly one project"):
config.only_project()
with pytest.raises(ConfigError, match="exactly one project"):
_ = config.gitea_repo
def test_at_least_one_project_is_needed() -> None:
with pytest.raises(ConfigError, match="projects: must name at least one"):
build(projects={})
def test_project_problems_are_named_by_project() -> None:
bad = tree(projects__other={**SECOND, "repo": "don/demo", "colour": "red"})
with pytest.raises(ConfigError) as excinfo:
AgentConfig.from_tree(bad, SECRETS)
message = str(excinfo.value)
assert "projects.other.colour: unknown setting" in message
assert "projects.other.repo: don/demo is also project greet's" in message
def test_a_project_name_is_a_slug() -> None:
with pytest.raises(ConfigError, match="lowercase letters, digits"):
build(projects__Other_Project=SECOND)
def test_labels_and_egress_are_lists_of_names() -> None:
config = build(**{f"{P}labels": ["discuss"], f"{P}egress": ["pypi.org"]})
assert config.only_project().labels == ("discuss",)
assert config.only_project().egress == ("pypi.org",)
with pytest.raises(ConfigError, match="projects.greet.labels: must be a list"):
build(**{f"{P}labels": "discuss"})
def test_secrets_are_by_reference_with_a_visibility() -> None:
config = build(
**{
f"{P}secrets": {
"DB_URL": {"value": "${TEST_LLM_TOKEN}", "visible": "commands"},
"DEPLOY": {"value": "${TEST_GITEA_TOKEN}", "visible": "harness"},
}
}
)
project = config.only_project()
assert project.command_secrets() == {"DB_URL": "sk-x"}
assert project.secrets["DEPLOY"].visible == "harness"
def test_a_secret_written_inline_is_refused() -> None:
with pytest.raises(ConfigError, match="must be a \\$\\{VAR\\} reference"):
build(**{f"{P}secrets": {"DB": {"value": "hunter2", "visible": "commands"}}})
def test_a_secret_needs_a_known_visibility() -> None:
with pytest.raises(ConfigError, match="visible: must be one of commands, harness"):
build(
**{f"{P}secrets": {"DB": {"value": "${TEST_LLM_TOKEN}", "visible": "all"}}}
)
def test_a_workspace_of_any_project_inside_is_refused(tmp_path: Path) -> None:
work = tmp_path / "other"
with pytest.raises(ConfigError, match="project other"):
build(
state_db=str(work / "state.db"),
projects__other={**SECOND, "workspace": str(work)},
)
# --- roles and profiles (L9) -------------------------------------------------
def test_the_new_loops_roles_fall_back() -> None:
config = build()
assert config.roles.resolve("driver") == "claude-opus-5" # the planner's
assert config.roles.resolve("classifier") == "claude-haiku-4-5" # the expert's
named = build(roles__driver="qwen3.8-27b", roles__classifier="gemma-4-12b")
assert named.roles.resolve("driver") == "qwen3.8-27b"
assert named.roles.pinned()["classifier"] == "gemma-4-12b"
def test_capability_profiles_are_read() -> None:
config = build(
profiles={"qwen3.8-27b": {"tools": "reliable", "weaknesses": "speed"}}
)
profile = config.profiles["qwen3.8-27b"]
assert profile.describe() == "tool calls: reliable; weak at speed"
assert build().profiles == {}
def test_a_bad_profile_is_reported() -> None:
with pytest.raises(ConfigError, match="profiles.m.colour: unknown setting"):
build(profiles={"m": {"colour": "red"}})
+83
View File
@@ -14,6 +14,7 @@ from nisse_dev.canonical import (
from nisse_dev.envfile import load_env_file from nisse_dev.envfile import load_env_file
from nisse_dev.llm import ( from nisse_dev.llm import (
DEFAULT_MAX_OUTPUT_TOKENS, DEFAULT_MAX_OUTPUT_TOKENS,
OUTAGE_BACKOFF_S,
RATE_LIMIT_RETRY_S, RATE_LIMIT_RETRY_S,
AnthropicPassthroughClient, AnthropicPassthroughClient,
BudgetExhausted, BudgetExhausted,
@@ -197,6 +198,10 @@ def test_role_map_resolves_and_pins() -> None:
"planner": "anthropic/claude-opus-5", "planner": "anthropic/claude-opus-5",
"helper": "anthropic/claude-opus-5", "helper": "anthropic/claude-opus-5",
"expert": "anthropic/claude-opus-5", "expert": "anthropic/claude-opus-5",
# Unset, the new loop's roles fall back: driver to the planner's
# model, classifier to the expert's (L9).
"driver": "anthropic/claude-opus-5",
"classifier": "anthropic/claude-opus-5",
} }
@@ -428,3 +433,81 @@ def test_the_answering_model_is_recorded() -> None:
) )
client.complete(EVENTS, tools=None, model="local-fast") client.complete(EVENTS, tools=None, model="local-fast")
assert client.last_model == "gpt-test-1" assert client.last_model == "gpt-test-1"
# --- outages delay, never fail (L5) --------------------------------------------
def test_a_gateway_error_is_waited_out_with_backoff() -> None:
responses = [
httpx.Response(502, text="bad gateway"),
httpx.Response(503, text="unavailable"),
completion({"content": "back"}),
]
waited: list[float] = []
client = make_client(lambda request: responses.pop(0), sleep=waited.append)
[event] = client.complete(EVENTS, tools=None, model="local/qwen")
assert isinstance(event, AssistantText) and event.text == "back"
assert waited == list(OUTAGE_BACKOFF_S[:2])
# Waiting isn't the model's time.
assert client.last_rate_limit_wait_s == sum(OUTAGE_BACKOFF_S[:2])
def test_an_outage_that_outlasts_the_backoff_is_surfaced() -> None:
attempts: list[int] = []
waited: list[float] = []
def handler(request: httpx.Request) -> httpx.Response:
attempts.append(1)
return httpx.Response(502, text="bad gateway")
client = make_client(handler, sleep=waited.append)
with pytest.raises(UpstreamUnavailable, match="502"):
client.complete(EVENTS, tools=None, model="local/qwen")
assert len(attempts) == len(OUTAGE_BACKOFF_S) + 1
assert waited == list(OUTAGE_BACKOFF_S)
def test_a_dropped_connection_is_retried() -> None:
calls: list[int] = []
def handler(request: httpx.Request) -> httpx.Response:
calls.append(1)
if len(calls) == 1:
raise httpx.ConnectError("refused")
return completion({"content": "ok"})
[event] = make_client(handler).complete(EVENTS, tools=None, model="local/qwen")
assert isinstance(event, AssistantText) and len(calls) == 2
def test_a_500_or_a_timeout_is_not_retried() -> None:
# A 500 may be the request's fault; a timeout already had its full
# request_timeout_s.
for failure in (
lambda request: httpx.Response(500, text="boom"),
lambda request: (_ for _ in ()).throw(httpx.ReadTimeout("slow")),
):
calls: list[int] = []
def handler(
request: httpx.Request, failure=failure, calls=calls
) -> httpx.Response:
calls.append(1)
return failure(request)
with pytest.raises(LLMError):
make_client(handler).complete(EVENTS, tools=None, model="local/qwen")
assert len(calls) == 1
def test_a_budget_refusal_is_not_waited_on() -> None:
calls: list[int] = []
def handler(request: httpx.Request) -> httpx.Response:
calls.append(1)
return httpx.Response(503, text="Budget has been exceeded! Current cost: 5.1")
with pytest.raises(BudgetExhausted):
make_client(handler).complete(EVENTS, tools=None, model="local/qwen")
assert len(calls) == 1
+11 -1
View File
@@ -50,6 +50,14 @@ class FakeGitea:
self.closed: list[int] = [] self.closed: list[int] = []
self.deps: dict[int, list[int]] = {} self.deps: dict[int, list[int]] = {}
def create_issue(self, repo: str, title: str, body: str) -> int:
n = max([i.number for i in self.issues] + [0]) + 1
self.issues.append(Issue(n, title, body, "nisse-dev", ()))
return n
def assign_issue(self, repo: str, index: int, assignee: str) -> None:
pass
def open_dependencies(self, repo: str, index: int) -> list[str]: def open_dependencies(self, repo: str, index: int) -> list[str]:
return [f"{repo}#{n}" for n in self.deps.get(index, []) if n not in self.closed] return [f"{repo}#{n}" for n in self.deps.get(index, []) if n not in self.closed]
@@ -206,7 +214,9 @@ def test_go_starts_the_work_and_close_item_closes_the_issue(h: Harness) -> None:
def test_a_go_for_an_earlier_revision_is_not_applied(h: Harness) -> None: def test_a_go_for_an_earlier_revision_is_not_applied(h: Harness) -> None:
h.model.turns.append([call("restate", text="revised: in a new module")]) h.model.turns.append(
[call("restate", text="revised: in a new module", changed="a new module")]
)
h.agent.intake() h.agent.intake()
h.say("put it in its own module") # changes, so the driver restates rev-2 h.say("put it in its own module") # changes, so the driver restates rev-2
assert loop_of(h.item()).rev == 2 assert loop_of(h.item()).rev == 2
+239
View File
@@ -0,0 +1,239 @@
"""Conversations on the new loop (M2.5, PR A): the `general` scope, a
thread per conversation, idle close and continuation (L1, L5), and agreed
work filed as a record issue whose item starts as agreed (D1, L8). Real git
on the throwaway build host; a fake Gitea."""
import re
from typing import Any
from test_loop_milestones import DON, Forge, ScriptedModel, call, loop_agent
from test_loop_milestones import world as world # noqa: PLC0414 -- the fixture
from nisse_dev.canonical import Fact, ToolResult
from nisse_dev.gitea import Issue
from nisse_dev.loop.agent import FILE_ISSUE
from nisse_dev.loop.items import CHAT, GO_AHEAD, MERGE, WAITING, loop_of
PROPOSAL = call(
"propose_work",
project="sandbox",
title="Add count_lines",
summary="Add count_lines(text) to text.py, counting newline-separated lines, "
"with tests. First step: the function and its tests.",
)
def conversations(world: dict[str, Any]) -> list[Any]:
return [i for i in world["store"].work_items() if i.scope == "general"]
def facts(world: dict[str, Any], item: Any) -> list[str]:
return [
e.event.text
for e in world["store"].transcript(item.id, "orchestrator").read()
if isinstance(e.event, Fact)
]
def quiet(world: dict[str, Any]) -> None:
"""No assigned issue: only conversations in these tests."""
world["forge"].issues.clear()
def test_a_top_level_message_is_a_conversation_in_its_own_thread(
world: dict[str, Any],
) -> None:
quiet(world)
model = ScriptedModel(
[call("reply", text="A 4090 runs a 27B model at 4-bit comfortably.")],
[call("reply", text="Yes, with a smaller context.")],
)
agent, transport = loop_agent(world, model)
asked = transport.receive(DON, "Could a 4090 run a 27B model?")
[chat] = conversations(world)
loop = loop_of(chat)
assert loop.thread == asked # the thread is under the operator's message
assert (chat.position, loop.wait) == (WAITING, CHAT)
reply = transport.outbound[-1]
assert reply.thread == asked and "4-bit" in reply.text
assert 'conversation "Could a 4090 run a 27B model?": talking' in agent.status()
transport.receive(DON, "And a 32B?", thread=asked)
assert len(conversations(world)) == 1 # the same conversation
assert transport.outbound[-1].text == "Yes, with a smaller context."
def test_agreed_work_is_filed_and_starts_as_agreed(world: dict[str, Any]) -> None:
quiet(world)
forge: Forge = world["forge"]
model = ScriptedModel(
[PROPOSAL],
[ # the filed item's first run: straight to work, no restatement
call("start_milestone", name="count_lines"),
call("write_file", path="lines.py", content="L = 1\n"),
call("finish_milestone", summary="count_lines", final="yes"),
],
)
_, transport = loop_agent(world, model)
root = transport.receive(DON, "I'd like a line counter in the sandbox.")
assert transport.outbound[-1].text.startswith(
'rev-1: I\'d open an issue in don/sandbox, "Add count_lines"'
)
transport.receive(DON, "go", thread=root)
[filed] = forge.issues
marker = re.search(r"<!-- nisse-item: ([\w-]+) -->", filed.body)
assert (
marker and "count newline-separated lines" not in filed.body
) # it's the summary
assert "counting newline-separated lines" in filed.body
assert forge.assigned == [filed.number]
work = world["store"].work_item(marker.group(1))
assert work is not None and work.scope == "sandbox"
assert loop_of(work).issue == filed.number and loop_of(work).started
assert (work.position, loop_of(work).wait) == (WAITING, MERGE) # it worked
[pr] = forge.prs
assert f"Closes #{filed.number}" in forge.prs[pr]["body"]
# The conversation says where the work went and goes on.
[chat] = conversations(world)
assert loop_of(chat).became == (f"don/sandbox#{filed.number}",)
assert loop_of(chat).wait == CHAT
assert any("filed from the conversation" in f for f in facts(world, work))
def test_intake_never_takes_a_record_issue(world: dict[str, Any]) -> None:
quiet(world)
forge: Forge = world["forge"]
body = "brief\n\n<!-- nisse-item: sandbox-cabc123 -->"
forge.issues.append(Issue(30, "a record", body, "nisse-dev", ()))
agent, _ = loop_agent(world, ScriptedModel())
agent.intake()
assert world["store"].work_items() == []
def test_a_proposal_for_an_unknown_project_is_sent_back(world: dict[str, Any]) -> None:
quiet(world)
model = ScriptedModel(
[call("propose_work", project="nope", title="t", summary="s.")],
[call("reply", text="Which project should it go in?")],
)
_, transport = loop_agent(world, model)
transport.receive(DON, "Let's build a thing")
results = [
e.event.result
for e in world["store"].transcript(conversations(world)[0].id, "driver").read()
if isinstance(e.event, ToolResult)
]
assert any("no project 'nope': the projects are sandbox" in r for r in results)
assert loop_of(conversations(world)[0]).wait == CHAT
def test_changes_to_a_proposal_go_back_to_the_driver(world: dict[str, Any]) -> None:
quiet(world)
model = ScriptedModel([PROPOSAL], [PROPOSAL])
_, transport = loop_agent(world, model)
root = transport.receive(DON, "a line counter please")
transport.receive(DON, "call it line_count instead", thread=root)
loop = loop_of(conversations(world)[0])
assert (loop.wait, loop.rev) == (GO_AHEAD, 2)
assert world["forge"].issues == []
def test_an_idle_conversation_closes_and_a_reply_continues_it(
world: dict[str, Any],
) -> None:
quiet(world)
model = ScriptedModel(
[call("reply", text="Sure.")], [call("reply", text="Picking up again.")]
)
agent, transport = loop_agent(world, model)
clock = [1000.0]
agent._clock = lambda: clock[0]
agent._idle = 3600
root = transport.receive(DON, "thinking about a backup tool")
clock[0] += 3599
agent.intake()
assert conversations(world)[0].status == "open"
clock[0] += 1
agent.intake()
[old] = conversations(world)
assert (old.status, old.outcome) == ("closed", "idle")
transport.receive(DON, "back to that backup tool", thread=root)
new = [c for c in conversations(world) if c.status == "open"]
assert len(new) == 1 and loop_of(new[0]).follows == old.id
assert loop_of(new[0]).thread == root
assert any("this continues the conversation" in f for f in facts(world, new[0]))
def test_a_plain_yes_files_the_work(world: dict[str, Any]) -> None:
quiet(world)
class Reader:
def agrees(self, proposal: str, reply: str, model: str = "") -> bool:
return reply == "sounds right"
model = ScriptedModel([PROPOSAL], [call("ask_human", question="anything else?")])
agent, transport = loop_agent(world, model)
agent._reader = Reader()
root = transport.receive(DON, "a line counter please")
transport.receive(DON, "sounds right", thread=root)
assert len(world["forge"].issues) == 1
def test_a_filing_cut_off_by_a_crash_is_settled_at_boot(world: dict[str, Any]) -> None:
quiet(world)
forge: Forge = world["forge"]
model = ScriptedModel([PROPOSAL])
_, transport = loop_agent(world, model)
root = transport.receive(DON, "a line counter please")
chat = conversations(world)[0]
store = world["store"]
# The issue was opened, then the agent died before its item existed.
intent = store.begin_intent(
FILE_ISSUE,
{"repo": "don/sandbox", "item": "sandbox-cdead01", "title": "Add count_lines"},
cause="filing",
work_item=chat.id,
)
forge.issues.append(
Issue(
40, "Add count_lines", "b\n\n<!-- nisse-item: sandbox-cdead01 -->", "x", ()
)
)
agent2, _ = loop_agent(world, ScriptedModel([call("ask_human", question="ok?")]))
agent2.boot()
work = store.work_item("sandbox-cdead01")
assert work is not None and loop_of(work).issue == 40 and loop_of(work).started
assert not [i for i in store.pending_intents() if i.id == intent]
assert root # the conversation itself is untouched
def test_a_filing_that_never_reached_gitea_is_put_up_again(
world: dict[str, Any],
) -> None:
quiet(world)
_, transport = loop_agent(world, ScriptedModel([PROPOSAL]))
transport.receive(DON, "a line counter please")
chat = conversations(world)[0]
world["store"].begin_intent(
FILE_ISSUE,
{"repo": "don/sandbox", "item": "sandbox-cnever1", "title": "Add count_lines"},
cause="filing",
work_item=chat.id,
)
agent2, _ = loop_agent(world, ScriptedModel())
agent2.boot()
loop = loop_of(world["store"].work_item(chat.id))
assert loop.wait == GO_AHEAD and "cut off by a restart" in loop.note
assert world["store"].work_item("sandbox-cnever1") is None
def test_commands_alone_at_top_level_are_not_conversations(
world: dict[str, Any],
) -> None:
quiet(world)
_, transport = loop_agent(world, ScriptedModel())
transport.receive(DON, "status")
transport.receive(DON, "go")
assert conversations(world) == []
+326
View File
@@ -0,0 +1,326 @@
"""Help for the driver on the new loop (M2.6, L9): roles pinned per item,
delegation on a branch off the milestone's (merged back on a handback,
discarded otherwise), the stuck trigger, ask_expert and its budget, and
briefings from capability profiles. Real git on the throwaway build host."""
import json
from typing import Any
from test_loop_milestones import (
RESTATE,
Forge,
ScriptedModel,
call,
git,
item,
loop_agent,
say,
)
from test_loop_milestones import world as world # noqa: PLC0414 -- the fixture
from nisse_dev.canonical import AssistantText, Fact, ToolCall, ToolResult
from nisse_dev.config import Profile
from nisse_dev.delegation import DELEGATION_INTENT, Caps
from nisse_dev.llm import RoleMap
from nisse_dev.loop.help import Helpers
from nisse_dev.loop.items import ANSWER, MERGE, loop_of
ROLES = RoleMap(executor="", planner="drv", helper="hlp", expert="exp", driver="drv")
class Spy:
"""A scripted model that records what each call was told."""
def __init__(self, *turns: list[Any]) -> None:
self.turns = list(turns)
self.calls: list[tuple[str, str]] = [] # (model, system)
def complete(self, events, tools, model, system=None) -> list[Any]:
self.calls.append((model, system or ""))
assert self.turns, f"{model} was called more often than scripted"
return self.turns.pop(0)
def hand_back(succeeded: bool = True) -> ToolCall:
return ToolCall(
call_id=f"hb-{succeeded}",
tool="hand_back",
arguments={
"changed": "wrote helped.py",
"now_true": "the helper's part is done",
"next": "finish the milestone",
"succeeded": succeeded,
},
)
def helpers(helper: Spy, expert: Spy | None = None, **kw: Any) -> Helpers:
return Helpers(
client=expert or Spy(), # type: ignore[arg-type]
roles=ROLES,
client_for=lambda model: helper, # type: ignore[arg-type,return-value]
caps=Caps(turns=5),
profiles={
"hlp": Profile(tools="reliable", strengths="tricky tests"),
"drv": Profile(tools="fair", weaknesses="long refactors"),
},
**kw,
)
def driver_results(world: dict[str, Any], tool: str) -> list[str]:
events = world["store"].transcript(item(world).id, "driver").read()
ids = {
e.event.call_id
for e in events
if isinstance(e.event, ToolCall) and e.event.tool == tool
}
return [
e.event.result
for e in events
if isinstance(e.event, ToolResult) and e.event.call_id in ids
]
def local_branches(world: dict[str, Any]) -> list[str]:
return world["repo"].local_branches()
def test_a_helpers_work_lands_on_the_milestone(world: dict[str, Any]) -> None:
forge: Forge = world["forge"]
helper = Spy(
[call("write_file", path="helped.py", content="H = 1\n")],
[hand_back()],
)
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="words"),
call("delegate", task="write helped.py with H = 1"),
call("finish_milestone", summary="with help", final="yes"),
],
)
agent, transport = loop_agent(world, model, helpers=helpers(helper))
agent.intake()
say(transport, world, "go")
[result] = driver_results(world, "delegate")
assert result.startswith("the helper finished:") and "helped.py" in result
assert helper.calls[0][0] == "hlp" # the helper role's model
# Briefed about the driver by its profile, never by name.
assert "tool calls: fair; weak at long refactors" in helper.calls[0][1]
[pr] = forge.prs
assert forge.files(pr) == ["helped.py"]
assert not [b for b in local_branches(world) if "-help-" in b]
assert loop_of(item(world)).wait == MERGE
def test_a_failed_helpers_work_is_discarded(world: dict[str, Any]) -> None:
helper = Spy(
[call("write_file", path="half.py", content="half\n")],
[hand_back(succeeded=False)],
)
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="words"),
call("write_file", path="mine.py", content="M = 1\n"),
call("delegate", task="write half.py"),
call("ask_human", question="the helper couldn't; what now?"),
],
)
agent, transport = loop_agent(world, model, helpers=helpers(helper))
agent.intake()
say(transport, world, "go")
[result] = driver_results(world, "delegate")
assert "didn't finish" in result and "discarded" in result
root = world["repo"].root
assert not (root / "half.py").exists()
assert (root / "mine.py").exists() # the driver's own work stays
assert world["repo"].current_branch() == loop_of(item(world)).branch
assert not [b for b in local_branches(world) if "-help-" in b]
assert not [
i for i in world["store"].pending_intents() if i.kind == DELEGATION_INTENT
]
def test_delegating_needs_a_milestone(world: dict[str, Any]) -> None:
model = ScriptedModel(
RESTATE,
[
call("delegate", task="anything"),
call("ask_human", question="ok?"),
],
)
agent, transport = loop_agent(world, model, helpers=helpers(Spy()))
agent.intake()
say(transport, world, "go")
assert driver_results(world, "delegate") == [
"start a milestone first: a helper works on its branch"
]
def test_a_stuck_driver_gets_a_helper_once_per_milestone(
world: dict[str, Any],
) -> None:
forge: Forge = world["forge"]
helper = Spy(
[call("write_file", path="fixed.py", content="F = 1\n")],
[hand_back()],
)
failing = [
call("run_command", command="false"),
call("run_command", command="false"),
]
model = ScriptedModel(
RESTATE,
[call("start_milestone", name="words"), *failing],
[call("finish_milestone", summary="fixed with help", final="yes")],
)
agent, transport = loop_agent(world, model, helpers=helpers(helper))
agent._driver._failure_limit = 2
agent.intake()
say(transport, world, "go")
assert any("so a helper is taking over" in m.text for m in transport.outbound)
assert "The driver got stuck" in str(helper.calls) # in the brief
[pr] = forge.prs
assert forge.files(pr) == ["fixed.py"]
assert loop_of(item(world)).helped_at == 1
def test_stuck_again_in_the_same_milestone_asks_the_operator(
world: dict[str, Any],
) -> None:
helper = Spy(
[call("write_file", path="tried.py", content="T = 1\n")],
[hand_back()],
)
failing = [
call("run_command", command="false"),
call("run_command", command="false"),
]
model = ScriptedModel(
RESTATE,
[call("start_milestone", name="words"), *failing],
failing, # after the helper: stuck again
)
agent, transport = loop_agent(world, model, helpers=helpers(helper))
agent._driver._failure_limit = 2
agent.intake()
say(transport, world, "go")
loop = loop_of(item(world))
assert loop.wait == ANSWER and "commands failed in a row" in loop.note
assert len(helper.calls) == 2 # one delegation: its two turns
def test_ask_expert_answers_within_its_budget(world: dict[str, Any]) -> None:
answer = {"answer": "use a dict", "confidence": "high", "caveats": ""}
expert = Spy(
[AssistantText(text=json.dumps(answer), model="exp")],
[AssistantText(text=json.dumps(answer), model="exp")],
)
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="words"),
call("ask_expert", question="dict or list?"),
call("ask_expert", question="and again?"),
call("ask_expert", question="once more?"),
call("ask_human", question="ok?"),
],
)
agent, transport = loop_agent(
world, model, helpers=helpers(Spy(), expert, expert_budget=2)
)
agent.intake()
say(transport, world, "go")
first, second, third = driver_results(world, "ask_expert")
assert first.startswith("use a dict") and second.startswith("use a dict")
assert "budget" in third
assert expert.calls[0][0] == "exp"
def test_roles_are_pinned_when_the_item_opens(world: dict[str, Any]) -> None:
spy = Spy(RESTATE)
agent, _ = loop_agent(world, spy, helpers=helpers(Spy())) # type: ignore[arg-type]
agent._roles = {**ROLES.pinned(), "driver": "pinned-driver"}
agent.intake()
assert spy.calls[0][0] == "pinned-driver"
pinned = item(world).data["roles"]
assert pinned["driver"] == "pinned-driver" and pinned["classifier"] == "exp"
def test_the_driver_is_told_about_its_helpers_by_profile(
world: dict[str, Any],
) -> None:
spy = Spy(RESTATE, [call("ask_human", question="ok?")])
agent, transport = loop_agent(world, spy, helpers=helpers(Spy())) # type: ignore[arg-type]
agent.intake()
say(transport, world, "go")
after_go = spy.calls[1][1]
assert "Asking for help is normal" in after_go
assert "tool calls: reliable; strong at tricky tests" in after_go
assert "hlp" not in after_go # never a model's name
def test_a_delegation_cut_off_by_a_crash_is_discarded_at_boot(
world: dict[str, Any],
) -> None:
model = ScriptedModel(
RESTATE,
[call("start_milestone", name="words"), call("ask_human", question="?")],
)
agent, transport = loop_agent(world, model, helpers=helpers(Spy()))
agent.intake()
say(transport, world, "go")
milestone = loop_of(item(world)).branch
repo = world["repo"]
repo.checkout_new_branch(f"{milestone}-help-1")
(repo.root / "partial.py").write_text("p\n")
repo.commit_all("partial")
world["store"].begin_intent(
DELEGATION_INTENT,
{"branch": f"{milestone}-help-1", "base": milestone},
cause="delegating",
work_item=item(world).id,
)
agent2, _ = loop_agent(world, ScriptedModel(), helpers=helpers(Spy()))
agent2.boot()
assert repo.current_branch() == milestone
assert not (repo.root / "partial.py").exists()
assert f"{milestone}-help-1" not in repo.local_branches()
assert not [
i for i in world["store"].pending_intents() if i.kind == DELEGATION_INTENT
]
facts = [
e.event.text
for e in world["store"].transcript(item(world).id, "orchestrator").read()
if isinstance(e.event, Fact)
]
assert any("cut off by a restart and discarded" in f for f in facts)
assert git
def test_a_helpers_progress_is_posted(world: dict[str, Any]) -> None:
helper = Spy(
[
AssistantText(text="Writing helped.py with the constant.", model="hlp"),
call("write_file", path="helped.py", content="H = 1\n"),
],
[hand_back()],
)
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="words"),
call("delegate", task="write helped.py"),
call("ask_human", question="done?"),
],
)
agent, transport = loop_agent(world, model, helpers=helpers(helper))
agent.intake()
say(transport, world, "go")
said = [m.text for m in transport.outbound]
assert "helper: Writing helped.py with the constant." in said
+41 -5
View File
@@ -32,7 +32,7 @@ class Reader:
self.yes = set(yes) self.yes = set(yes)
self.asked: list[tuple[str, str]] = [] self.asked: list[tuple[str, str]] = []
def agrees(self, proposal: str, reply: str) -> bool: def agrees(self, proposal: str, reply: str, model: str = "") -> bool:
self.asked.append((proposal, reply)) self.asked.append((proposal, reply))
return reply in self.yes return reply in self.yes
@@ -118,7 +118,7 @@ def test_a_plain_yes_is_a_go_ahead_said_so_first(tmp_path: Path) -> None:
def test_anything_else_goes_to_the_driver(tmp_path: Path) -> None: def test_anything_else_goes_to_the_driver(tmp_path: Path) -> None:
h = Harness( h = Harness(
tmp_path, tmp_path,
ScriptedModel(RESTATED, [call("restate", text="v2")]), ScriptedModel(RESTATED, [call("restate", text="v2", changed="as you asked")]),
FakeGitea(issue()), FakeGitea(issue()),
reader=Reader(), reader=Reader(),
) )
@@ -132,7 +132,7 @@ def test_a_question_never_reaches_the_reader(tmp_path: Path) -> None:
reader = Reader("sounds good?") reader = Reader("sounds good?")
h = Harness( h = Harness(
tmp_path, tmp_path,
ScriptedModel(RESTATED, [call("restate", text="v2")]), ScriptedModel(RESTATED, [call("restate", text="v2", changed="as you asked")]),
FakeGitea(issue()), FakeGitea(issue()),
reader=reader, reader=reader,
) )
@@ -145,7 +145,9 @@ def test_a_plain_yes_to_an_older_revision_is_not_a_go(tmp_path: Path) -> None:
reader = Reader("yes") reader = Reader("yes")
h = Harness( h = Harness(
tmp_path, tmp_path,
ScriptedModel(RESTATED, [call("restate", text="v2")], STARTED), ScriptedModel(
RESTATED, [call("restate", text="v2", changed="as you asked")], STARTED
),
FakeGitea(issue()), FakeGitea(issue()),
reader=reader, reader=reader,
) )
@@ -224,7 +226,12 @@ def test_a_reply_stops_the_bugs_countdown(tmp_path: Path) -> None:
def test_a_new_revision_of_a_bug_gets_its_own_countdown(tmp_path: Path) -> None: def test_a_new_revision_of_a_bug_gets_its_own_countdown(tmp_path: Path) -> None:
h, clock = bug(tmp_path, RESTATED, [call("restate", text="v2")], STARTED) h, clock = bug(
tmp_path,
RESTATED,
[call("restate", text="v2", changed="as you asked")],
STARTED,
)
h.agent.intake() h.agent.intake()
clock.now += 300 clock.now += 300
h.say("use the other file") h.say("use the other file")
@@ -280,3 +287,32 @@ def _msg(text: str, thread: str) -> Any:
from nisse_dev.transport import InboundMessage from nisse_dev.transport import InboundMessage
return InboundMessage(DON, text, thread=thread) return InboundMessage(DON, text, thread=thread)
def test_a_restatement_that_asks_is_sent_back(tmp_path: Path) -> None:
"""A restatement is agreed to as it stands; one ending in a question
(models add "Is this right?") never goes up."""
h = Harness(
tmp_path,
ScriptedModel(
[call("restate", text="I'll add count_words. Is this right?")],
[call("restate", text="I'll add count_words to text.py, with tests.")],
),
FakeGitea(issue()),
)
h.agent.intake()
loop = loop_of(h.item())
assert (loop.wait, loop.rev) == (GO_AHEAD, 1)
posted = [p for p in h.posted() if p.startswith("rev-")]
assert posted == [
(
"rev-1: I'll add count_words to text.py, with tests.\n\n"
"Say go to start, or reply with changes."
)
]
results = [
e.event.result
for e in h.store.transcript(h.item().id, "driver").read()
if hasattr(e.event, "result")
]
assert any("a restatement asks no questions" in r for r in results)
+60 -8
View File
@@ -14,7 +14,7 @@ from nisse_dev.cancel import Cancel
from nisse_dev.canonical import ToolCall from nisse_dev.canonical import ToolCall
from nisse_dev.execution import SSHHost from nisse_dev.execution import SSHHost
from nisse_dev.git import GitRepo from nisse_dev.git import GitRepo
from nisse_dev.gitea import Comment, Issue, PullRequest, ReviewNote from nisse_dev.gitea import Comment, GiteaError, Issue, PullRequest, ReviewNote
from nisse_dev.loop.agent import LoopAgent from nisse_dev.loop.agent import LoopAgent
from nisse_dev.loop.driver import Driver from nisse_dev.loop.driver import Driver
from nisse_dev.loop.items import ANSWER, MERGE, WAITING, loop_of from nisse_dev.loop.items import ANSWER, MERGE, WAITING, loop_of
@@ -60,11 +60,20 @@ class Forge:
self.closed_issues: list[int] = [] self.closed_issues: list[int] = []
self.issues = [Issue(14, "Add count_words", "count words", "don", ())] self.issues = [Issue(14, "Add count_words", "count words", "don", ())]
self.deps: dict[int, list[int]] = {} # issue -> the issues it depends on self.deps: dict[int, list[int]] = {} # issue -> the issues it depends on
self.assigned: list[int] = [] # issues the agent assigned to itself
# intake # intake
def assigned_issues(self, repo: str, assignee: str) -> list[Issue]: def assigned_issues(self, repo: str, assignee: str) -> list[Issue]:
return [i for i in self.issues if i.number not in self.closed_issues] return [i for i in self.issues if i.number not in self.closed_issues]
def create_issue(self, repo: str, title: str, body: str) -> int:
n = max([i.number for i in self.issues] + [0]) + 1
self.issues.append(Issue(n, title, body, "nisse-dev", ()))
return n
def assign_issue(self, repo: str, index: int, assignee: str) -> None:
self.assigned.append(index)
def open_dependencies(self, repo: str, index: int) -> list[str]: def open_dependencies(self, repo: str, index: int) -> list[str]:
return [ return [
f"{repo}#{n}" f"{repo}#{n}"
@@ -115,6 +124,32 @@ class Forge:
def close_pull_request(self, repo: str, index: int) -> None: def close_pull_request(self, repo: str, index: int) -> None:
self.prs[index]["state"] = "closed" self.prs[index]["state"] = "closed"
# what a new project's checks read (D4)
protected = False
def protect(self) -> None:
"""Protection as the server enforces it: pushes to main refused by a
hook; merging needs an approval nobody here gives."""
hook = self.bare / "hooks" / "pre-receive"
hook.write_text(
"#!/bin/sh\nwhile read old new ref; do\n"
' [ "$ref" = refs/heads/main ] && echo "main is protected" && exit 1\n'
"done\nexit 0\n"
)
hook.chmod(0o755)
self.protected = True
def repository(self, repo: str) -> dict[str, Any]:
return {"permissions": {"push": True}}
def branch(self, repo: str, name: str) -> dict[str, Any]:
return {"protected": self.protected, "user_can_push": not self.protected}
def merge_pull_request(self, repo: str, index: int) -> None:
if self.protected:
raise GiteaError("gitea POST merge returned 405: approvals needed")
self.merge(index)
# what the operator does in Gitea # what the operator does in Gitea
def merge(self, n: int) -> None: def merge(self, n: int) -> None:
head = f"refs/heads/{self.prs[n]['head']}" head = f"refs/heads/{self.prs[n]['head']}"
@@ -157,6 +192,18 @@ class Forge:
@pytest.fixture @pytest.fixture
def world(ssh_host: SSHHost, tmp_path: Path) -> dict[str, Any]: def world(ssh_host: SSHHost, tmp_path: Path) -> dict[str, Any]:
return make_world(ssh_host, tmp_path)
def make_world(
ssh_host: SSHHost,
tmp_path: Path,
store: StateStore | None = None,
repo_name: str = "don/sandbox",
) -> dict[str, Any]:
"""A project's forge (a bare repository and a fake Gitea over it), its
clone and milestones; a second one can share the first's store."""
tmp_path.mkdir(parents=True, exist_ok=True)
bare = tmp_path / "forge.git" bare = tmp_path / "forge.git"
git("init", "-q", "--bare", "-b", "main", str(bare)) git("init", "-q", "--bare", "-b", "main", str(bare))
seed = tmp_path / "seed" seed = tmp_path / "seed"
@@ -179,13 +226,16 @@ def world(ssh_host: SSHHost, tmp_path: Path) -> dict[str, Any]:
repo.clone(str(bare)) repo.clone(str(bare))
repo.configure_identity("nisse-dev", "n@t") repo.configure_identity("nisse-dev", "n@t")
repo.install_excludes() repo.install_excludes()
store = StateStore(tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO())) if store is None:
store = StateStore(
tmp_path / "state.db", oplog=OpLog("nisse-dev", io.StringIO())
)
forge = Forge(bare) forge = Forge(bare)
milestones = Milestones( milestones = Milestones(
store, store,
repo, repo,
forge, forge, # type: ignore[arg-type]
repo="don/sandbox", # type: ignore[arg-type] repo=repo_name,
remote=str(bare), remote=str(bare),
credential=None, credential=None,
) )
@@ -197,15 +247,16 @@ def world(ssh_host: SSHHost, tmp_path: Path) -> dict[str, Any]:
"host": ssh_host, "host": ssh_host,
"bare": bare, "bare": bare,
"seed": seed, "seed": seed,
"repo_name": repo_name,
} }
def loop_agent( def loop_agent(
world: dict[str, Any], model: ScriptedModel world: dict[str, Any], model: ScriptedModel, **agent: Any
) -> tuple[LoopAgent, InMemoryTransport]: ) -> tuple[LoopAgent, InMemoryTransport]:
transport = InMemoryTransport() transport = InMemoryTransport()
cancel = Cancel() cancel = Cancel()
agent = LoopAgent( loop = LoopAgent(
world["store"], world["store"],
transport, transport,
world["forge"], world["forge"],
@@ -220,9 +271,10 @@ def loop_agent(
cancel=cancel, cancel=cancel,
oplog=OpLog("nisse-dev", io.StringIO()), oplog=OpLog("nisse-dev", io.StringIO()),
milestones=world["milestones"], milestones=world["milestones"],
**agent,
) )
transport.on_message(agent.on_message) transport.on_message(loop.on_message)
return agent, transport return loop, transport
def item(world: dict[str, Any]) -> Any: def item(world: dict[str, Any]) -> Any:
+43 -1
View File
@@ -19,6 +19,7 @@ from test_loop_milestones import (
from test_loop_milestones import world as world # noqa: PLC0414 -- the fixture from test_loop_milestones import world as world # noqa: PLC0414 -- the fixture
from nisse_dev.canonical import ToolCall, ToolResult from nisse_dev.canonical import ToolCall, ToolResult
from nisse_dev.gitea import Issue
from nisse_dev.loop.items import MERGE, WAITING, loop_of from nisse_dev.loop.items import MERGE, WAITING, loop_of
@@ -94,7 +95,7 @@ def test_the_next_milestone_waits_for_the_merge(world: dict[str, Any]) -> None:
agent, transport = loop_agent(world, model) agent, transport = loop_agent(world, model)
agent.intake() agent.intake()
say(transport, world, "go") say(transport, world, "go")
say(transport, world, "how's it going?") say(transport, world, "carry on")
assert "hasn't merged yet" in result_of(world, "start_milestone") assert "hasn't merged yet" in result_of(world, "start_milestone")
assert (item(world).position, loop_of(item(world)).wait) == (WAITING, MERGE) assert (item(world).position, loop_of(item(world)).wait) == (WAITING, MERGE)
assert len(forge.prs) == 1 assert len(forge.prs) == 1
@@ -174,3 +175,44 @@ def test_an_unsaid_final_closes_the_issue(world: dict[str, Any]) -> None:
say(transport, world, "go") say(transport, world, "go")
[pr] = forge.prs [pr] = forge.prs
assert "Closes #14" in forge.prs[pr]["body"] assert "Closes #14" in forge.prs[pr]["body"]
def test_a_design_milestone_holds_documents_only(world: dict[str, Any]) -> None:
"""D1, D2: a `design` item agrees its design before any code; the
harness refuses a design milestone with code in it."""
forge: Forge = world["forge"]
forge.issues[0] = Issue(14, "Add count_words", "count words", "don", ("design",))
model = ScriptedModel(
RESTATE,
[
call("start_milestone", name="design"),
call("write_file", path="ARCHITECTURE.md", content="# arch\n"),
call("write_file", path="docs/plans/words.md", content="# plan\n"),
call("write_file", path="words.py", content="W = 1\n"), # too soon
call("finish_milestone", summary="design", final="no"),
],
[
call("run_command", command="rm words.py"),
call("finish_milestone", summary="design", final="no"),
],
milestone("words.py", final="yes"), # after the design merged
)
agent, transport = loop_agent(world, model)
agent.intake()
say(transport, world, "go")
[design] = forge.prs
assert sorted(forge.files(design)) == ["ARCHITECTURE.md", "docs/plans/words.md"]
refused = [
e.event.result
for e in world["store"].transcript(item(world).id, "driver").read()
if isinstance(e.event, ToolResult)
and "this milestone is the design" in e.event.result
]
assert refused and "words.py" in refused[0]
forge.merge(design)
agent.watch()
agent._schedule()
code = max(forge.prs)
assert code != design and forge.files(code) == ["words.py"]
assert loop_of(item(world)).design_merged
+193
View File
@@ -0,0 +1,193 @@
"""New projects on the new loop (M2.5, PR B): a conversation proposes one,
the agent lists what it needs, and on "done" reads its configuration again
and checks what was provided (D1, D4, D8) before opening the tracking issue
and starting on the architecture. Real git on the throwaway build host,
with protection enforced by a hook on the fake forge's bare repository."""
import io
import re
from pathlib import Path
from typing import Any
import pytest
import yaml
from test_loop_milestones import DON, Forge, ScriptedModel, call, make_world
from test_loop_projects import Router
from nisse_dev.cancel import Cancel
from nisse_dev.config import _parse_projects
from nisse_dev.execution import SSHHost
from nisse_dev.loop.agent import LoopAgent, Project
from nisse_dev.loop.driver import Driver
from nisse_dev.loop.items import CHAT, GO_AHEAD, MERGE, PROVISION, loop_of
from nisse_dev.oplog import OpLog
from nisse_dev.tools import Workspace
from nisse_dev.transport import InMemoryTransport
from nisse_dev.worker import InlineWorker
PROPOSAL = call(
"propose_project",
name="tally",
purpose="A small command-line tally counter.",
decisions="Python 3.12 with pytest; no services, secrets or extra egress.",
)
@pytest.fixture
def worlds(ssh_host: SSHHost, tmp_path: Path) -> dict[str, Any]:
sandbox = make_world(ssh_host, tmp_path / "sandbox")
tally = make_world(
ssh_host, tmp_path / "tally", store=sandbox["store"], repo_name="don/tally"
)
sandbox["forge"].issues.clear() # only the conversation here
tally["forge"].issues.clear()
return {"sandbox": sandbox, "tally": tally}
def agent_for(
worlds: dict[str, Any], model: ScriptedModel, *, entry_added: list[bool]
) -> tuple[LoopAgent, InMemoryTransport]:
"""An agent that knows only the sandbox; tally's entry appears in its
configuration once ``entry_added[0]`` is set."""
transport = InMemoryTransport()
cancel = Cancel()
router = Router({w["repo_name"]: w["forge"] for w in worlds.values()})
for w in worlds.values():
w["milestones"]._gitea = router
def runtime(name: str) -> Project:
w = worlds[name]
return Project(
name=name,
repo=w["repo_name"],
workspace=Workspace(w["host"], w["repo"].root, timeout_s=30),
milestones=w["milestones"],
)
def reload(known: set[str]) -> list[Project]:
names = ["sandbox", "tally"] if entry_added[0] else ["sandbox"]
return [runtime(n) for n in names if n not in known]
agent = LoopAgent(
worlds["sandbox"]["store"],
transport,
router, # type: ignore[arg-type]
projects=[runtime("sandbox")],
account="nisse-dev",
operators={DON},
gitea_operators={"don"},
driver=Driver(model, "m", cancel=cancel), # type: ignore[arg-type]
worker=InlineWorker(),
cancel=cancel,
oplog=OpLog("nisse-dev", io.StringIO()),
reload=reload,
)
transport.on_message(agent.on_message)
return agent, transport
def conversation(worlds: dict[str, Any]) -> Any:
[chat] = [
i for i in worlds["sandbox"]["store"].work_items() if i.scope == "general"
]
return chat
def proposed(
worlds: Any, model: ScriptedModel, added: list[bool]
) -> tuple[Any, Any, str]:
agent, transport = agent_for(worlds, model, entry_added=added)
root = transport.receive(DON, "I want a little tally counter, as its own project.")
assert transport.outbound[-1].text.startswith("rev-1: a new project, `tally`.")
assert loop_of(conversation(worlds)).wait == GO_AHEAD
transport.receive(DON, "go", thread=root)
return agent, transport, root
def test_an_agreed_project_gets_a_checklist_with_its_entry(worlds: Any) -> None:
_, transport, _ = proposed(worlds, ScriptedModel([PROPOSAL]), [False])
loop = loop_of(conversation(worlds))
assert (loop.wait, loop.new_project) == (PROVISION, "tally")
checklist = transport.outbound[-1].text
for need in ("repository", "Protection", "collaborator", "build VM", "deploy key"):
assert need in checklist
# The entry is valid configuration once its placeholders are filled in.
entry = re.search(r"```yaml\n(.*?)\n```", checklist, re.DOTALL)
assert entry is not None
filled = (
entry.group(1)
.replace("<owner>/tally", "don/tally")
.replace("<the build VM>", "build.test")
.replace("<my SSH key for it>", "/k")
.replace("<the deploy key's private half>", "/p")
)
errors: list[str] = []
[project] = _parse_projects(yaml.safe_load(filled)["projects"], {}, errors)
assert errors == [] and project.name == "tally" and project.repo == "don/tally"
def test_done_before_the_entry_is_added_says_so(worlds: Any) -> None:
_, transport, root = proposed(worlds, ScriptedModel([PROPOSAL]), [False])
transport.receive(DON, "done", thread=root)
assert "I don't see `tally` in my configuration yet" in transport.outbound[-1].text
assert loop_of(conversation(worlds)).new_project == "tally"
def test_what_isnt_set_up_is_named_and_nothing_starts(worlds: Any) -> None:
added = [False]
_, transport, root = proposed(worlds, ScriptedModel([PROPOSAL]), added)
added[0] = True # the entry is there, but tally's main isn't protected
transport.receive(DON, "done", thread=root)
said = transport.outbound[-1].text
assert said.startswith("Not ready yet:")
assert "main isn't protected" in said and "could merge my own PR" in said
assert loop_of(conversation(worlds)).new_project == "tally"
assert worlds["tally"]["forge"].issues == []
# The check's PR was closed and its branch deleted.
assert all(p["state"] == "closed" for p in worlds["tally"]["forge"].prs.values())
def test_a_checked_project_gets_its_record_and_starts_on_its_design(
worlds: Any,
) -> None:
added = [False]
model = ScriptedModel(
[PROPOSAL],
[ # tally's first item: the architecture and the build plan
call("start_milestone", name="architecture"),
call("write_file", path="ARCHITECTURE.md", content="# tally\n"),
call("write_file", path="docs/plans/tally.md", content="# plan\n"),
call("finish_milestone", summary="architecture and plan", final="no"),
],
)
agent, transport, root = proposed(worlds, model, added)
forge: Forge = worlds["tally"]["forge"]
forge.protect()
added[0] = True
transport.receive(DON, "done", thread=root)
assert "tally" in agent._projects
[record] = forge.issues
marker = re.search(r"<!-- nisse-item: ([\w-]+) -->", record.body)
assert marker and "tally counter" in record.body
work = worlds["sandbox"]["store"].work_item(marker.group(1))
assert work.scope == "tally" and loop_of(work).started
assert "design" in loop_of(work).labels
assert loop_of(work).wait == MERGE
pr = max(n for n, p in forge.prs.items() if p["state"] == "open")
assert sorted(forge.files(pr)) == ["ARCHITECTURE.md", "docs/plans/tally.md"]
assert f"Part of #{record.number}" in forge.prs[pr]["body"]
chat = loop_of(conversation(worlds))
assert chat.became == (f"don/tally#{record.number}",)
assert (chat.wait, chat.new_project) == (CHAT, "")
def test_a_project_name_that_exists_is_sent_back(worlds: Any) -> None:
model = ScriptedModel(
[call("propose_project", name="sandbox", purpose="again", decisions="x.")],
[call("reply", text="sandbox exists; shall I propose work on it?")],
)
agent, transport = agent_for(worlds, model, entry_added=[False])
transport.receive(DON, "a new sandbox?")
assert loop_of(conversation(worlds)).wait == CHAT
assert agent
+188
View File
@@ -0,0 +1,188 @@
"""Several projects on the new loop (M1.4, D8): intake from each project's
repository, each item working in its own project's workspace and publishing
to its own repository, issue numbers told apart, label defaults, and parking
across projects. Real git on the throwaway build host; fake Gitea per repo."""
import io
from pathlib import Path
from typing import Any
import pytest
from test_loop_milestones import DON, Forge, ScriptedModel, call, git, make_world
from nisse_dev.cancel import Cancel
from nisse_dev.execution import SSHHost
from nisse_dev.loop.agent import LoopAgent, Project
from nisse_dev.loop.driver import Driver
from nisse_dev.loop.items import ANSWER, GO_AHEAD, MERGE, item_for_issue, loop_of
from nisse_dev.oplog import OpLog
from nisse_dev.tools import Workspace
from nisse_dev.transport import InMemoryTransport
from nisse_dev.worker import InlineWorker
REPOS = {"alpha": "don/alpha", "beta": "don/beta"}
class Router:
"""One Gitea for the agent, routing each call to its repository's fake."""
def __init__(self, forges: dict[str, Forge]) -> None:
self.forges = forges
def __getattr__(self, name: str) -> Any:
def call(repo: str, *args: Any, **kwargs: Any) -> Any:
return getattr(self.forges[repo], name)(repo, *args, **kwargs)
return call
@pytest.fixture
def worlds(ssh_host: SSHHost, tmp_path: Path) -> dict[str, dict[str, Any]]:
alpha = make_world(ssh_host, tmp_path / "alpha", repo_name=REPOS["alpha"])
beta = make_world(
ssh_host, tmp_path / "beta", store=alpha["store"], repo_name=REPOS["beta"]
)
return {"alpha": alpha, "beta": beta}
def agent_for(
worlds: dict[str, dict[str, Any]],
model: ScriptedModel,
labels: dict[str, tuple[str, ...]] | None = None,
) -> tuple[LoopAgent, InMemoryTransport]:
transport = InMemoryTransport()
cancel = Cancel()
router = Router({REPOS[n]: w["forge"] for n, w in worlds.items()})
projects = [
Project(
name=n,
repo=REPOS[n],
workspace=Workspace(w["host"], w["repo"].root, timeout_s=30),
milestones=w["milestones"],
labels=(labels or {}).get(n, ()),
)
for n, w in worlds.items()
]
for w in worlds.values():
w["milestones"]._gitea = router # publishing goes through the router too
agent = LoopAgent(
worlds["alpha"]["store"],
transport,
router, # type: ignore[arg-type]
projects=projects,
account="nisse-dev",
operators={DON},
gitea_operators={"don"},
driver=Driver(model, "m", cancel=cancel), # type: ignore[arg-type]
worker=InlineWorker(),
cancel=cancel,
oplog=OpLog("nisse-dev", io.StringIO()),
)
transport.on_message(agent.on_message)
return agent, transport
def item(worlds: dict[str, Any], project: str) -> Any:
found = item_for_issue(worlds["alpha"]["store"], 14, project)
assert found is not None
return found
def say(transport: InMemoryTransport, worlds: Any, project: str, text: str) -> None:
transport.receive(DON, text, thread=loop_of(item(worlds, project)).thread)
def restate(name: str) -> list[Any]:
return [call("restate", text=f"{name}: add count_words")]
def work(path: str) -> list[Any]:
return [
call("start_milestone", name=path),
call("write_file", path=path, content=f"# {path}\n"),
call("finish_milestone", summary=path, final="yes"),
]
def test_both_projects_issues_are_taken_and_told_apart(worlds: Any) -> None:
# Both forges have an issue #14.
agent, transport = agent_for(worlds, ScriptedModel(restate("a"), restate("b")))
agent.intake()
a, b = item(worlds, "alpha"), item(worlds, "beta")
assert (a.scope, b.scope) == ("alpha", "beta")
said = [m.text for m in transport.outbound]
assert any(t.startswith("Queued don/alpha#14") for t in said)
assert any(t.startswith("Queued don/beta#14") for t in said)
status = agent.status()
assert "alpha#14 Add count_words" in status and "beta#14 Add count_words" in status
def test_each_item_works_and_publishes_in_its_own_project(worlds: Any) -> None:
model = ScriptedModel(restate("a"), restate("b"), work("a.py"), work("b.py"))
agent, transport = agent_for(worlds, model)
agent.intake()
say(transport, worlds, "alpha", "go")
say(transport, worlds, "beta", "go")
fa, fb = worlds["alpha"]["forge"], worlds["beta"]["forge"]
[pa], [pb] = fa.prs, fb.prs
assert fa.files(pa) == ["a.py"] and fb.files(pb) == ["b.py"]
assert fa.prs[pa]["head"].startswith(item(worlds, "alpha").id)
assert fb.prs[pb]["head"].startswith(item(worlds, "beta").id)
# Each workspace has only its own project's work.
assert not (worlds["alpha"]["repo"].root / "b.py").exists()
assert loop_of(item(worlds, "beta")).wait == MERGE
def test_a_bare_number_in_two_projects_is_asked_about(worlds: Any) -> None:
model = ScriptedModel(restate("a"), restate("b"), work("b.py"))
agent, transport = agent_for(worlds, model)
agent.intake()
transport.receive(DON, "go #14")
asked = transport.outbound[-1].text
assert "open in more than one project" in asked
assert "go alpha#14" in asked and "go beta#14" in asked
assert not loop_of(item(worlds, "alpha")).started
transport.receive(DON, "go beta#14")
assert loop_of(item(worlds, "beta")).started
assert not loop_of(item(worlds, "alpha")).started
def test_do_next_names_a_project(worlds: Any) -> None:
agent, transport = agent_for(worlds, ScriptedModel(restate("a"), restate("b")))
agent.intake()
transport.receive(DON, "do #14 next")
assert "do alpha#14 next or do beta#14 next" in transport.outbound[-1].text
transport.receive(DON, "do beta#14 next")
assert "beta#14 goes first" in transport.outbound[-1].text
def test_a_projects_label_defaults_steer_its_items(worlds: Any) -> None:
model = ScriptedModel(
[call("ask_human", question="what's it for?")], # alpha: discuss first
restate("b"),
)
agent, _ = agent_for(worlds, model, labels={"alpha": ("discuss",)})
agent.intake()
assert loop_of(item(worlds, "alpha")).labels == ("discuss",)
assert loop_of(item(worlds, "alpha")).wait == ANSWER
assert loop_of(item(worlds, "beta")).wait == GO_AHEAD
def test_switching_projects_parks_the_other_projects_work(worlds: Any) -> None:
model = ScriptedModel(
restate("a"),
restate("b"),
[ # alpha: half done, then a question
call("start_milestone", name="a"),
call("write_file", path="half.py", content="H = 1\n"),
call("ask_human", question="one more thing?"),
],
work("b.py"), # beta runs meanwhile
)
agent, transport = agent_for(worlds, model)
agent.intake()
say(transport, worlds, "alpha", "go")
say(transport, worlds, "beta", "go")
bare = ["--git-dir", str(worlds["alpha"]["bare"])]
parked = f"refs/heads/{item(worlds, 'alpha').id}-parked"
assert git(*bare, "show", f"{parked}^3:half.py") == "H = 1"
+279 -45
View File
@@ -9,25 +9,34 @@ the seed and the steps, so it replays exactly.
""" """
import random import random
import re
import subprocess import subprocess
import uuid
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
import pytest import pytest
from model_contract import assert_renderable from model_contract import assert_renderable
from test_loop_milestones import DON, Forge, call from test_loop_milestones import DON, Forge, call, make_world
from test_loop_milestones import world as world # noqa: PLC0414 -- the fixture from test_loop_milestones import world as world # noqa: PLC0414 -- the fixture
from test_loop_projects import Router
from nisse_dev.cancel import Cancel from nisse_dev.cancel import Cancel
from nisse_dev.canonical import Fact, PlanSubmitted, ToolCall, UserMessage from nisse_dev.canonical import Fact, PlanSubmitted, ToolCall, UserMessage
from nisse_dev.delegation import DELEGATION_INTENT, Caps
from nisse_dev.execution import SSHHost
from nisse_dev.gitea import Comment, Issue from nisse_dev.gitea import Comment, Issue
from nisse_dev.loop.agent import LoopAgent from nisse_dev.llm import RoleMap
from nisse_dev.loop.agent import LoopAgent, Project
from nisse_dev.loop.driver import Driver from nisse_dev.loop.driver import Driver
from nisse_dev.loop.help import Helpers
from nisse_dev.loop.items import ( from nisse_dev.loop.items import (
ANSWER, ANSWER,
BLOCKED, BLOCKED,
CHAT,
GO_AHEAD, GO_AHEAD,
MERGE, MERGE,
PROVISION,
RUNNING, RUNNING,
WAITING, WAITING,
loop_of, loop_of,
@@ -38,6 +47,7 @@ from nisse_dev.transport import InboundMessage, InMemoryTransport
from nisse_dev.worker import InlineWorker from nisse_dev.worker import InlineWorker
SEEDS = range(32) SEEDS = range(32)
TWO_PROJECT_SEEDS = range(100, 116)
STEPS = 30 STEPS = 30
@@ -46,8 +56,9 @@ class PolicyModel:
after comments, close after a merge; sometimes asks a question or after comments, close after a merge; sometimes asks a question or
replies in prose. Seeded, so a walk replays exactly.""" replies in prose. Seeded, so a walk replays exactly."""
def __init__(self, rng: random.Random) -> None: def __init__(self, rng: random.Random, projects: list[str]) -> None:
self.rng = rng self.rng = rng
self.projects = projects
self.n = 0 self.n = 0
def complete(self, events, tools, model, system=None) -> list[Any]: def complete(self, events, tools, model, system=None) -> list[Any]:
@@ -55,22 +66,51 @@ class PolicyModel:
self.n += 1 self.n += 1
offered = {t["function"]["name"] for t in tools} offered = {t["function"]["name"] for t in tools}
facts = [getattr(e.event, "text", "") for e in events] facts = [getattr(e.event, "text", "") for e in events]
# A `design` item writes documents until its design has merged.
design = "labelled design" in (system or "") and not any(
"the design has merged" in t for t in facts
)
path = f"docs/d{self.n}.md" if design else f"f{self.n}.py"
roll = self.rng.random() roll = self.rng.random()
if roll < 0.08: if roll < 0.08:
return [call("ask_human", question=f"question {self.n}?")] return [call("ask_human", question=f"question {self.n}?")]
if roll < 0.12: if roll < 0.12:
return [ToolCall(call_id=f"p{self.n}", tool="nonsense", arguments={})] return [ToolCall(call_id=f"p{self.n}", tool="nonsense", arguments={})]
if "reply" in offered and "propose_work" not in offered:
return [call("reply", text=f"because {self.n}")] # a question run
if "reply" in offered: # a conversation: talk, or propose work
if self.rng.random() < 0.1:
return [
call(
"propose_project",
name=f"proj{self.n}",
purpose=f"a project {self.n}.",
decisions="Python.",
)
]
if self.rng.random() < 0.4:
return [
call(
"propose_work",
project=self.rng.choice(self.projects),
title=f"filed {self.n}",
summary=f"do thing {self.n}.",
)
]
return [call("reply", text=f"an answer {self.n}")]
if "restate" not in offered and "finish_milestone" not in offered: if "restate" not in offered and "finish_milestone" not in offered:
return [call("ask_human", question=f"talk first {self.n}?")] # discuss return [call("ask_human", question=f"talk first {self.n}?")] # discuss
if "finish_milestone" not in offered: if "finish_milestone" not in offered:
return [call("restate", text=f"restatement {self.n}")] return [call("restate", text=f"restatement {self.n}", changed=f"c{self.n}")]
if "delegate" in offered and self.rng.random() < 0.1:
return [call("delegate", task=f"help {self.n}")]
if "wait_for_review" in offered and self.rng.random() < 0.2: if "wait_for_review" in offered and self.rng.random() < 0.2:
return [call("wait_for_review", note=f"waiting {self.n}")] return [call("wait_for_review", note=f"waiting {self.n}")]
if self.rng.random() < 0.15: if self.rng.random() < 0.15:
# Half done, then a question: work a switch must park. # Half done, then a question: work a switch must park.
return [ return [
call("start_milestone", name=f"m{self.n}"), call("start_milestone", name=f"m{self.n}"),
call("write_file", path=f"f{self.n}.py", content=f"x = {self.n}\n"), call("write_file", path=path, content=f"x = {self.n}\n"),
call("ask_human", question=f"halfway {self.n}?"), call("ask_human", question=f"halfway {self.n}?"),
] ]
merged = any("merged" in t for t in facts[-6:]) merged = any("merged" in t for t in facts[-6:])
@@ -78,7 +118,7 @@ class PolicyModel:
return [call("close_item", summary=f"done {self.n}")] return [call("close_item", summary=f"done {self.n}")]
return [ return [
call("start_milestone", name=f"m{self.n}"), call("start_milestone", name=f"m{self.n}"),
call("write_file", path=f"f{self.n}.py", content=f"x = {self.n}\n"), call("write_file", path=path, content=f"x = {self.n}\n"),
call( call(
"finish_milestone", "finish_milestone",
summary=f"work {self.n}", summary=f"work {self.n}",
@@ -87,14 +127,43 @@ class PolicyModel:
] ]
class HelperModel:
"""A scripted helper: does some work, then hands back, succeeding or
not as the seed says."""
def __init__(self, rng: random.Random) -> None:
self.rng = rng
self.n = 0
def complete(self, events, tools, model, system=None) -> list[Any]:
assert_renderable(events)
self.n += 1
return [
call("write_file", path=f"h{self.n}.py", content=f"h = {self.n}\n"),
ToolCall(
call_id=f"hb{self.n}",
tool="hand_back",
arguments={
"changed": f"h{self.n}.py",
"now_true": "helped",
"next": "go on",
"succeeded": self.rng.random() < 0.6,
},
),
]
class Walk: class Walk:
def __init__(self, world: dict[str, Any], seed: int) -> None: """``worlds``: project name -> its world (forge, bare repo, clone,
self.world = world milestones), all over one store."""
def __init__(self, worlds: dict[str, dict[str, Any]], seed: int) -> None:
self.worlds = worlds
self.world = next(iter(worlds.values())) # the store's home
self.rng = random.Random(seed) self.rng = random.Random(seed)
self.seed = seed self.seed = seed
self.steps: list[str] = [] self.steps: list[str] = []
self.shown_at_go: list[str] = [] # item:revision on screen at each go self.shown_at_go: list[str] = [] # item:revision on screen at each go
self.next_issue = 15
self.now = 1000.0 self.now = 1000.0
self.agent, self.transport = self._agent() self.agent, self.transport = self._agent()
@@ -105,58 +174,129 @@ class Walk:
def _agent(self) -> tuple[LoopAgent, InMemoryTransport]: def _agent(self) -> tuple[LoopAgent, InMemoryTransport]:
transport = InMemoryTransport() transport = InMemoryTransport()
# Event IDs are unique across restarts, as a server's are: a new
# transport goes on numbering where the old one stopped.
old = getattr(self, "transport", None)
transport._next_id = old._next_id if old is not None else 0
cancel = Cancel() cancel = Cancel()
w = self.world router = Router({w["repo_name"]: w["forge"] for w in self.worlds.values()})
for w in self.worlds.values():
w["milestones"]._gitea = router
agent = LoopAgent( agent = LoopAgent(
w["store"], self.world["store"],
transport, transport,
w["forge"], router, # type: ignore[arg-type]
repo="don/sandbox", projects=[
project="sandbox", Project(
name=name,
repo=w["repo_name"],
workspace=Workspace(w["host"], w["repo"].root, timeout_s=30),
milestones=w["milestones"],
)
for name, w in self.worlds.items()
],
account="nisse-dev", account="nisse-dev",
operators={DON}, operators={DON},
gitea_operators={"don"}, gitea_operators={"don"},
driver=Driver(PolicyModel(self.rng), "m", cancel=cancel), # type: ignore[arg-type] driver=Driver(
workspace=Workspace(w["host"], w["repo"].root, timeout_s=30), PolicyModel(self.rng, list(self.worlds)), # type: ignore[arg-type]
"m",
cancel=cancel,
),
worker=InlineWorker(), worker=InlineWorker(),
cancel=cancel, cancel=cancel,
oplog=OpLog("nisse-dev", _Null()), oplog=OpLog("nisse-dev", _Null()),
milestones=w["milestones"],
reader=self, # type: ignore[arg-type] reader=self, # type: ignore[arg-type]
helpers=Helpers(
client=HelperModel(self.rng), # type: ignore[arg-type]
roles=RoleMap(executor="", planner="m", helper="h", expert="x"),
caps=Caps(turns=4),
),
bug_grace_s=600, bug_grace_s=600,
clock=lambda: self.now, clock=lambda: self.now,
) )
transport.on_message(agent.on_message) transport.on_message(agent.on_message)
return agent, transport return agent, transport
def agrees(self, proposal: str, reply: str) -> bool: def agrees(self, proposal: str, reply: str, model: str = "") -> bool:
return self.reads_yes(proposal, reply) return self.reads_yes(proposal, reply)
def items(self) -> list[Any]: def items(self) -> list[Any]:
return self.world["store"].work_items() return self.world["store"].work_items()
def name(self, item: Any) -> str:
"""How the operator names an item: by project when there are two."""
issue = loop_of(item).issue
return f"#{issue}" if len(self.worlds) == 1 else f"{item.scope}#{issue}"
def step(self) -> None: def step(self) -> None:
forge: Forge = self.world["forge"]
open_items = [i for i in self.items() if i.status == "open"] open_items = [i for i in self.items() if i.status == "open"]
# Weighted towards what an item can actually do, so review paths # Weighted towards what an item can actually do, so review paths
# are walked often, not drowned out by intake and crashes. # are walked often, not drowned out by intake and crashes.
weights = {"intake": 2 if not open_items else 1, "crash": 1, "issue": 1} weights = {
"intake": 2 if not open_items else 1,
"crash": 1,
"issue": 1,
"chat": 1,
}
item = self.rng.choice(open_items) if open_items else None item = self.rng.choice(open_items) if open_items else None
loop = loop_of(item) if item else None loop = loop_of(item) if item else None
# The forge an item's project lives on; a new issue picks one.
project = (
item.scope
if item and item.scope != "general"
else self.rng.choice(list(self.worlds))
)
forge: Forge = self.worlds[project]["forge"]
if item and loop: if item and loop:
weights.update({"go": 3, "answer": 2, "abandon": 1, "go-stale": 1}) weights.update({"go": 3, "answer": 2, "abandon": 1, "go-stale": 1})
weights.update({"first": 1, "plain-yes": 2, "named-go": 1, "wait": 2}) weights.update({"plain-yes": 2, "wait": 2, "question": 2})
if loop.new_project:
weights.update({"done": 2})
if item.scope != "general": # conversations aren't named by number
weights.update({"first": 1, "named-go": 1})
if loop.pr: if loop.pr:
weights.update( weights.update(
{"comment": 4, "merge": 2, "close-pr": 1, "comment-merge": 2} {"comment": 4, "merge": 2, "close-pr": 1, "comment-merge": 2}
) )
action = self.rng.choices(list(weights), list(weights.values()))[0] action = self.rng.choices(list(weights), list(weights.values()))[0]
self.steps.append(f"{action}#{loop.issue}" if loop else action) self.steps.append(f"{action}{self.name(item)}" if item else action)
if action == "intake": if action == "intake":
self.agent.intake() self.agent.intake()
elif action == "done" and loop:
# The operator says a new project is ready; no entry was added.
self.transport.receive(DON, "done", thread=loop.thread)
elif action == "question" and loop:
# L5: a question never changes the item it's asked about.
before = (item.position, loop.wait, loop.rev, loop.pr, loop.branch)
self.transport.receive(DON, f"why {len(self.steps)}?", thread=loop.thread)
after_item = self.world["store"].work_item(item.id)
after = loop_of(after_item)
if (
item.scope != "general"
and item.position == WAITING
and loop.wait in (GO_AHEAD, ANSWER, MERGE)
):
now = (
after_item.position,
after.wait,
after.rev,
after.pr,
after.branch,
)
assert now == before, (
f"questions_change_nothing: {before} -> {now}: "
f"seed {self.seed}, steps {self.steps}"
)
elif action == "chat":
# A top-level message: a conversation of its own (L1, L5).
self.transport.receive(DON, f"thinking aloud {len(self.steps)}")
elif action == "issue": elif action == "issue":
# A new issue, sometimes depending on one that is still open. # A new issue, sometimes depending on one that is still open.
n, self.next_issue = self.next_issue, self.next_issue + 1 project = self.rng.choice(list(self.worlds))
forge = self.worlds[project]["forge"]
# Numbered by the forge, as filed records are: never reused.
n = max([i.number for i in forge.issues] + [14]) + 1
labels = self.rng.choice([(), (), ("bug",), ("discuss",), ("design",)]) labels = self.rng.choice([(), (), ("bug",), ("discuss",), ("design",)])
forge.issues.append(Issue(n, f"task {n}", f"do task {n}", "don", labels)) forge.issues.append(Issue(n, f"task {n}", f"do task {n}", "don", labels))
live = [i.number for i in forge.assigned_issues("", "") if i.number != n] live = [i.number for i in forge.assigned_issues("", "") if i.number != n]
@@ -171,10 +311,10 @@ class Walk:
elif action == "named-go" and loop: elif action == "named-go" and loop:
if item.position == WAITING and loop.wait == GO_AHEAD: if item.position == WAITING and loop.wait == GO_AHEAD:
self.shown_at_go.append(f"{item.id}:{loop.rev}") self.shown_at_go.append(f"{item.id}:{loop.rev}")
self.transport.receive(DON, f"go #{loop.issue}") self.transport.receive(DON, f"go {self.name(item)}")
elif action == "wait": elif action == "wait":
# Time passes: bug countdowns may run out. # Time passes: bug countdowns may run out, conversations idle.
self.now += self.rng.choice([60, 700]) self.now += self.rng.choice([60, 700, 90000])
for other in open_items: for other in open_items:
o = loop_of(other) o = loop_of(other)
if ( if (
@@ -185,7 +325,7 @@ class Walk:
self.shown_at_go.append(f"{other.id}:{o.rev}") self.shown_at_go.append(f"{other.id}:{o.rev}")
self.agent.intake() self.agent.intake()
elif action == "first" and loop: elif action == "first" and loop:
self.agent.handle(InboundMessage(DON, f"do #{loop.issue} next"), None) self.agent.handle(InboundMessage(DON, f"do {self.name(item)} next"), None)
elif action == "crash": elif action == "crash":
# Volatile state is lost: a new agent over the same store. # Volatile state is lost: a new agent over the same store.
self.agent, self.transport = self._agent() self.agent, self.transport = self._agent()
@@ -231,15 +371,17 @@ class Walk:
def check(self) -> None: def check(self) -> None:
where = f"seed {self.seed}, steps {self.steps}" where = f"seed {self.seed}, steps {self.steps}"
store, forge = self.world["store"], self.world["forge"] store = self.world["store"]
items = self.items() items = self.items()
issues = [loop_of(i).issue for i in items if loop_of(i).issue] issues = [(i.scope, loop_of(i).issue) for i in items if loop_of(i).issue]
assert len(issues) == len(set(issues)), f"one_item_per_issue: {where}" assert len(issues) == len(set(issues)), f"one_item_per_issue: {where}"
for item in items: for item in items:
loop = loop_of(item) loop = loop_of(item)
if item.status == "open" and item.position == WAITING: if item.status == "open" and item.position == WAITING:
explained = ( explained = (
(loop.wait == GO_AHEAD and loop.rev > 0) (loop.wait == CHAT and item.scope == "general")
or (loop.wait == PROVISION and bool(loop.new_project))
or (loop.wait == GO_AHEAD and loop.rev > 0)
or (loop.wait == ANSWER and bool(loop.note)) or (loop.wait == ANSWER and bool(loop.note))
or (loop.wait == MERGE and loop.pr > 0) or (loop.wait == MERGE and loop.pr > 0)
) )
@@ -255,9 +397,13 @@ class Walk:
assert all(f"{item.id}:{rev}" in self.shown_at_go for rev in gos), ( assert all(f"{item.id}:{rev}" in self.shown_at_go for rev in gos), (
f"go_binds: {item.id} {gos} not in {self.shown_at_go}: {where}" f"go_binds: {item.id} {gos} not in {self.shown_at_go}: {where}"
) )
if item.scope == "general":
continue # no forge: nothing below applies
forge = self.worlds[item.scope]["forge"]
repo = self.worlds[item.scope]["repo_name"]
# Blocked comes from external truth (L1): an item whose issue # Blocked comes from external truth (L1): an item whose issue
# had an open dependency from the start never ran before it closed. # had an open dependency from the start never ran before it closed.
if forge.open_dependencies("don/sandbox", loop.issue): if forge.open_dependencies(repo, loop.issue):
assert item.position == BLOCKED or item.status == "closed", ( assert item.position == BLOCKED or item.status == "closed", (
f"blocked_never_runs: #{loop.issue} {item.position}: {where}" f"blocked_never_runs: #{loop.issue} {item.position}: {where}"
) )
@@ -311,11 +457,70 @@ class Walk:
assert not [i for i in items if i.position == RUNNING], ( assert not [i for i in items if i.position == RUNNING], (
f"one_active (nothing left running): {where}" f"one_active (nothing left running): {where}"
) )
self._parked_work_safe(where) for item in items:
self._milestones_start_fresh(where) if item.scope == "general":
loop = loop_of(item)
# L1: a conversation has no workspace, milestone or PR.
assert not (loop.prepared or loop.milestone or loop.pr), (
f"conversations_do_no_work: {item.id}: {where}"
)
# L9: a delegation never outlives its step: merged back or discarded.
assert not [
i for i in store.pending_intents() if i.kind == DELEGATION_INTENT
], f"delegations_settle (intent): {where}"
for name, w in self.worlds.items():
helping = [
b
for b in [
*w["repo"].local_branches(),
*_git(
w["bare"], "for-each-ref", "--format=%(refname:short)"
).split(),
]
if "-help-" in b
]
assert not helping, f"delegations_settle (branch {helping}): {where}"
self._records_have_one_item(name, w, where)
self._own_repository_only(name, w, where)
self._parked_work_safe(w, name, where)
self._milestones_start_fresh(w, where)
self._review_fixes_land(w, where)
def _records_have_one_item(self, name: str, world: Any, where: str) -> None:
"""D1: a record issue filed from a conversation is its item's, from
the moment it exists, and intake never takes it for another."""
store = self.world["store"]
for issue in world["forge"].issues:
found = re.search(r"<!-- nisse-item: ([\w-]+) -->", issue.body or "")
if not found:
continue
item = store.work_item(found.group(1))
assert item is not None and item.scope == name, (
f"records_have_one_item: {name}#{issue.number} has no item: {where}"
)
assert loop_of(item).issue == issue.number, (
f"records_have_one_item: {item.id} isn't #{issue.number}: {where}"
)
assert loop_of(item).started, f"filed work starts agreed: {where}"
def _own_repository_only(self, name: str, world: Any, where: str) -> None:
"""D8: an item's branches and PRs reach only its own project's
repository."""
others = [i.id for i in self.items() if i.scope != name]
heads = [p["head"] for p in world["forge"].prs.values()]
branches = _git(
world["bare"], "for-each-ref", "--format=%(refname:short)", "refs/heads"
).split()
for ref in [*heads, *branches]:
assert not any(ref.startswith(f"{i}-") for i in others), (
f"own_repository_only: {ref} in {name}: {where}"
)
def _review_fixes_land(self, world: Any, where: str) -> None:
forge: Forge = world["forge"]
heads = [p["head"] for p in forge.prs.values()] heads = [p["head"] for p in forge.prs.values()]
assert len(heads) == len(set(heads)), f"one_pr_per_milestone: {where}" assert len(heads) == len(set(heads)), f"one_pr_per_milestone: {where}"
bare = self.world["bare"] bare = world["bare"]
for n, pr in forge.prs.items(): for n, pr in forge.prs.items():
if not pr["merged"]: if not pr["merged"]:
continue continue
@@ -330,11 +535,11 @@ class Walk:
) )
assert landed, f"review_fixes_land (PR {n}): {where}" assert landed, f"review_fixes_land (PR {n}): {where}"
def _milestones_start_fresh(self, where: str) -> None: def _milestones_start_fresh(self, world: Any, where: str) -> None:
"""D4: a milestone starts from the latest default branch, so one """D4: a milestone starts from the latest default branch, so one
redone after a closed PR never carries the rejected work.""" redone after a closed PR never carries the rejected work."""
forge: Forge = self.world["forge"] forge: Forge = world["forge"]
bare = self.world["bare"] bare = world["bare"]
for item in self.items(): for item in self.items():
mine = sorted( mine = sorted(
(n, p) (n, p)
@@ -353,13 +558,13 @@ class Walk:
bare, rejected, f"refs/heads/{later['head']}" bare, rejected, f"refs/heads/{later['head']}"
), f"milestones_start_fresh: PR {m} carries PR {n}: {where}" ), f"milestones_start_fresh: PR {m} carries PR {n}: {where}"
def _parked_work_safe(self, where: str) -> None: def _parked_work_safe(self, world: Any, name: str, where: str) -> None:
"""L1, D4: nothing of a parked item lives only on the build VM: what """L1, D4: nothing of a parked item lives only on the build VM: what
every branch of an open item that isn't on the workspace points at is every branch of an open item that isn't on the workspace points at is
in the forge.""" in its project's forge."""
root, bare = self.world["repo"].root, self.world["bare"] root, bare = world["repo"].root, world["bare"]
current = _git(root, "rev-parse", "--abbrev-ref", "HEAD") current = _git(root, "rev-parse", "--abbrev-ref", "HEAD")
ids = [i.id for i in self.items() if i.status == "open"] ids = [i.id for i in self.items() if i.status == "open" and i.scope == name]
for line in _git( for line in _git(
root, "for-each-ref", "--format=%(refname:short)", "refs/heads" root, "for-each-ref", "--format=%(refname:short)", "refs/heads"
).splitlines(): ).splitlines():
@@ -384,12 +589,15 @@ class Walk:
# And no write is lost: every file an open item's driver wrote is on # And no write is lost: every file an open item's driver wrote is on
# the workspace while it's that item's, or in the forge. # the workspace while it's that item's, or in the forge.
for item in self.items(): for item in self.items():
if item.status != "open": if item.status != "open" or item.scope != name:
continue continue
written = { written = {
str(e.event.arguments.get("path")) str(e.event.arguments.get("path"))
for e in self.world["store"].transcript(item.id, "driver").read() for e in self.world["store"].transcript(item.id, "driver").read()
if isinstance(e.event, ToolCall) and e.event.tool == "write_file" # The driver's own: a failed helper's are discarded on purpose.
if isinstance(e.event, ToolCall)
and e.event.tool == "write_file"
and e.actor == "driver"
} }
here = current.startswith(f"{item.id}-") here = current.startswith(f"{item.id}-")
for path in written: for path in written:
@@ -442,9 +650,35 @@ def _is_ancestor(bare: Path, commit: str, ref: str) -> bool:
) )
def _seeded_ids(monkeypatch: pytest.MonkeyPatch, seed: int) -> None:
"""Item IDs and branch names come from uuid4: seeded too, so a walk
replays exactly from its seed."""
ids = random.Random(seed)
monkeypatch.setattr(uuid, "uuid4", lambda: uuid.UUID(int=ids.getrandbits(128)))
@pytest.mark.parametrize("seed", SEEDS) @pytest.mark.parametrize("seed", SEEDS)
def test_the_loop_keeps_its_properties(world: dict[str, Any], seed: int) -> None: def test_the_loop_keeps_its_properties(
walk = Walk(world, seed) world: dict[str, Any], seed: int, monkeypatch: pytest.MonkeyPatch
) -> None:
_seeded_ids(monkeypatch, seed)
walk = Walk({"sandbox": world}, seed)
for _ in range(STEPS):
walk.step()
walk.check()
@pytest.mark.parametrize("seed", TWO_PROJECT_SEEDS)
def test_two_projects_keep_the_properties(
ssh_host: SSHHost, tmp_path: Path, seed: int, monkeypatch: pytest.MonkeyPatch
) -> None:
"""D8: the same walk over two projects, whose issue numbers collide."""
_seeded_ids(monkeypatch, seed)
sandbox = make_world(ssh_host, tmp_path / "sandbox")
other = make_world(
ssh_host, tmp_path / "other", store=sandbox["store"], repo_name="don/other"
)
walk = Walk({"sandbox": sandbox, "other": other}, seed)
for _ in range(STEPS): for _ in range(STEPS):
walk.step() walk.step()
walk.check() walk.check()
+163
View File
@@ -0,0 +1,163 @@
"""Conversation on the new loop (M2.7, L5, #42, #77): questions answered
without changing the item, revisions that say what changed, progress in
prose at a readable rate, raw tool activity on request, and a note for a
message that can't be read yet."""
from pathlib import Path
from typing import Any
from test_loop import DON, RESTATED, FakeGitea, Harness, ScriptedModel, call, issue
from nisse_dev.canonical import AssistantText
from nisse_dev.loop.items import ANSWER, GO_AHEAD, WAITING, loop_of
class Clock:
def __init__(self) -> None:
self.now = 1000.0
def __call__(self) -> float:
return self.now
def harness(tmp_path: Path, *turns: list[Any], **kw: Any) -> Harness:
return Harness(tmp_path, ScriptedModel(*turns), FakeGitea(issue()), **kw)
# --- questions never change state (#77) -------------------------------------
def test_a_question_about_a_restatement_leaves_it_up(tmp_path: Path) -> None:
h = harness(
tmp_path,
RESTATED,
[call("reply", text="Because text.py already holds the string helpers.")],
[call("ask_human", question="started; anything else?")],
)
h.agent.intake()
h.say("Why text.py?")
loop = loop_of(h.item())
assert (h.item().position, loop.wait, loop.rev) == (WAITING, GO_AHEAD, 1)
assert not loop.question and not loop.started
assert h.posted()[-1] == "Because text.py already holds the string helpers."
h.say("go") # still bound to rev-1
assert loop_of(h.item()).started
def test_a_question_back_keeps_the_agents_own_question_open(tmp_path: Path) -> None:
h = harness(
tmp_path,
[call("ask_human", question="One function or two?")],
[call("reply", text="One handles words, two would split lines too.")],
)
h.agent.intake()
h.say("What would the second one do?")
loop = loop_of(h.item())
assert (loop.wait, loop.note) == (ANSWER, "One function or two?")
def test_a_question_that_isnt_answered_still_goes_back(tmp_path: Path) -> None:
h = harness(
tmp_path,
RESTATED,
[call("restate", text="sneaky v2", changed="x")], # not offered
[AssistantText(text="hmm", model="m")],
[AssistantText(text="hmm", model="m")], # prose twice: the run fails
)
h.agent.intake()
h.say("why?")
loop = loop_of(h.item())
# restate isn't offered in a question run: no new revision.
assert (loop.wait, loop.rev) == (GO_AHEAD, 1)
assert h.posted()[-1].startswith("I couldn't answer that")
# --- explanations ------------------------------------------------------------
def test_a_revision_says_what_changed(tmp_path: Path) -> None:
h = harness(
tmp_path,
RESTATED,
[call("restate", text="count_words in words.py", changed="")],
[
call(
"restate",
text="count_words in words.py",
changed="a module of its own, as you asked",
)
],
)
h.agent.intake()
h.say("put it in its own module")
loop = loop_of(h.item())
assert loop.rev == 2
assert "Changed from rev-1: a module of its own, as you asked" in h.posted()[-1]
# --- progress and details (#42) ----------------------------------------------
def test_progress_is_posted_at_a_readable_rate(tmp_path: Path) -> None:
clock = Clock()
class Ticking(ScriptedModel):
def complete(self, events, tools, model, system=None): # type: ignore[no-untyped-def]
clock.now += 30 # each turn takes a while
return super().complete(events, tools, model, system)
h = Harness(
tmp_path,
Ticking(
[
AssistantText(text="Looking at text.py first.", model="m"),
call("list_dir", path="."),
],
[
AssistantText(text="Reading the tests too.", model="m"),
call("list_dir", path="tests"),
],
[
AssistantText(text="x" * 1000, model="m"),
call("list_dir", path="."),
],
[call("restate", text="I'll add count_words.", changed="")],
),
FakeGitea(issue()),
clock=clock,
)
h.agent.intake()
said = h.posted()
assert "Looking at text.py first." in said
assert "Reading the tests too." not in said # 30 s later: folded away
long = [p for p in said if p.startswith("xxx")]
assert long and len(long[0]) < 420 and long[0].endswith(" …") # 60 s on
def test_details_shows_the_last_tool_calls(tmp_path: Path) -> None:
h = harness(
tmp_path,
[
call("list_dir", path="src"),
call("read_file", path="text.py"),
call("restate", text="I'll add count_words.", changed=""),
],
)
h.agent.intake()
h.say("details")
shown = h.posted()[-1]
assert shown.startswith("The last tool calls on #14:")
assert "list_dir(src)" in shown and "read_file(text.py)" in shown
assert loop_of(h.item()).wait == GO_AHEAD # nothing changed
def test_a_message_mid_run_is_acknowledged(tmp_path: Path) -> None:
h = harness(tmp_path, RESTATED)
h.agent.intake()
h.agent._running = True # as if a driver run were under way
h.agent._active_thread = "elsewhere"
h.agent._active_name = "#15"
h.transport.receive(DON, "one more thing", thread=h.thread())
assert "Noted; I'm in the middle of #15" in h.posted()[-1]
h.transport.receive(DON, "and another", thread="elsewhere")
assert h.posted()[-1] == "Noted; I'll read it when this turn ends."
+3 -1
View File
@@ -14,7 +14,8 @@ testpaths = [
"nisse-ops/tests", "nisse-ops/tests",
"tests", "tests",
] ]
addopts = "--strict-markers" # The seed is a test project's own code, pushed to each run's repository.
addopts = "--strict-markers --ignore=tests/integration/seed"
markers = [ markers = [
"integration: end-to-end tests that exercise installed entry points or external services", "integration: end-to-end tests that exercise installed entry points or external services",
"e2e: the whole agent against real services, run after each milestone (scripts/e2e.sh)", "e2e: the whole agent against real services, run after each milestone (scripts/e2e.sh)",
@@ -22,6 +23,7 @@ markers = [
[tool.ruff] [tool.ruff]
target-version = "py312" target-version = "py312"
extend-exclude = ["tests/integration/seed"] # a test project's own code
[tool.ruff.lint] [tool.ruff.lint]
extend-select = ["I"] extend-select = ["I"]
+54 -22
View File
@@ -1,13 +1,19 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Bring-up runs of nisse-dev on this development VM (Development Plan M0.2). # Bring-up runs of nisse-dev on this development VM (Development Plan M0.2).
# #
# The agent runs as root, as it will on its own VM in production (D7): it may # The agent runs as its container image (M1.5), built from the code under
# install what it needs. It touches only test resources (the sandbox repo, the # test, as a transient systemd unit, so it outlives the shell that started
# test Matrix room, a budget-capped LiteLLM key), from the run's config. It # it. It touches only test resources (an operator-made test repository, the
# runs as a transient systemd unit, so it outlives the shell that started it. # test Matrix room, a budget-capped LiteLLM key), from the run's config.
# #
# scripts/bringup.sh start [REF] set the last run aside, export REF (default # $E2E is mounted at the same path inside the container, so the config's
# HEAD) and start the agent # paths (state_db, keys) mean the same there; for that it runs as root here,
# since the keys are root's. In production the image runs as its own user
# with a state volume (Dockerfile).
#
# scripts/bringup.sh start [REF] set the last run aside, build REF (default
# HEAD) into an image and start the agent
# scripts/bringup.sh plan [REF] print what start would build and run
# scripts/bringup.sh stop stop it cleanly (SIGTERM) # scripts/bringup.sh stop stop it cleanly (SIGTERM)
# scripts/bringup.sh status the unit, and the agent's state # scripts/bringup.sh status the unit, and the agent's state
# scripts/bringup.sh logs follow the agent's log # scripts/bringup.sh logs follow the agent's log
@@ -16,7 +22,7 @@
# #
# Layout, under $E2E (default /root/nisse-e2e): # Layout, under $E2E (default /root/nisse-e2e):
# nisse-dev.yaml, .env the run config and its secrets (made by hand once) # nisse-dev.yaml, .env the run config and its secrets (made by hand once)
# code/ the exported code under test, with its own .venv # code/ the exported code under test, built into the image
# state.db the agent's state (its workspace is on the build VM) # state.db the agent's state (its workspace is on the build VM)
# ssh/ the agent's key for the build VM, and push keys # ssh/ the agent's key for the build VM, and push keys
# run-info, collected/ what was run, and the output of `collect` # run-info, collected/ what was run, and the output of `collect`
@@ -27,11 +33,28 @@ E2E=${NISSE_E2E:-/root/nisse-e2e}
SRC=${NISSE_SRC:-/root/nisse} # repository to export code from SRC=${NISSE_SRC:-/root/nisse} # repository to export code from
CODE=$E2E/code CODE=$E2E/code
UNIT=nisse-bringup UNIT=nisse-bringup
UV=${UV:-$(command -v uv || echo /snap/bin/uv)} IMAGE=nisse-dev:bringup
die() { echo "bringup: $*" >&2; exit 1; } die() { echo "bringup: $*" >&2; exit 1; }
running() { systemctl is-active --quiet "$UNIT"; } running() { systemctl is-active --quiet "$UNIT"; }
# The container the unit runs: host networking (Matrix, Gitea, LiteLLM as
# here), $E2E at its own path, the config and secrets from it.
docker_run() {
echo docker run --rm --name "$UNIT" --network host --user 0 \
-v "$E2E:$E2E" \
-e NISSE_CONFIG="$E2E/nisse-dev.yaml" -e NISSE_ENV="$E2E/.env" \
"$IMAGE"
}
cmd_plan() {
local ref=${1:-HEAD} sha
sha=$(git -C "$SRC" rev-parse --verify "$ref^{commit}") || die "unknown ref $ref"
echo "git -C $SRC archive $sha | tar -x -C $CODE"
echo "docker build -t $IMAGE $CODE"
echo "systemd-run --unit=$UNIT ... $(docker_run)"
}
cmd_start() { cmd_start() {
local ref=${1:-HEAD} sha local ref=${1:-HEAD} sha
[ -f "$E2E/nisse-dev.yaml" ] && [ -f "$E2E/.env" ] || die "no config in $E2E" [ -f "$E2E/nisse-dev.yaml" ] && [ -f "$E2E/.env" ] || die "no config in $E2E"
@@ -58,16 +81,16 @@ cmd_start() {
rm -rf "$CODE" rm -rf "$CODE"
mkdir -p "$CODE" mkdir -p "$CODE"
git -C "$SRC" archive "$sha" | tar -x -C "$CODE" git -C "$SRC" archive "$sha" | tar -x -C "$CODE"
(cd "$CODE" && "$UV" sync --quiet --all-packages) docker build --quiet -t "$IMAGE" "$CODE" > /dev/null
printf 'ref: %s\ncommit: %s\nstarted: %s\n' "$ref" "$sha" "$(date -Is)" > "$E2E/run-info" printf 'ref: %s\ncommit: %s\nstarted: %s\n' "$ref" "$sha" "$(date -Is)" > "$E2E/run-info"
systemctl reset-failed "$UNIT" 2>/dev/null || true systemctl reset-failed "$UNIT" 2>/dev/null || true
# SIGTERM reaches the agent through docker run's signal proxy; it shuts
# down cleanly (main.py).
# shellcheck disable=SC2046
systemd-run --quiet --unit="$UNIT" --collect \ systemd-run --quiet --unit="$UNIT" --collect \
--property=WorkingDirectory="$CODE" \
--property=KillSignal=SIGTERM --property=TimeoutStopSec=120 \ --property=KillSignal=SIGTERM --property=TimeoutStopSec=120 \
--setenv=NISSE_CONFIG="$E2E/nisse-dev.yaml" \ $(docker_run)
--setenv=NISSE_ENV="$E2E/.env" \
"$CODE/.venv/bin/nisse-dev"
echo "started $UNIT at ${sha:0:8} ($ref)" echo "started $UNIT at ${sha:0:8} ($ref)"
} }
@@ -78,15 +101,17 @@ remote_aside() {
where=$(python3 - "$E2E/nisse-dev.yaml" <<'PY' where=$(python3 - "$E2E/nisse-dev.yaml" <<'PY'
import sys, yaml import sys, yaml
c = yaml.safe_load(open(sys.argv[1])) c = yaml.safe_load(open(sys.argv[1]))
b = c["build_host"] for p in c["projects"].values(): # each project's workspace, on its build VM (D8)
print(b.get("user", "root"), b["address"], b.get("port", 22), b["key"], c["workspace"]) b = p["build_host"]
print(b.get("user", "root"), b["address"], b.get("port", 22), b["key"], p["workspace"])
PY PY
) || die "cannot read build_host and workspace from the config" ) || die "cannot read the projects' build VMs and workspaces from the config"
read -r user address port key workspace <<< "$where" while read -r user address port key workspace; do
ssh -i "$key" -p "$port" -o BatchMode=yes -o StrictHostKeyChecking=accept-new \ ssh -n -i "$key" -p "$port" -o BatchMode=yes -o StrictHostKeyChecking=accept-new \
-o UserKnownHostsFile="$E2E/ssh/known_hosts" "$user@$address" \ -o UserKnownHostsFile="$E2E/ssh/known_hosts" "$user@$address" \
"if [ -e '$workspace' ]; then mv '$workspace' '$workspace.$stamp' && echo 'build VM: set aside $workspace.$stamp'; fi" \ "if [ -e '$workspace' ]; then mv '$workspace' '$workspace.$stamp' && echo 'build VM: set aside $workspace.$stamp'; fi" \
|| die "cannot reach the build VM $address" || die "cannot reach the build VM $address"
done <<< "$where"
} }
cmd_stop() { cmd_stop() {
@@ -98,28 +123,35 @@ cmd_stop() {
cmd_status() { cmd_status() {
if running; then echo "unit: running"; else echo "unit: not running"; fi if running; then echo "unit: running"; else echo "unit: not running"; fi
[ -f "$E2E/run-info" ] && sed 's/^/ /' "$E2E/run-info" [ -f "$E2E/run-info" ] && sed 's/^/ /' "$E2E/run-info"
if [ -f "$E2E/state.db" ] && [ -x "$CODE/.venv/bin/python" ]; then if [ -f "$E2E/state.db" ]; then
"$CODE/.venv/bin/python" -m nisse_dev.dump "$E2E/state.db" | head -n 3 dump "$E2E/state.db" | head -n 3
fi fi
} }
cmd_logs() { journalctl -u "$UNIT" -f -o cat; } cmd_logs() { journalctl -u "$UNIT" -f -o cat; }
# The state store, read with the image's own code.
dump() {
docker run --rm --user 0 -v "$E2E:$E2E" --entrypoint python "$IMAGE" \
-m nisse_dev.dump "$@"
}
cmd_collect() { cmd_collect() {
[ -f "$E2E/run-info" ] || die "no run to collect" [ -f "$E2E/run-info" ] || die "no run to collect"
local out=$E2E/collected since local out=$E2E/collected since
since=$(sed -n 's/^started: //p' "$E2E/run-info") since=$(sed -n 's/^started: //p' "$E2E/run-info")
mkdir -p "$out" mkdir -p "$out"
journalctl -u "$UNIT" --since "$since" -o cat > "$out/agent.log" journalctl -u "$UNIT" --since "$since" -o cat > "$out/agent.log"
"$CODE/.venv/bin/python" -m nisse_dev.dump --full "$E2E/state.db" > "$out/transcript.txt" dump --full "$E2E/state.db" > "$out/transcript.txt"
echo "collected into $out" echo "collected into $out"
} }
case "${1:-}" in case "${1:-}" in
start) shift; cmd_start "$@" ;; start) shift; cmd_start "$@" ;;
plan) shift; cmd_plan "$@" ;;
stop) cmd_stop ;; stop) cmd_stop ;;
status) cmd_status ;; status) cmd_status ;;
logs) cmd_logs ;; logs) cmd_logs ;;
collect) cmd_collect ;; collect) cmd_collect ;;
*) sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;; *) sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
esac esac
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# Delete test repositories a live test run left behind (it died before its
# teardown): the test operator's nisse-e2e-* repositories older than a day,
# or of any age with --all. Needs NISSE_OPERATOR_TOKEN (from .env).
#
# scripts/e2e-cleanup.sh older than a day
# scripts/e2e-cleanup.sh --all every one, e.g. when nothing is running
set -euo pipefail
cd "$(dirname "$0")/.."
if [ -f .env ]; then set -a; . ./.env; set +a; fi
exec uv run python - "$@" <<'PY'
import os, sys
from datetime import datetime, timedelta, timezone
from nisse_dev.gitea import GiteaClient
gitea = GiteaClient("https://gitea.dongar.net", os.environ["NISSE_OPERATOR_TOKEN"])
cutoff = datetime.now(timezone.utc) - (timedelta(0) if "--all" in sys.argv else timedelta(days=1))
page, gone = 1, 0
while True:
repos = gitea._request("GET", "/user/repos", params={"page": page, "limit": 50})
if not repos:
break
for repo in repos:
created = datetime.fromisoformat(repo["created_at"].replace("Z", "+00:00"))
if repo["name"].startswith("nisse-e2e-") and created < cutoff:
gitea._request("DELETE", f"/repos/{repo['full_name']}")
print("deleted", repo["full_name"])
gone += 1
page += 1
print(f"{gone} deleted")
PY
+44 -4
View File
@@ -1,16 +1,33 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# The end-to-end test (Development Plan M1.6): the whole agent against real # The end-to-end test (Development Plan M1.6): the whole agent against real
# services, driven by a scripted operator. Run after each milestone; it # services, driven by a scripted operator. CI runs it on every PR (with
# spends model tokens and takes minutes, so it is not part of check.sh or CI. # --local); locally, run only the tests you're writing (-k) or diagnosing
# (--lf).
# #
# scripts/e2e.sh on the bring-up build VM (configured below) # scripts/e2e.sh on the bring-up build VM (configured below)
# scripts/e2e.sh --local on a throwaway local sshd instead # scripts/e2e.sh --local on a throwaway local sshd instead
# scripts/e2e.sh --image REF inside an image's `test` stage (Dockerfile),
# on its own sshd: checks a release as built
# #
# Uses tests/integration/nisse-test.yaml and its secrets (.env at the repo # Uses tests/integration/nisse-test.yaml and its secrets (.env at the repo
# root), like the integration tests. # root), like the integration tests. Every role runs on a local model
# (tests/integration/e2e_support.py; NISSE_E2E_MODEL overrides it), so tell
# the operator before a full run: it keeps the RTX 4090 busy for a while.
set -euo pipefail set -euo pipefail
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."
if [ "${1:-}" = "--image" ]; then
image=${2:?--image needs an image reference}
shift 2
runs=${NISSE_E2E_RUNS:-/root/nisse-e2e/test-runs}
mkdir -p "$runs"
secrets=()
[ -f .env ] && secrets=(--env-file .env)
# Host networking: the suite reaches Matrix, Gitea and LiteLLM as here.
exec docker run --rm --network host "${secrets[@]}" \
-e NISSE_E2E_RUNS=/runs -v "$runs:/runs" \
"$image" scripts/e2e.sh --local "$@"
fi
if [ "${1:-}" = "--local" ]; then if [ "${1:-}" = "--local" ]; then
shift shift
else else
@@ -19,4 +36,27 @@ else
fi fi
# Each run's state store is kept here, for nisse_dev.dump after a failure. # Each run's state store is kept here, for nisse_dev.dump after a failure.
export NISSE_E2E_RUNS=${NISSE_E2E_RUNS:-/root/nisse-e2e/test-runs} export NISSE_E2E_RUNS=${NISSE_E2E_RUNS:-/root/nisse-e2e/test-runs}
exec uv run pytest -m e2e -rs tests/integration/test_end_to_end.py tests/integration/test_end_to_end_loop.py "$@"
# What the run spent on the test key, read from LiteLLM (spend only). The
# end-to-end tests run on a local model, so anything above $0 means a call
# went to a paid one: an override, or a proxy fallback.
spend() {
(set -a; [ -f .env ] && . ./.env; set +a
uv run python -c '
import os, httpx
r = httpx.get("https://litellm.dongar.net/key/info", timeout=30,
headers={"Authorization": "Bearer " + os.environ["NISSE_LLM_TOKEN"]})
i = r.json()["info"]
print(i["spend"], i.get("max_budget"))' 2>/dev/null) || echo "? ?"
}
read -r before budget < <(spend)
status=0
uv run pytest -m e2e -rs tests/integration/test_end_to_end.py tests/integration/test_end_to_end_loop.py "$@" || status=$?
read -r after _ < <(spend)
python3 -c "
b, a, cap = '$before', '$after', '$budget'
try:
print(f'e2e spend: \${float(a) - float(b):.4f} this run; \${float(a):.2f} of \${float(cap):.2f} today')
except ValueError:
print('e2e spend: unknown (could not read the key)')"
exit "$status"
+112 -11
View File
@@ -1,12 +1,27 @@
"""Shared by the end-to-end tests: each run gets work that can't already be """Shared by the live tests: each run gets a repository of its own, made by
done, and a base branch of its own, so runs never depend on one another or the test operator and deleted afterwards, so runs never depend on one
on what earlier runs merged.""" another or on what earlier runs merged."""
import base64
import contextlib import contextlib
import os import os
import subprocess
from collections.abc import Iterator from collections.abc import Iterator
from dataclasses import dataclass
from pathlib import Path
from nisse_dev.gitea import GiteaClient, GiteaError from nisse_dev.gitea import GiteaClient, GiteaError
from nisse_dev.llm import RoleMap
# The end-to-end tests run every role on a local model, which costs nothing
# per call; NISSE_E2E_MODEL overrides it (e.g. claude-haiku-4-5 for a quick,
# paid run).
E2E_MODEL = "qwen3.8-27b"
def e2e_roles() -> RoleMap:
model = os.environ.get("NISSE_E2E_MODEL", "").strip() or E2E_MODEL
return RoleMap(executor=model, planner=model, helper=model, expert=model)
def task(run: str) -> str: def task(run: str) -> str:
@@ -34,18 +49,104 @@ def operator_gitea(url: str) -> GiteaClient:
return GiteaClient(url, token) return GiteaClient(url, token)
OWNER = "nisse-testing-operator" # the test operator: owns every test repository
SEED = Path(__file__).parent / "seed"
@dataclass(frozen=True)
class FreshRepo:
name: str # owner/repo
push_key: Path # this repository's deploy key, private half
@contextlib.contextmanager @contextlib.contextmanager
def base_branch(operator: GiteaClient, repo: str, run: str) -> Iterator[str]: def fresh_repo(
"""A branch off main for this run's PRs to target and merge into, so operator: GiteaClient, run: str, keys: Path, *, agent: str, seed: bool = True
nothing accumulates in main; deleted afterwards.""" ) -> Iterator[FreshRepo]:
name = f"e2e-base-{run}" """A repository of its own for one test run, as the operator would make a
project's (D4, D8): seeded with a small Python package, the ``agent``
account a collaborator with write access, the default branch protected (pushes only by the
owner, an approval to merge), and a deploy key of its own, written under
``keys``. Deleted afterwards, unless NISSE_E2E_KEEP_REPOS is set."""
name = f"nisse-e2e-{run}"
full = f"{OWNER}/{name}"
operator._request( operator._request(
"POST", "POST",
f"/repos/{repo}/branches", "/user/repos",
json={"new_branch_name": name, "old_branch_name": "main"}, json={
"name": name,
"private": True,
"auto_init": True,
"default_branch": "main",
},
) )
try: try:
yield name files = [
{
"operation": "create",
"path": str(path.relative_to(SEED)),
"content": base64.b64encode(path.read_bytes()).decode(),
}
for path in sorted(SEED.rglob("*"))
if seed and path.is_file() and path.name != "README.md"
]
if files:
operator._request(
"POST",
f"/repos/{full}/contents",
json={
"branch": "main",
"message": "Seed the test project",
"files": files,
},
)
readme = operator._request("GET", f"/repos/{full}/contents/README.md")
# A new project's README says only what it is; a seeded one, the seed's.
text = (SEED / "README.md").read_bytes() if seed else f"# {name}\n".encode()
operator._request(
"PUT",
f"/repos/{full}/contents/README.md",
json={
"branch": "main",
"message": "README",
"sha": readme["sha"],
"content": base64.b64encode(text).decode(),
},
)
operator._request(
"PUT", f"/repos/{full}/collaborators/{agent}", json={"permission": "write"}
)
keys.mkdir(parents=True, exist_ok=True)
key = keys / f"{name}.key"
key.unlink(missing_ok=True)
subprocess.run(
["ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-C", name, "-f", str(key)],
check=True,
)
operator._request(
"POST",
f"/repos/{full}/keys",
json={
"title": f"{name} push",
"key": key.with_suffix(".key.pub").read_text().strip(),
"read_only": False,
},
)
operator._request(
"POST",
f"/repos/{full}/branch_protections",
# Pushes only by the operator (who sets up fixtures, as an owner
# can); the agent's are refused, and merging needs an approval.
json={
"branch_name": "main",
"enable_push": True,
"enable_push_whitelist": True,
"push_whitelist_usernames": [OWNER],
"required_approvals": 1,
},
)
yield FreshRepo(full, key)
finally: finally:
if not os.environ.get("NISSE_E2E_KEEP_REPOS"):
with contextlib.suppress(GiteaError): with contextlib.suppress(GiteaError):
operator._request("DELETE", f"/repos/{repo}/branches/{name}") operator._request("DELETE", f"/repos/{full}")
+17 -11
View File
@@ -7,19 +7,30 @@
# the live tests that need it, rather than skipping them, so a broken # the live tests that need it, rather than skipping them, so a broken
# credential cannot hide as a skip. # credential cannot hide as a skip.
# Unused by the live tests, which work in pytest's temporary directories on a # Unused by the live tests, which work in pytest's temporary directories;
# throwaway local sshd (the `ssh_host` fixture); present because every agent # present because every agent config has it.
# config has them.
state_db: /tmp/nisse-test/state.db state_db: /tmp/nisse-test/state.db
workspace: /tmp/nisse-test/work
project: nisse-dev-testing projects:
build_host: nisse-dev-testing:
# Placeholders: every live test that touches a repository makes one of
# its own (tests/integration/e2e_support.py, fresh_repo), owned by the
# test operator and deleted afterwards, with its own deploy key; and runs
# on a throwaway local sshd (the `ssh_host` fixture) or the bring-up build
# VM, in temporary directories.
repo: nisse-testing-operator/made-per-run
workspace: /tmp/nisse-test/work
build_host:
address: 127.0.0.1 address: 127.0.0.1
key: /tmp/nisse-test/unused-key key: /tmp/nisse-test/unused-key
llm: llm:
base_url: https://litellm.dongar.net base_url: https://litellm.dongar.net
token: ${NISSE_LLM_TOKEN} token: ${NISSE_LLM_TOKEN}
# The test key caps each request's worst-case cost ($0.10): input plus
# this much reserved output. The test agent's replies are short, so half
# the default leaves room for ~40k input tokens on Haiku.
max_output_tokens: 8192
# The live tests exercise the gated loop and the routes over a real proxy, # The live tests exercise the gated loop and the routes over a real proxy,
# not local-model competence (measured separately by test_model_capability), # not local-model competence (measured separately by test_model_capability),
@@ -40,11 +51,6 @@ matrix:
gitea: gitea:
url: https://gitea.dongar.net url: https://gitea.dongar.net
token: ${NISSE_GITEA_TOKEN} token: ${NISSE_GITEA_TOKEN}
repo: dongar/nisse-dev-testing-sandbox # the sandbox, never a real project
# nisse-testing-operator plays the operator in the end-to-end tests # nisse-testing-operator plays the operator in the end-to-end tests
# (NISSE_OPERATOR_TOKEN); an operator here and nowhere else. # (NISSE_OPERATOR_TOKEN); an operator here and nowhere else.
operators: [dongar, nisse-testing-operator] operators: [dongar, nisse-testing-operator]
# What pushes go by (D4): the sandbox's deploy key, whose private half CI
# writes to this path from the NISSE_PUSH_KEY secret (locally: copy it
# there). An HTTPS push token works too: push_user plus push_token.
push_key: /tmp/nisse-test/push_key
+4
View File
@@ -0,0 +1,4 @@
# sandbox
A small Python package for nisse-dev's end-to-end tests: each run gets a
fresh repository seeded with this. Run its tests with `python3 -m pytest`.
@@ -0,0 +1,2 @@
from .text import truncate
from .mathutils import factorial
@@ -0,0 +1,16 @@
"""Math utilities."""
def factorial(n: int) -> int:
"""Return n! for a non-negative integer n.
Raises TypeError if n is not an int (bool is rejected), ValueError if n < 0.
"""
if isinstance(n, bool) or not isinstance(n, int):
raise TypeError(f"n must be an int, got {type(n).__name__}")
if n < 0:
raise ValueError(f"n must be >= 0, got {n}")
result = 1
for i in range(2, n + 1):
result *= i
return result
+10
View File
@@ -0,0 +1,10 @@
import re
import unicodedata
_NON_ALNUM = re.compile(r"[^a-z0-9]+")
def slugify(text: str) -> str:
"""Lowercase; runs of non-alphanumerics -> single '-'; no leading/trailing '-'."""
ascii_text = unicodedata.normalize("NFKD", text).encode("ascii", "ignore").decode("ascii")
return _NON_ALNUM.sub("-", ascii_text.lower()).strip("-")
+16
View File
@@ -0,0 +1,16 @@
"""Text utilities."""
ELLIPSIS = "\u2026"
def truncate(text: str, n: int) -> str:
"""Return text unchanged if len(text) <= n, else its first n-1 chars followed by '…'.
Raises ValueError if n < 1.
"""
if n < 1:
raise ValueError(f"n must be >= 1, got {n}")
if len(text) <= n:
return text
return text[: n - 1] + ELLIPSIS
@@ -0,0 +1,42 @@
import math
import pytest
from sandbox import factorial
@pytest.mark.parametrize(
"n, expected",
[
(0, 1),
(1, 1),
(2, 2),
(3, 6),
(5, 120),
(10, 3628800),
(20, 2432902008176640000),
],
)
def test_factorial_known_values(n, expected):
assert factorial(n) == expected
def test_factorial_matches_math_factorial():
for n in range(0, 101):
assert factorial(n) == math.factorial(n)
def test_factorial_large_input_no_recursion_limit():
assert factorial(2000) == math.factorial(2000)
@pytest.mark.parametrize("n", [-1, -5, -100])
def test_factorial_negative_raises_value_error(n):
with pytest.raises(ValueError):
factorial(n)
@pytest.mark.parametrize("n", [5.0, 2.5, "5", None, True, False])
def test_factorial_non_int_raises_type_error(n):
with pytest.raises(TypeError):
factorial(n)
@@ -0,0 +1,25 @@
import pytest
from sandbox.slugify import slugify
@pytest.mark.parametrize(
"text, expected",
[
("Hello World", "hello-world"),
("Hello, World!", "hello-world"),
(" leading and trailing ", "leading-and-trailing"),
("multiple spaces", "multiple-spaces"),
("a--b__c..d", "a-b-c-d"),
("!!!", ""),
("", ""),
("-already-slugged-", "already-slugged"),
("MiXeD CaSe 123", "mixed-case-123"),
("tab\tand\nnewline", "tab-and-newline"),
("Café Déjà Vu", "cafe-deja-vu"),
("emoji 🎉 party", "emoji-party"),
("Foo Bar", "foo-bar"),
],
)
def test_slugify(text, expected):
assert slugify(text) == expected
@@ -0,0 +1,37 @@
import pytest
from sandbox import truncate
@pytest.mark.parametrize(
"text, n, expected",
[
("", 1, ""),
("abc", 3, "abc"),
("abc", 10, "abc"),
("abcd", 3, "ab\u2026"),
("hello world", 5, "hell\u2026"),
("ab", 1, "\u2026"),
("a", 1, "a"),
("h\u00e9llo w\u00f6rld", 4, "h\u00e9l\u2026"),
],
)
def test_truncate_cases(text, n, expected):
assert truncate(text, n) == expected
@pytest.mark.parametrize("n", range(1, 16))
def test_result_never_exceeds_n(n):
assert len(truncate("hello world!!", n)) <= n
def test_uses_single_ellipsis_char():
result = truncate("hello world", 5)
assert result.endswith("\u2026")
assert not result.endswith("...")
@pytest.mark.parametrize("n", [0, -5])
def test_invalid_n_raises(n):
with pytest.raises(ValueError):
truncate("abc", n)
+14 -7
View File
@@ -25,7 +25,7 @@ from pathlib import Path
from typing import Any from typing import Any
import pytest import pytest
from e2e_support import base_branch, operator_gitea from e2e_support import e2e_roles, fresh_repo, operator_gitea
from e2e_support import task as make_task from e2e_support import task as make_task
from nisse_dev.agent import Agent, build_agent from nisse_dev.agent import Agent, build_agent
@@ -38,7 +38,7 @@ from nisse_dev.control import (
PR_OPEN, PR_OPEN,
) )
from nisse_dev.execution import SSHHost from nisse_dev.execution import SSHHost
from nisse_dev.gitea import GiteaError from nisse_dev.gitea import GiteaClient, GiteaError
from nisse_dev.transport import InMemoryTransport from nisse_dev.transport import InMemoryTransport
pytestmark = [pytest.mark.integration, pytest.mark.e2e] pytestmark = [pytest.mark.integration, pytest.mark.e2e]
@@ -72,13 +72,20 @@ def agent(
run = uuid.uuid4().hex[:8] run = uuid.uuid4().hex[:8]
stack = contextlib.ExitStack() stack = contextlib.ExitStack()
operator = operator_gitea(live_config.gitea_url) operator = operator_gitea(live_config.gitea_url)
base = stack.enter_context(base_branch(operator, live_config.gitea_repo, run)) account = GiteaClient(live_config.gitea_url, live_config.gitea_token).whoami()
fresh = stack.enter_context(
fresh_repo(operator, run, tmp_path / "keys", agent=account)
)
config = replace( config = replace(
live_config, live_config.with_project(
base_branch=base, repo=fresh.name,
state_db=tmp_path / "state.db", base_branch="main",
push_key=fresh.push_key,
workspace=Path(f"/tmp/nisse-e2e-{run}/sandbox"), workspace=Path(f"/tmp/nisse-e2e-{run}/sandbox"),
build_host=build_host(ssh_host), build_host=build_host(ssh_host),
),
roles=e2e_roles(),
state_db=tmp_path / "state.db",
) )
transport = InMemoryTransport() transport = InMemoryTransport()
monkeypatch.setattr("nisse_dev.agent.MatrixTransport", lambda cfg: transport) monkeypatch.setattr("nisse_dev.agent.MatrixTransport", lambda cfg: transport)
@@ -99,7 +106,7 @@ def agent(
state_dir=tmp_path / "ssh", state_dir=tmp_path / "ssh",
) )
host.execute(f"rm -rf /tmp/nisse-e2e-{run}") host.execute(f"rm -rf /tmp/nisse-e2e-{run}")
stack.close() # the run's base branch stack.close() # the run's repository
class Operator: class Operator:
+197 -12
View File
@@ -12,6 +12,7 @@ scripts/e2e.sh after each milestone.
import base64 import base64
import contextlib import contextlib
import os import os
import re
import time import time
import uuid import uuid
from collections.abc import Iterator from collections.abc import Iterator
@@ -21,7 +22,7 @@ from pathlib import Path
from typing import Any from typing import Any
import pytest import pytest
from e2e_support import base_branch, dependent_task, operator_gitea, task from e2e_support import dependent_task, e2e_roles, fresh_repo, operator_gitea, task
from nisse_dev.config import AgentConfig, LoopTuning from nisse_dev.config import AgentConfig, LoopTuning
from nisse_dev.execution import SSHHost from nisse_dev.execution import SSHHost
@@ -63,14 +64,19 @@ def running(
keep.mkdir(parents=True, exist_ok=True) keep.mkdir(parents=True, exist_ok=True)
print(f"\ne2e run {run}: state store in {keep}") print(f"\ne2e run {run}: state store in {keep}")
operator = operator_gitea(live_config.gitea_url) operator = operator_gitea(live_config.gitea_url)
base = stack.enter_context(base_branch(operator, live_config.gitea_repo, run)) account = GiteaClient(live_config.gitea_url, live_config.gitea_token).whoami()
fresh = stack.enter_context(fresh_repo(operator, run, keep / "keys", agent=account))
config = replace( config = replace(
live_config, live_config.with_project(
loop="new", repo=fresh.name,
base_branch=base, base_branch="main",
state_db=keep / "state.db", push_key=fresh.push_key,
workspace=Path(f"/tmp/nisse-e2e-{run}/sandbox"), workspace=Path(f"/tmp/nisse-e2e-{run}/sandbox"),
build_host=build_host(ssh_host), build_host=build_host(ssh_host),
),
loop="new",
roles=e2e_roles(),
state_db=keep / "state.db",
# Short, so a bug's grace period runs out within the test. # Short, so a bug's grace period runs out within the test.
tuning=replace(live_config.tuning, loop=LoopTuning(bug_grace_s=BUG_GRACE_S)), tuning=replace(live_config.tuning, loop=LoopTuning(bug_grace_s=BUG_GRACE_S)),
) )
@@ -78,7 +84,11 @@ def running(
monkeypatch.setattr("nisse_dev.loop.build.MatrixTransport", lambda cfg: transport) monkeypatch.setattr("nisse_dev.loop.build.MatrixTransport", lambda cfg: transport)
agent = None agent = None
try: try:
agent = build_loop_agent(config) # What the agent reads when told a new project is ready: replaced by
# a test that adds one (D8).
source = [config]
agent = build_loop_agent(config, load=lambda: source[0])
agent.source = source # type: ignore[attr-defined]
# The test drives intake and PR watching itself, not on a timer. # The test drives intake and PR watching itself, not on a timer.
agent.loop._interval = 3600 agent.loop._interval = 3600
agent.start() agent.start()
@@ -94,7 +104,7 @@ def running(
state_dir=tmp_path / "ssh", state_dir=tmp_path / "ssh",
) )
host.execute(f"rm -rf /tmp/nisse-e2e-{run}") host.execute(f"rm -rf /tmp/nisse-e2e-{run}")
stack.close() # the run's base branch stack.close() # the run's repository
def is_artifact(path: str) -> bool: def is_artifact(path: str) -> bool:
@@ -107,8 +117,14 @@ def is_artifact(path: str) -> bool:
def merge(gitea: GiteaClient, repo: str, pr: int, wait_s: float = 60) -> None: def merge(gitea: GiteaClient, repo: str, pr: int, wait_s: float = 60) -> None:
"""Merge, waiting out Gitea's "Please try again later": it answers that """Approve and merge, waiting out Gitea's "Please try again later": it
while it is still checking a freshly pushed PR can merge.""" answers that while it is still checking a freshly pushed PR can merge."""
with contextlib.suppress(GiteaError): # approved already
gitea._request(
"POST",
f"/repos/{repo}/pulls/{pr}/reviews",
json={"event": "APPROVED", "body": "looks good"},
)
deadline = time.monotonic() + wait_s deadline = time.monotonic() + wait_s
while True: while True:
try: try:
@@ -134,11 +150,13 @@ class Operator:
"""Intake and PR watching, as the agent's timer would do.""" """Intake and PR watching, as the agent's timer would do."""
self.agent.worker.call(self.agent.loop.intake, timeout=DEADLINE_S) self.agent.worker.call(self.agent.loop.intake, timeout=DEADLINE_S)
project: str | None = None # whose issues ``issue`` numbers, if not the only
def item(self, issue: int = 0) -> Any: def item(self, issue: int = 0) -> Any:
"""The work item for ``issue``; by default the first one.""" """The work item for ``issue``; by default the first one."""
if issue: if issue:
store = self.agent.store store, project = self.agent.store, self.project
return self.agent.worker.call(lambda: item_for_issue(store, issue)) return self.agent.worker.call(lambda: item_for_issue(store, issue, project))
items = self.agent.worker.call(self.agent.store.work_items) items = self.agent.worker.call(self.agent.store.work_items)
return items[0] if items else None return items[0] if items else None
@@ -481,3 +499,170 @@ def test_a_two_milestone_feature(
operator.merge_and_close(issues[0]) operator.merge_and_close(issues[0])
finally: finally:
operator.tidy(issues, prs) operator.tidy(issues, prs)
def test_a_conversation_becomes_filed_work(
running: tuple[RunningLoop, InMemoryTransport, str],
) -> None:
"""M2.5: a top-level message is a conversation; the work it turns into
is proposed, agreed, filed as a record issue, and started as agreed."""
agent, transport, run = running
operator = Operator(agent, transport)
gitea, repo = agent.gitea, agent.config.gitea_repo
issues: list[int] = []
prs: list[int] = []
def conversation() -> Any:
items = agent.worker.call(agent.store.work_items)
return next(i for i in items if i.scope == "general")
try:
root = transport.receive(
OPERATOR,
f"I've been thinking the sandbox could use this: {task(run)} "
"Could you file that as an issue and start on it?",
)
operator.settle()
deadline = time.monotonic() + DEADLINE_S
answers = 0
while loop_of(conversation()).wait != GO_AHEAD:
assert time.monotonic() < deadline, f"never proposed: {operator.last()}"
answers += 1
assert answers <= MAX_ANSWERS, f"never proposed: {operator.last()}"
transport.receive(
OPERATOR,
"Yes, propose it as an issue in the sandbox project.",
thread=root,
)
operator.settle()
assert operator.last().startswith(f"rev-{loop_of(conversation()).rev}:")
transport.receive(OPERATOR, "go", thread=root)
operator.settle()
[filed] = loop_of(conversation()).became
number = int(filed.split("#")[1])
issues.append(number)
data = gitea._request("GET", f"/repos/{repo}/issues/{number}")
assert "<!-- nisse-item: " in data["body"]
assert agent.account in [a["login"] for a in data["assignees"] or []]
item = operator.drive_until(MERGE, number)
assert loop_of(item).started # no second restatement
prs.append(loop_of(item).pr)
pr = gitea._request("GET", f"/repos/{repo}/pulls/{prs[0]}")
assert f"Closes #{number}" in pr["body"]
finally:
operator.tidy(issues, prs)
def test_a_conversation_becomes_a_new_project(
running: tuple[RunningLoop, InMemoryTransport, str],
) -> None:
"""M2.5: a conversation proposes a new project; the operator provides
what the checklist asks for (here, a repository of its own made the way
fresh_repo makes one, and the entry the agent wrote); "done" has the
agent check it all, open the tracking issue, and start the architecture."""
import yaml
from nisse_dev.config import _parse_projects
agent, transport, run = running
operator = Operator(agent, transport)
as_operator = operator_gitea(agent.config.gitea_url)
name = f"tally-{run}"
def conversation() -> Any:
items = agent.worker.call(agent.store.work_items)
return next(i for i in items if i.scope == "general")
root = transport.receive(
OPERATOR,
f"I'd like a new project, named {name}: a tiny command-line tally "
"counter in Python 3 with pytest, no services, secrets or extra network "
"access. Please propose it as a new project.",
)
operator.settle()
for _ in range(MAX_ANSWERS):
if loop_of(conversation()).wait == GO_AHEAD:
break
transport.receive(
OPERATOR, f"Yes: propose it as a new project named {name}.", thread=root
)
operator.settle()
assert loop_of(conversation()).proposal_kind == "project", operator.last()
transport.receive(OPERATOR, "go", thread=root)
operator.settle()
checklist = operator.last()
entry = re.search(r"```yaml\n(.*?)\n```", checklist, re.DOTALL)
assert entry is not None, checklist
# --- the operator provides it all, as the checklist asks ---
keep = agent.config.state_db.parent
with fresh_repo(
as_operator, f"{run}-new", keep / "keys", agent=agent.account, seed=False
) as repo:
host = agent.config.build_host
filled = (
entry.group(1)
.replace(f"<owner>/{name}", repo.name)
.replace("<the build VM>", host.address)
.replace("<my SSH key for it>", str(host.key))
.replace("<the deploy key's private half>", str(repo.push_key))
)
errors: list[str] = []
[project] = _parse_projects(yaml.safe_load(filled)["projects"], {}, errors)
assert errors == [], errors
# On this run's build host and in its scratch directory.
project = replace(
project, build_host=host, workspace=Path(f"/tmp/nisse-e2e-{run}/{name}")
)
agent.source[0] = replace( # type: ignore[attr-defined]
agent.config, projects=(*agent.config.projects, project)
)
transport.receive(OPERATOR, "done", thread=root)
operator.settle()
became = loop_of(conversation()).became
assert became, f"not set up: {operator.last()}"
number = int(became[0].split("#")[1])
operator.project = name
item = operator.drive_until(MERGE, number)
assert "design" in loop_of(item).labels
pr = loop_of(item).pr
files = [
f["filename"]
for f in agent.gitea._request("GET", f"/repos/{repo.name}/pulls/{pr}/files")
]
assert "ARCHITECTURE.md" in files, files
assert any(f.startswith("docs/plans/") for f in files), files
def test_a_question_leaves_the_restatement_up(
running: tuple[RunningLoop, InMemoryTransport, str],
) -> None:
"""M2.7: a question while a restatement is up is answered, and changes
nothing: the same revision stays up, and a go binds to it."""
agent, transport, run = running
operator = Operator(agent, transport)
repo = agent.config.gitea_repo
as_operator = operator_gitea(agent.config.gitea_url)
issues: list[int] = []
try:
issues.append(as_operator.create_issue(repo, f"vowels_{run}", task(run)))
as_operator._request(
"PATCH",
f"/repos/{repo}/issues/{issues[0]}",
json={"assignees": [agent.account]},
)
operator.tick()
item = operator.drive_until(GO_AHEAD, issues[0])
rev = loop_of(item).rev
said = len(transport.outbound)
operator.say("Why put it in that module rather than a new one?", issues[0])
answer = [m.text for m in transport.outbound[said:]]
assert answer and not answer[-1].startswith("rev-"), answer
loop = loop_of(operator.item(issues[0]))
assert (loop.wait, loop.rev) == (GO_AHEAD, rev), loop
operator.say("go", issues[0])
assert loop_of(operator.item(issues[0])).started
finally:
operator.tidy(issues, [])
+1 -1
View File
@@ -43,5 +43,5 @@ def test_dev_entry_point_refuses_to_start_unconfigured(tmp_path: Path) -> None:
assert result.returncode == 2 assert result.returncode == 2
assert "cannot start" in result.stderr assert "cannot start" in result.stderr
# All of them at once, not the first one it happened to reach. # All of them at once, not the first one it happened to reach.
for expected in ("state_db", "matrix.token", "gitea.repo"): for expected in ("state_db", "matrix.token", "projects: must name at least one"):
assert expected in result.stderr assert expected in result.stderr
+33
View File
@@ -0,0 +1,33 @@
"""The per-run test repository (e2e_support.fresh_repo): what the operator
provides for a project (D4, D8), made and deleted by the test operator."""
import uuid
from pathlib import Path
import pytest
from e2e_support import fresh_repo, operator_gitea
from nisse_dev.config import AgentConfig
from nisse_dev.gitea import GiteaClient, GiteaError
pytestmark = pytest.mark.integration
def test_a_fresh_repository_is_seeded_shared_and_protected(
tmp_path: Path, live_config: AgentConfig, gitea: GiteaClient
) -> None:
operator = operator_gitea(live_config.gitea_url)
run = uuid.uuid4().hex[:8]
with fresh_repo(operator, run, tmp_path / "keys", agent=gitea.whoami()) as repo:
# Seeded: the package and its tests, and the README replaced.
text = gitea._request("GET", f"/repos/{repo.name}/contents/sandbox/text.py")
assert text["name"] == "text.py"
assert gitea._request("GET", f"/repos/{repo.name}/contents/tests")
# The agent sees it as a collaborator that may push, and main as
# protected against it (what D4's verification reads).
main = gitea._request("GET", f"/repos/{repo.name}/branches/main")
assert main["protected"] and not main["user_can_push"]
assert gitea._request("GET", f"/repos/{repo.name}")["permissions"]["push"]
assert repo.push_key.exists()
with pytest.raises(GiteaError, match="404"):
gitea._request("GET", f"/repos/{repo.name}")
+22 -12
View File
@@ -1,19 +1,18 @@
"""Live Gitea round trip against the sandbox repo: push a branch, open a PR, """Live Gitea round trip in a repository of the run's own: push a branch,
comment, and prove the L8 reconciler reads external truth correctly. open a PR, comment, and prove the L8 reconciler reads external truth
correctly, and that protection holds against the agent (D4).
Cleans up after itself (closes the PR, deletes the branch) so repeated runs
do not silt up the sandbox.
""" """
import uuid import uuid
from pathlib import Path from pathlib import Path
import pytest import pytest
from e2e_support import fresh_repo, operator_gitea
from nisse_dev.agent import push_credential, remote_url from nisse_dev.agent import remote_url
from nisse_dev.config import AgentConfig from nisse_dev.config import AgentConfig
from nisse_dev.execution import SSHHost from nisse_dev.execution import SSHHost
from nisse_dev.git import GitError, GitRepo from nisse_dev.git import GitError, GitRepo, SSHKey
from nisse_dev.gitea import GiteaClient, pull_request_reconciler from nisse_dev.gitea import GiteaClient, pull_request_reconciler
from nisse_dev.state import Intent from nisse_dev.state import Intent
@@ -34,9 +33,21 @@ def intent_for(branch: str) -> Intent:
def test_push_open_pr_and_reconcile( def test_push_open_pr_and_reconcile(
tmp_path: Path, live_config: AgentConfig, gitea: GiteaClient, ssh_host: SSHHost tmp_path: Path, live_config: AgentConfig, gitea: GiteaClient, ssh_host: SSHHost
) -> None: ) -> None:
repo_path = live_config.gitea_repo # the sandbox
client = gitea client = gitea
with fresh_repo(
operator_gitea(live_config.gitea_url),
uuid.uuid4().hex[:8],
tmp_path / "keys",
agent=client.whoami(),
) as fresh:
_round_trip(
tmp_path, client, ssh_host, fresh.name, SSHKey(fresh.push_key.read_text())
)
def _round_trip(
tmp_path: Path, client: GiteaClient, ssh_host: SSHHost, repo_path: str, key: SSHKey
) -> None:
account = client.whoami() account = client.whoami()
assert "testing" in account, ( assert "testing" in account, (
f"live tests must run as a testing account, not {account}" f"live tests must run as a testing account, not {account}"
@@ -49,12 +60,12 @@ def test_push_open_pr_and_reconcile(
assert reconcile(intent_for(branch)) is False assert reconcile(intent_for(branch)) is False
# Pushed the way the agent does: from the build VM (here the throwaway # Pushed the way the agent does: from the build VM (here the throwaway
# sshd), with the push credential the config names, never the API token. # sshd), with the repository's deploy key, never the API token.
key = push_credential(live_config)
assert key is not None, "nisse-test.yaml names no push credential (D4)"
url = remote_url(client, repo_path, key) url = remote_url(client, repo_path, key)
repo = GitRepo(ssh_host, tmp_path / "work", known_hosts=str(tmp_path / "kh")) repo = GitRepo(ssh_host, tmp_path / "work", known_hosts=str(tmp_path / "kh"))
repo.init() repo.init()
repo._git("fetch", "-q", url, "main", credential=key)
repo._git("checkout", "-q", "-B", "work", "FETCH_HEAD")
repo.configure_identity(account, f"{account}@example.test") repo.configure_identity(account, f"{account}@example.test")
(repo.root / f"{branch}.txt").write_text("live test artifact\n") (repo.root / f"{branch}.txt").write_text("live test artifact\n")
assert repo.commit_all("Live test commit") is not None assert repo.commit_all("Live test commit") is not None
@@ -93,5 +104,4 @@ def test_push_open_pr_and_reconcile(
finally: finally:
if pr is not None: if pr is not None:
client.close_pull_request(repo_path, pr.number) client.close_pull_request(repo_path, pr.number)
repo.delete_remote_branch(url, branch, credential=key)
client.close() client.close()