dev/nisse
2
1 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
9968a0922af0abaa756b08b8945f5df8b5ec6182
nisse/nisse-dev/src/nisse_dev/loop/agent.py
T
dongarandClaude Opus 5.5 9968a0922a
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 2m59s
Details
CI / e2e (pull_request) Successful in 10m33s
Details
M4.1: toolchain preflight 
A model works out and sets up what a project needs on its build VM;
code makes sure that happens, bounds it and checks the result (D3).

- Before each driver run on a project, one round trip checks the build
  VM: the harness's own baseline (missing: the item waits, a template
  problem), the workspace clone (missing: cloned again and said; the
  item resumes from what it pushed) and a per-project marker.
- No marker (a new project, a new or wiped VM): the item's driver model
  runs a bounded preflight first, with its own prompt, the execution
  tools (screened as ever) and toolchain_ready. The harness runs each
  reported version command itself (version-shaped commands only, even
  from the marker, which the model's commands could edit) and writes
  the marker. Gaps and an unfinished preflight are facts and work goes
  on; the thread hears only of installs and gaps. A rebuilt VM starts
  from the last toolchain.
- Later runs check the versions again; a change is a fact and a thread
  line. The PR body says what the work was tested with.
- A changed host key is its own failure (HostKeyChanged): the item
  waits for the operator's `trust new host key`, which drops the pin;
  then the VM is set up again.
- A prompt line: install what the work needs on the build VM, say why,
  and put lasting needs in the project's own pins.
- tuning.preflight: max_turns, command_timeout_s.
- The property walk gains a VM wipe and ready_before_work (checked
  against a mutant that skips preflight); parked_work_safe accepts that
  a wipe loses what wasn't pushed (D3).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE
2026-10-08 15:43:30 +00:00

2841 lines
121 KiB
Python
Raw Blame History

"""The loop's scheduler (L1, L2, L5, D1): intake, messages and driver runs.
Everything durable happens on one worker thread, in order. Inputs (an
operator's message, an issue assigned to the agent) are handed to it as they
arrive; a message is stamped with the revision on screen in its thread first,
because by the time the worker reaches it a newer one may be up (L3).
One work item at a time is active. An item runs until its driver calls a
waiting tool; it then waits on the operator, the active pointer is cleared
(a safe point, L1), and the next ready item runs: the operator's order when
given ("do #14 next"), otherwise the oldest. Switching parks the item that was
on the workspace: its work is committed and its branch pushed (D4). An issue
with open Gitea dependencies is blocked until they close. Conversation is not work:
a message is recorded and changes its item's state, but never interrupts a
run (L1).
"""
import contextlib
import json
import re
import threading
import time
import uuid
from collections.abc import Callable, Iterable, Sequence
from dataclasses import dataclass, replace
from datetime import UTC, datetime
from typing import Any
from nisse_core.actions import DANGEROUS, REFUSED, UNKNOWN, Action, Verdict
from nisse_dev.cancel import Cancel
from nisse_dev.canonical import (
ActionVerdict,
DelegationCompleted,
DelegationStarted,
Fact,
PlanSubmitted,
StoredEvent,
ToolCall,
ToolResult,
UserMessage,
)
from nisse_dev.config import GENERAL
from nisse_dev.delegation import DELEGATION_INTENT, Delegation
from nisse_dev.execution import HostKeyChanged, HostUnavailable
from nisse_dev.expert import BudgetExhausted, ExpertDesk
from nisse_dev.git import GitError
from nisse_dev.gitea import GiteaClient, GiteaError, Issue, PullRequest
from nisse_dev.llm import LLMError, RoleMap
from nisse_dev.loop import preflight
from nisse_dev.loop.actions import classify, commits
from nisse_dev.loop.classifier import IN_SCOPE, ActionClassifier, Scope
from nisse_dev.loop.driver import ASK_ALLOW, Driver, RunOutcome, Screening
from nisse_dev.loop.help import ASK_EXPERT, DELEGATE, Helpers
from nisse_dev.loop.items import (
ANSWER,
BLOCKED,
CHAT,
CLOSED,
GO_AHEAD,
MERGE,
PROVISION,
READY,
RUNNING,
WAITING,
Loop,
item_for_issue,
item_for_thread,
loop_of,
save,
)
from nisse_dev.loop.milestones import OPEN_PR, Milestones
from nisse_dev.loop.reader import GoAheadReader, command_form, is_question
from nisse_dev.loop.tools import (
ASK_HUMAN,
CLOSE_ITEM,
FINISH_MILESTONE,
PROPOSE_PROJECT,
PROPOSE_WORK,
REPLY,
RESTATE,
START_MILESTONE,
WAIT_FOR_REVIEW,
)
from nisse_dev.oplog import OpLog
from nisse_dev.state import StateStore, WorkItem
from nisse_dev.tools import EXECUTION_TOOLS, Workspace
from nisse_dev.transport import InboundMessage, Transport
from nisse_dev.worker import Worker
INTAKE_INTERVAL_S = 30.0
_GO = frozenset({"go", "approve"})
# An issue, named: "#14", "14", or with its project, "sandbox#14" (D8).
_ISSUE = r"(?:([a-z0-9][a-z0-9-]*)#|#)?(\d+)"
_NEXT = re.compile(rf"(?:do|work on) {_ISSUE} (?:next|first|now)")
# A command that names its item applies to it, wherever it is said (L5).
_NAMED = re.compile(r"(go|approve|abandon|stop|allow) (?:([a-z0-9][a-z0-9-]*))?#(\d+)")
# At top level these need an item; without one they are ambiguous (L5).
# Added to a command's result when it commits (#129).
_COMMITTED = (
"committed locally only: nothing is published until you call "
"finish_milestone, which pushes the milestone's branch and opens or "
"updates its PR. There is no PR until then."
)
_NEEDS_ITEM = frozenset({"go", "approve", "abandon", "allow", "trust new host key"})
# Near-misses for abandon: asked about, never taken for it or for a reply.
_NEAR_ABANDON = frozenset(
{
"abort",
"abort it",
"cancel",
"cancel it",
"drop it",
"quit",
"kill it",
"scrap it",
}
)
BUG_GRACE_S = 600.0
GENERAL_IDLE_S = 86400.0
PROGRESS_EVERY_S = 60.0 # progress in prose at a readable rate (L5, #42)
PROGRESS_CHARS = 400
DETAILS = 10 # tool calls shown by `details`
FILE_ISSUE = "file_issue" # the intent of opening a record issue (L8)
# Carried by a record issue the agent opens for work started in conversation:
# intake never takes it for new work, even before its item exists (D1).
_MARKER = re.compile(r"<!-- nisse-item: ([A-Za-z0-9-]+) -->")
# Before the go-ahead the driver may look around (read, list, run a command
# to reproduce a bug) but not change anything (D1); after it, everything.
_LOOK = [t for t in EXECUTION_TOOLS if t["function"]["name"] != "write_file"]
SYSTEM = """You are nisse-dev, a software agent working with the operator on
the repository {repo}. You are on one work item: issue #{issue}, "{title}".
The issue and the conversation about it are below.
Before any work, find out what you need, then use restate to put your
understanding to the operator: what you take the request to mean, how it fits
the project, your first step, and why that way rather than the main
alternative. A later restatement says what changed. Wait for their go-ahead.
If the request is unclear, ask_human instead of guessing: a restatement asks
no questions, so the operator can agree to it as it stands. At any point,
asking is better than guessing.
Only the operator stops or closes an item (their stop and abandon commands):
never say you have stopped or closed anything yourself.
Commands that reach beyond this repository and its build VM (pushing, other
hosts, uploads, publishing) or could break the host are put to the operator
before they run; if they allow one, run it again exactly as before.
The build VM is the project's: if the work needs a tool or library, install
it there and say why, and put lasting needs in the project's own pins
(.python-version, rust-toolchain.toml, requires-python, engines) so they're
reviewed with the work.
Never change branches yourself.
Before each group of tool calls, say in a sentence what you're doing and why:
the operator sees it as your progress.
When the operator says something that needs an answer but no change (a
remark, a status question), answer with reply: the item goes back to what it
was waiting for. Every reply of yours is a tool call, never bare prose.
The project's documents are in its repository: read ARCHITECTURE.md and any
build plans under docs/plans/ before you restate.
After the go-ahead, work in milestones, one PR each. Call start_milestone
before changing anything (never create or switch branches yourself; each
milestone starts from the latest default branch), make the change, run the
project's tests, then call finish_milestone with a summary, saying whether it
is the final milestone. The harness commits, pushes and opens a PR, and you
wait for review. Review comments arrive as messages: make the changes within
the same milestone (don't start another), run the tests, and call
finish_milestone again to push the fix. When a review needs no change from
you, or you are only waiting for the operator to merge (you can't merge it
yourself), call wait_for_review. The next milestone starts only after the
previous PR has merged. A milestone's summary says what was done, what was
verified, and what is left open. When the last PR has merged, call close_item with
what was done, what was verified, and what is left open. If a PR is closed
without merging, ask the operator what next.
Work longer than about one PR gets a build plan: a Markdown file under
docs/plans/ giving, per milestone, the goal, the interfaces and behaviour, the
edge cases, and the tests that are its acceptance criteria. Commit it with the
first milestone, and mark each milestone done in it in that milestone's PR.
Keep ARCHITECTURE.md current: a design change updates it in the same PR.
{labels}{helpers}Never label the issue yourself; you may suggest a label to the operator.
{workspace}"""
QUESTION = """
The operator has asked a question (the last message). Answer it with reply,
from what you know and what you can look at; don't change anything. The item
then goes back to waiting exactly as it was, so a restatement up stays up."""
CONVERSATION = """You are nisse-dev, a software agent, in a conversation with
the operator in a Matrix thread. It isn't about a project yet: it may be a
question, thoughts on feasibility or hardware, or the start of some work.
Answer with reply; ask_human only for a question you need answered. You have
no workspace and can't run anything here; if an experiment is needed, say so.
If the conversation turns into work on one of these projects, use
propose_work, with a summary of what was decided (not the conversation): it
is the new issue's body and the work's whole brief.
{projects}
If it becomes a new project, use propose_project: its name, what it's for,
and the decisions setting it up depends on. Once agreed, you'll be asked for
what it needs set up; when the operator has it ready and it checks out, its
architecture and build plan are the first piece of work."""
# What an issue's labels add to the driver's instructions (D1).
LABELS = {
"bug": "This issue is labelled bug. Before restating, reproduce it with "
"your tools (run the code or the tests). If you can't reproduce it, "
"ask_human instead of restating: never fix a bug you couldn't see. Once "
"you restate, the fix goes ahead after a grace period unless the operator "
"replies; make it a failing test first, then the fix.",
"discuss": "This issue is labelled discuss: talk it through with the "
"operator (ask_human) before restating, and restate only once you agree "
"on what is wanted.",
"design": "This issue is labelled design: the design comes before any "
"code. Your first milestone is documents only: ARCHITECTURE.md and a "
"build plan under docs/plans/, in a PR of their own (if you restate, put "
"the design to the operator). Code starts in a later milestone, after that "
"PR has merged; the harness refuses a design milestone with anything else.",
}
@dataclass(frozen=True)
class Project:
"""What the loop needs of one project (D8): its repository, label
defaults, and its own workspace and milestones on its build VM."""
name: str
repo: str
workspace: Workspace | None = None
milestones: Milestones | None = None
labels: tuple[str, ...] = ()
base: str = "main"
merge: str = "operator" # the trust dial (L3): who merges its PRs
def _restatement_problem(args: dict[str, object]) -> str | None:
"""L5, D1: a restatement is agreed to as it stands, so it asks nothing.
One that ends in a question is sent back: questions go by ask_human."""
text = str(args.get("text", "")).strip()
if not text:
return "a restatement needs text: what you take the request to mean"
if is_question(text):
return (
"not posted: a restatement asks no questions, so the operator can "
"agree to it as it stands. If you need an answer first, use "
"ask_human; otherwise restate it as a statement."
)
return None
class LoopAgent:
def __init__(
self,
store: StateStore,
transport: Transport,
gitea: GiteaClient,
*,
account: str,
operators: Iterable[str],
gitea_operators: Iterable[str],
driver: Driver,
worker: Worker,
cancel: Cancel,
oplog: OpLog,
projects: Sequence[Project] = (),
# One project, given directly: the same as projects=[Project(...)].
repo: str = "",
project: str = "",
workspace: Workspace | None = None,
roles: dict[str, str] | None = None,
milestones: Milestones | None = None,
reader: GoAheadReader | None = None,
bug_grace_s: float = BUG_GRACE_S,
general_idle_s: float = GENERAL_IDLE_S,
# A new project's entry, once the operator has added it (D1, D8):
# the projects configured now that this agent doesn't know yet.
reload: Callable[[set[str]], Sequence[Project]] | None = None,
helpers: Helpers | None = None,
classifier: ActionClassifier | None = None,
preflight: bool = False,
preflight_turns: int = 20,
preflight_timeout_s: int = 900,
markers: str = preflight.MARKERS,
clock: Callable[[], float] = time.time,
intake_interval_s: float = INTAKE_INTERVAL_S,
) -> None:
self._store = store
self._transport = transport
self._gitea = gitea
if not projects:
projects = [Project(project, repo, workspace, milestones)]
self._projects = {p.name: p for p in projects}
self._account = account
self._operators = frozenset(operators)
self._gitea_operators = frozenset(gitea_operators)
self._driver = driver
self._worker = worker
self._cancel = cancel
self._oplog = oplog
self._roles = roles or {}
self._reader = reader
self._grace = bug_grace_s
self._idle = general_idle_s
self._reload = reload
self._helpers = helpers
# L4: judges what the fast path doesn't know; without one, it runs.
self._classifier = classifier
# Toolchain preflight (M4.1): off where a test drives the loop
# without it.
self._preflight = preflight
self._preflight_turns = preflight_turns
self._preflight_timeout_s = preflight_timeout_s
self._markers = markers # where the build VM keeps them
# The trust dial (M3.3): projects whose protection was checked, and
# whether the harness may merge their approved PRs.
self._merging: dict[str, bool] = {}
# Verdicts by (item, agreed revision, action): judged once (M3.2).
self._judged: dict[tuple[str, int, str], tuple[Verdict, tuple[str, str]]] = {}
self._clock = clock
self._interval = intake_interval_s
# thread root -> revision on screen there, read on the transport's
# thread to stamp replies on arrival (L3); (project, issue) -> thread
# root, for a command that names its item.
self._shown: dict[str, int] = {}
self._threads: dict[tuple[str, int], str] = {}
self._active_thread = ""
# The project whose workspace was last worked on: switching to
# another parks it (L1). None after a boot: unknown, so all are.
self._last_project: str | None = None
# Per item: when progress was last posted (L5: a readable rate).
self._progressed: dict[str, float] = {}
self._active_name = "" # the running item, for "noted" on arrival
self._running = False
self._stop = threading.Event()
self._intake_thread: threading.Thread | None = None
# -- projects (D8) ----------------------------------------------------------
def _at(self, item: WorkItem) -> Project:
if item.scope == GENERAL:
# A conversation: no repository, workspace or build VM (L1).
return Project(GENERAL, "")
return self._projects[item.scope or ""]
def _label(self, item: WorkItem) -> str:
"""How an item's issue is named to the operator: #14 with one
project, sandbox#14 with more, where #14 alone could be either."""
if item.scope == GENERAL:
return f'the conversation "{item.title}"'
issue = loop_of(item).issue
return f"#{issue}" if len(self._projects) == 1 else f"{item.scope}#{issue}"
def _named(self, project: str | None, issue: int) -> list[WorkItem]:
"""Open items for ``issue``, in ``project`` if it is given."""
return [
i
for i in self._store.work_items("open")
if i.scope != GENERAL
and loop_of(i).issue == issue
and (project is None or i.scope == project)
]
def _repos(self) -> str:
return ", ".join(p.repo for p in self._projects.values())
# -- lifecycle ----------------------------------------------------------
def start(self) -> None:
self._worker.call(self.boot)
self._transport.on_message(self.on_message)
start = getattr(self._transport, "start", None)
if start is not None:
start()
self._intake_thread = threading.Thread(target=self._intake_loop, daemon=True)
self._intake_thread.start()
def stop(self) -> None:
self._stop.set()
if self._intake_thread is not None:
self._intake_thread.join(timeout=5)
def _intake_loop(self) -> None:
while not self._stop.is_set():
self._worker.submit(self._settled("intake", self.intake))
self._stop.wait(self._interval)
def _settled(self, what: str, job: Callable[[], None]) -> Callable[[], None]:
"""A worker job that reports a failure instead of losing it."""
def run() -> None:
try:
job()
except Exception as exc: # noqa: BLE001 -- reported, the loop goes on
self._oplog.emit("loop.job_failed", cause=what, error=repr(exc))
self._transport.post(f"something went wrong with {what}: {exc}")
return run
# -- boot (L8) --------------------------------------------------------------
def boot(self) -> None:
"""Resume waiting points as they were, and never resume interrupted
work blindly: an item cut off mid-run waits on the operator."""
self._store.set_active(None, cause="boot")
for project in self._projects.values():
if project.merge == "approved":
self._may_merge(project) # D4: verified at start, not assumed
self._reconcile_publishing()
self._reconcile_filing()
self._reconcile_delegations()
for item in self._store.work_items("open"):
loop = loop_of(item)
self._threads[(item.scope or "", loop.issue)] = loop.thread
if item.position == RUNNING:
note = "I restarted while working on this. Reply to continue."
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
elif item.position == WAITING and loop.thread:
if loop.wait == GO_AHEAD:
self._shown[loop.thread] = loop.rev
self._transport.post(
f"(after a restart) rev-{loop.rev} is still up; say go "
"to start, or reply with changes.",
thread=loop.thread,
)
elif loop.wait != CHAT: # a conversation just goes on
self._transport.post(
f"(after a restart) still waiting: {loop.note}",
thread=loop.thread,
)
# -- the build VM: ready, and its toolchain (D3; M4.1) ----------------------
def _readiness(
self, item: WorkItem, project: Project, loop: Loop
) -> tuple[Loop, dict[str, Any] | None] | None:
"""The build VM as the harness relies on it: reachable with the key
it pinned, its baseline there, the workspace cloned (again, if the
VM was rebuilt or wiped). None when the item waits on the operator."""
assert project.workspace is not None and project.milestones is not None
host = project.workspace.host
root = str(project.workspace.root)
try:
ready = preflight.check(host, project.name, root, self._markers)
except HostKeyChanged:
note = (
f"the build VM for {project.name} ({host.describe()}) answered "
"with a different host key than the one I pinned. If you "
"rebuilt it, say `trust new host key` and I'll set it up "
"again; if you didn't, something is impersonating it."
)
self._wait(
item, replace(loop, wait=ANSWER, note=note, host_key_changed=True), note
)
self._store.set_active(None, cause="host key changed")
return None
except HostUnavailable as exc:
note = f"I can't reach the build VM: {exc}. Reply to retry."
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
self._store.set_active(None, cause="build VM unreachable")
return None
if ready.missing:
note = (
f"the build VM for {project.name} lacks {', '.join(ready.missing)}, "
"which I rely on there: that's for its template. Reply once it's "
"fixed."
)
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
self._store.set_active(None, cause="build VM baseline missing")
return None
if not ready.cloned:
try:
project.milestones.reclone(self._account)
except Exception as exc: # noqa: BLE001 -- the item waits, not the loop
note = f"I couldn't clone the workspace again: {exc}. Reply to retry."
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
self._store.set_active(None, cause="re-clone failed")
return None
if loop.prepared:
said = (
"the build VM was rebuilt or wiped: I cloned the workspace "
"again; work that wasn't pushed is gone, and pushed branches "
"come back from Gitea"
)
self._fact(item, said)
self._transport.post(
said[0].upper() + said[1:] + ".", thread=loop.thread
)
# Resuming takes the item's branch from Gitea when it was
# pushed, or starts it again from the default branch.
return loop, ready.marker
def _toolchain(
self,
item: WorkItem,
project: Project,
loop: Loop,
marker: dict[str, Any] | None,
starting: bool,
) -> Loop | None:
"""Preflight when the VM has no marker for the project (or only an
unfinished one, at an item's start); otherwise the toolchain's
versions checked again, and a change said."""
assert project.workspace is not None
if marker is None or (starting and not marker.get("complete")):
return self._preflight_run(item, project, loop, marker)
host = project.workspace.host
before = preflight.tools_of(loop.toolchain) or preflight.tools_of(
marker.get("toolchain")
)
try:
now = preflight.versions(host, before)
except HostUnavailable:
return loop # the run will find out, and say so
changed = preflight.drift(before, now)
if changed:
said = "the toolchain changed since it was checked: " + "; ".join(changed)
self._fact(item, said)
self._transport.post(said[0].upper() + said[1:] + ".", thread=loop.thread)
if starting:
self._fact(item, f"toolchain: {preflight.said(now)}")
return replace(loop, toolchain=tuple(tuple(t.to_data()) for t in now))
def _preflight_run(
self,
item: WorkItem,
project: Project,
loop: Loop,
marker: dict[str, Any] | None,
) -> Loop | None:
"""D3: before any planning, the item's driver gets the project's
toolchain ready; the harness checks what it reports."""
assert project.workspace is not None
workspace = project.workspace.with_timeout(self._preflight_timeout_s)
last = preflight.tools_of((marker or {}).get("toolchain"))
system = preflight.SYSTEM.format(
repo=project.repo,
workspace=workspace.describe(),
last=(
f"Last time, this project's toolchain was: {preflight.said(last)}. "
"Set that up again, then check whether it needs anything else.\n"
if last
else ""
),
)
self._fact(item, "preflight: getting the project's toolchain ready")
roles = self._roles_of(item)
self._active_thread = loop.thread
self._active_name = self._label(item)
self._running = True
try:
outcome = self._driver.run(
self._store.transcript(item.id, "driver"),
system=system,
tools=[*EXECUTION_TOOLS, preflight.TOOLCHAIN_READY],
workspace=workspace,
checks={"toolchain_ready": preflight.report_problem},
model=roles.resolve("driver"),
screen=lambda call: self._screen(item.id, str(workspace.root), call),
max_turns=self._preflight_turns,
)
finally:
self._running = False
if outcome.kind == "stopped":
self._cancel.clear()
self._settle(item.id, outcome)
return None
host = workspace.host
now = loop_of(self._store.work_item(item.id) or item)
if outcome.kind == "wait" and outcome.tool == "toolchain_ready":
report = preflight.report_of(outcome.arguments or {})
installed = [
str(i) for i in (outcome.arguments or {}).get("installed") or []
]
complete = True
else:
report = preflight.Report(
f"the preflight didn't finish ({outcome.note or outcome.kind})", last
)
installed, complete = [], False
try:
tools = preflight.versions(host, report.tools)
preflight.write_marker(
host,
project.name,
markers=self._markers,
data={
"toolchain": [t.to_data() for t in tools],
"complete": complete,
"item": item.id,
"at": datetime.now(UTC).isoformat(timespec="seconds"),
},
)
except HostUnavailable:
tools = report.tools
missing = [f"{name} ({why})" for name, why in report.missing]
parts = [f"toolchain: {preflight.said(tools)}"]
if installed:
parts.append("installed: " + ", ".join(installed))
if missing:
parts.append("missing: " + "; ".join(missing))
if not complete:
parts.append(report.summary)
said = "; ".join(parts)
self._fact(item, f"preflight: {said}")
if installed or missing or not complete:
self._transport.post(f"Preflight: {said}.", thread=loop.thread)
return replace(now, toolchain=tuple(tuple(t.to_data()) for t in tools))
# -- the trust dial (L3, D5; M3.3) --------------------------------------------
def _merge_if_approved(
self, item: WorkItem, loop: Loop, repo: str, pr: PullRequest
) -> tuple[PullRequest, Loop]:
"""Merge the PR when an operator approved its current commit and its
checks pass. Gitea enforces the approval (protection); this only
decides when to ask it. Each reason to hold off is said once."""
try:
approvals = self._gitea.approvals(repo, pr.number)
checks = self._gitea.checks(repo, pr.head_sha)
except GiteaError as exc:
self._oplog.emit("loop.watch_unavailable", cause=str(exc))
return pr, loop
by = [
a.author
for a in approvals
if a.author in self._gitea_operators
and a.commit == pr.head_sha
and not a.stale
]
if not by:
return pr, loop
held = ""
if checks in ("failure", "error"):
held = f"its checks failed ({checks}), so I'll leave it unmerged"
elif checks not in ("success", "none"):
return pr, loop # pending: wait for them
elif pr.mergeable is False:
held = "it has conflicts with the base branch, so I can't merge it"
elif not self._may_merge(self._at(item)):
return pr, loop # protection unverified: said when checked
if not held:
try:
self._gitea.merge_pull_request(repo, pr.number)
except GiteaError as exc:
held = (
f"Gitea refused my merge ({exc}); I'll try again, or you "
"can merge it"
)
else:
self._fact(item, f"merged PR #{pr.number} after {by[0]}'s approval")
self._transport.post(
f"Merged PR #{pr.number} after {by[0]}'s approval.",
thread=loop.thread,
)
return replace(pr, merged=True, state="closed"), loop
mark = f"{pr.head_sha}:{held}"
if loop.merge_held != mark:
self._transport.post(
f"{by[0]} approved PR #{pr.number}, but {held}.", thread=loop.thread
)
loop = replace(loop, merge_held=mark)
return pr, loop
def _may_merge(self, project: Project) -> bool:
"""D4: protection verified, not assumed. Once per start, a check
that Gitea refuses the agent's merge without an approval; if it
doesn't, the operator merges until it's fixed."""
if project.merge != "approved":
return False
if project.name not in self._merging:
problem = self._merge_protection_problem(project)
self._merging[project.name] = problem is None
self._oplog.emit(
"loop.merge_protection",
cause="start" if problem is None else problem,
project=project.name,
verified=problem is None,
)
if problem is not None:
self._transport.post(
f"{project.repo} is set to have me merge approved PRs, but "
f"{problem}. I'll leave merging to you until that's fixed "
"and I've restarted."
)
return self._merging[project.name]
def _merge_protection_problem(self, project: Project) -> str | None:
"""A throwaway branch and PR, made through the API (never the
workspace, which may hold an item's work): merging it must be
refused. Closed and deleted either way."""
branch = f"nisse-check-{uuid.uuid4().hex[:6]}"
try:
self._gitea.branch_with_file(
project.repo,
base=project.base,
branch=branch,
path=".nisse-merge-check",
text="Checks that merging needs an approval; never merged.\n",
message="nisse-dev: merge check",
)
except GiteaError as exc:
return f"I couldn't make a branch to check merging: {exc}"
pr = None
try:
pr = self._gitea.open_pull_request(
project.repo,
head=branch,
base=project.base,
title="nisse-dev merge check (never merged)",
body="Checks that merging needs an approval; closed straight away.",
)
try:
self._gitea.merge_pull_request(project.repo, pr.number)
except GiteaError:
return None # refused, as it should be
return (
f"I could merge my own PR #{pr.number} without an approval "
f"(it added .nisse-merge-check to {project.base})"
)
except GiteaError as exc:
return f"I couldn't open a PR to check merging: {exc}"
finally:
if pr is not None:
with contextlib.suppress(GiteaError):
self._gitea.close_pull_request(project.repo, pr.number)
with contextlib.suppress(GiteaError):
self._gitea.delete_branch(project.repo, branch)
def _reconcile_delegations(self) -> None:
"""L8, L9: a delegation cut off by a crash is discarded, its branch
deleted and the milestone's left as it was. The item, cut off mid-
run, then waits on the operator like any other."""
for intent in self._store.pending_intents():
if intent.kind != DELEGATION_INTENT or not intent.work_item:
continue
item = self._store.work_item(intent.work_item)
branch = str(intent.payload.get("branch", ""))
base = str(intent.payload.get("base", ""))
project = self._projects.get(item.scope or "") if item else None
milestones = project.milestones if project else None
if milestones is not None and branch and base:
with contextlib.suppress(GitError):
milestones.git.reset_to(base, base)
with contextlib.suppress(GitError):
milestones.git._git("branch", "-q", "-D", branch)
self._store.abort_intent(intent.id, cause="cut off by a restart")
if item is not None:
self._fact(
item,
f"a helper's work on {branch} was cut off by a restart and discarded",
)
def _reconcile_filing(self) -> None:
"""L8: a record issue cut off by a crash. An issue carrying the new
item's ID was opened: its item is started now. Otherwise nothing was
opened, and the conversation's proposal is up again."""
for intent in self._store.pending_intents():
if intent.kind != FILE_ISSUE or not intent.work_item:
continue
conversation = self._store.work_item(intent.work_item)
repo = str(intent.payload.get("repo", ""))
new_id = str(intent.payload.get("item", ""))
project = next((p for p in self._projects.values() if p.repo == repo), None)
if self._store.work_item(new_id) is not None:
self._store.commit_intent(intent.id, cause="its item exists")
continue
if conversation is None or project is None:
self._store.abort_intent(
intent.id, cause="conversation or project gone"
)
continue
try:
issues = self._gitea.assigned_issues(repo, self._account)
except GiteaError:
continue # unverifiable now; settled at the next boot
filed = [
i
for i in issues
if (m := _MARKER.search(i.body or "")) and m.group(1) == new_id
]
if filed:
self._start_filed(
conversation,
project,
new_id,
filed[0].number,
filed[0].title,
self._latest_proposal(conversation),
)
self._store.commit_intent(
intent.id, cause=f"adopted {repo}#{filed[0].number}"
)
else:
self._store.abort_intent(intent.id, cause="no issue was opened")
loop = loop_of(conversation)
note = (
"opening the issue was cut off by a restart; say go to try again."
)
self._shown[loop.thread] = loop.rev
self._wait(conversation, replace(loop, wait=GO_AHEAD, note=note), note)
def _reconcile_publishing(self) -> None:
"""L8 step 2: a publish cut off by a crash is settled against Gitea.
An open PR for the branch is adopted; otherwise the intent is aborted
and the operator is told, never a blind retry."""
for intent in self._store.pending_intents():
if intent.kind != OPEN_PR or not intent.work_item:
continue
item = self._store.work_item(intent.work_item)
branch = str(intent.payload.get("branch", ""))
if item is None or item.scope not in self._projects:
self._store.abort_intent(intent.id, cause="item or project unknown")
continue
try:
pr = self._gitea.find_pull_request(self._at(item).repo, branch)
except GiteaError:
continue # unverifiable: left pending, asked about below
if item is None or item.status != "open":
self._store.abort_intent(intent.id, cause="item no longer open")
elif pr is not None and pr.state == "open":
self._store.commit_intent(intent.id, cause=f"adopted PR #{pr.number}")
loop = replace(loop_of(item), pr=pr.number, pr_url=pr.url)
self._fact(item, f"after a restart, adopted PR #{pr.number} ({pr.url})")
save(
self._store,
item.id,
WAITING,
replace(loop, wait=MERGE, note=f"PR #{pr.number}"),
cause="adopted PR after restart",
)
else:
self._store.abort_intent(intent.id, cause="no PR was opened")
note = "publishing was cut off by a restart; reply to try again."
self._wait(item, replace(loop_of(item), wait=ANSWER, note=note), note)
# -- intake (D1) ---------------------------------------------------------
def intake(self) -> None:
"""Issues assigned to the agent, in every project's repository,
become work items, once each."""
for project in self._projects.values():
self._intake(project)
self.watch()
self.check_blocked()
self.check_grace()
self.check_idle()
self._schedule()
def _intake(self, project: Project) -> None:
try:
issues = self._gitea.assigned_issues(project.repo, self._account)
except GiteaError as exc:
# L5: an outage delays intake, never fails anything.
self._oplog.emit("loop.intake_unavailable", cause=str(exc))
return
for issue in issues:
before = item_for_issue(self._store, issue.number, project.name)
if before is not None and not self._assigned_again(project, issue, before):
continue
if _MARKER.search(issue.body or ""):
continue # D1: a record of work started in conversation
if not self._from_an_operator(project, issue):
self._oplog.emit(
"loop.intake_ignored",
cause="not filed or assigned by an operator",
issue=issue.number,
repo=project.repo,
)
continue
try:
blocked_by = self._gitea.open_dependencies(project.repo, issue.number)
except GiteaError as exc:
self._oplog.emit("loop.intake_unavailable", cause=str(exc))
continue # taken on the next round
self._open(project, issue, tuple(blocked_by), follows=before)
# -- PR review and merge (D4, D5) -------------------------------------------
def watch(self) -> None:
"""New operator comments on an item's PR, and its merge or close,
reach the item; a waiting item becomes ready to act on them."""
for item in self._store.work_items("open"):
loop = loop_of(item)
if not loop.pr:
continue
repo = self._at(item).repo
try:
pr = self._gitea.pull_request(repo, loop.pr)
comments = self._gitea.list_comments(repo, loop.pr)
notes = self._gitea.review_notes(repo, loop.pr)
except GiteaError as exc:
self._oplog.emit("loop.watch_unavailable", cause=str(exc))
continue
log = self._store.transcript(item.id, "operator")
new = False
for comment in comments:
if comment.id <= loop.comments:
continue
loop = replace(loop, comments=comment.id)
if comment.author in self._gitea_operators:
log.append(
UserMessage(
text=comment.body,
source=f"gitea:pr:{loop.pr}:comment:{comment.id}",
)
)
loop = replace(
loop,
asked_on_pr=loop.pr,
asked_where="",
asked_text=comment.body[:200],
)
new = True
for note in notes:
if note.key in loop.notes:
continue
loop = replace(loop, notes=(*loop.notes, note.key))
if note.author in self._gitea_operators:
where = f"on {note.where}: " if note.where else ""
log.append(
UserMessage(
text=f"{where}{note.body}",
source=f"gitea:pr:{loop.pr}:review:{note.key}",
)
)
loop = replace(
loop,
asked_on_pr=loop.pr,
asked_where=note.where,
asked_text=note.body[:200],
)
new = True
position = item.position
if (
not pr.merged
and pr.state == "open"
and not new
and item.position == WAITING
and loop.wait == MERGE
and self._at(item).merge == "approved"
):
pr, loop = self._merge_if_approved(item, loop, repo, pr)
if pr.merged:
self._fact(item, f"PR #{loop.pr} merged")
loop, position = replace(loop, pr=0, pr_url="", branch=""), READY
if "design" in loop.labels and not loop.design_merged:
self._fact(item, "the design has merged: code may follow")
loop = replace(loop, design_merged=True)
elif pr.state != "open":
self._fact(item, f"PR #{loop.pr} was closed without merging")
loop, position = replace(loop, pr=0, pr_url="", branch=""), READY
elif new and item.position == WAITING:
position = READY
# A comment can be answered with a reply, back to this wait.
loop = replace(loop, back_to=loop.wait, back_note=loop.note)
if position != item.position or loop != loop_of(item):
if position == READY:
loop = replace(loop, wait="", note="")
save(self._store, item.id, position, loop, cause="PR update")
def check_blocked(self) -> None:
"""L1: an item that hasn't started running yet is blocked while its
Gitea dependencies are open, and ready once they have all closed.
Only ready and blocked items are checked: the others are mid-work or
waiting on the operator, and an agent that finds a blocker mid-work
asks before recording it."""
for item in self._store.work_items("open"):
if item.position not in (READY, BLOCKED) or item.scope == GENERAL:
continue
loop = loop_of(item)
repo = self._at(item).repo
try:
blocked_by = tuple(self._gitea.open_dependencies(repo, loop.issue))
except GiteaError as exc:
self._oplog.emit("loop.watch_unavailable", cause=str(exc))
continue # as it was, until Gitea answers
position = BLOCKED if blocked_by else READY
if position == item.position and blocked_by == loop.blocked_by:
continue
name = self._label(item)
if position == READY:
said = f"{name} is unblocked: its dependencies have closed."
else:
said = f"{name} is blocked by {', '.join(blocked_by)}."
self._fact(item, said)
if position != item.position:
self._transport.post(said, thread=loop.thread)
save(
self._store,
item.id,
position,
replace(loop, blocked_by=blocked_by),
cause=said,
)
def check_idle(self) -> None:
"""L1: a conversation left without a message for a day closes; a
later reply in its thread starts a new one, linked to it."""
now = self._clock()
for item in self._store.work_items("open"):
loop = loop_of(item)
if (
item.scope == GENERAL
and item.position == WAITING
and not loop.new_project # setting one up can take days
and loop.idle_since
and now - loop.idle_since >= self._idle
):
said = sum(
1
for e in self._store.transcript(item.id, "operator").read()
if e.actor == "operator"
)
became = f"; it became {', '.join(loop.became)}" if loop.became else ""
self._close(
item,
loop,
"idle",
f"no message for {round(self._idle / 3600)} h, after {said} "
f"from the operator{became}",
)
def check_grace(self) -> None:
"""D1: a `bug` restatement left unanswered for the grace period is
taken as agreed, bound to the revision that was up all along."""
now = self._clock()
for item in self._store.work_items("open"):
loop = loop_of(item)
if (
item.position == WAITING
and loop.wait == GO_AHEAD
and loop.grace_until
and now >= loop.grace_until
):
self._transport.post(
f"No reply on rev-{loop.rev}, so I'm going ahead.",
thread=loop.thread,
)
self._go(item, loop, "the bug grace period")
def _from_an_operator(self, project: Project, issue: Issue) -> bool:
"""L11: only an operator's issue, or one an operator assigned, is
an instruction."""
if issue.author in self._gitea_operators:
return True
try:
who = self._gitea.assigned_by(project.repo, issue.number, self._account)
except GiteaError:
return False
return who in self._gitea_operators
def _assigned_again(self, project: Project, issue: Issue, before: WorkItem) -> bool:
"""D1: an issue is taken once, and again only when it's assigned to
the agent anew after its last item closed (abandoned, say, then
given back): never on the same assignment twice."""
if before.status == "open" or not before.closed_at:
return False
try:
when = self._gitea.assigned_at(project.repo, issue.number, self._account)
except GiteaError:
return False
return bool(when) and _instant(when) > _instant(before.closed_at)
def _open(
self,
project: Project,
issue: Issue,
blocked_by: tuple[str, ...],
follows: WorkItem | None = None,
) -> None:
item_id = f"{project.name}-{issue.number}-{uuid.uuid4().hex[:6]}"
cause = f"assigned {project.repo}#{issue.number}"
ahead = len(self._store.work_items("open"))
where = f", {ahead} ahead" if ahead else ""
if blocked_by:
where += f"; blocked by {', '.join(blocked_by)}"
root = self._transport.post(
f"Queued {project.repo}#{issue.number}: {issue.title}{where}"
)
# D8: the project's label defaults steer as if the issue had them.
labels = tuple(dict.fromkeys((*issue.labels, *project.labels)))
loop = Loop(
thread=root,
issue=issue.number,
blocked_by=blocked_by,
labels=labels,
follows=follows.id if follows else "",
)
self._threads[(project.name, issue.number)] = root
self._store.open_work_item(
item_id,
title=issue.title,
scope=project.name,
position=BLOCKED if blocked_by else READY,
data={"loop": loop.to_data(), "roles": self._roles},
cause=cause,
)
log = self._store.transcript(item_id, "orchestrator")
log.append(
Fact(
text=f"work item opened from {project.repo}#{issue.number} "
f"({issue.title}), filed by {issue.author}"
+ (f", labels: {', '.join(labels)}" if labels else "")
+ (
f" ({', '.join(project.labels)} by the project's default)"
if project.labels
else ""
)
+ (
f"; assigned again after its earlier item {follows.id} closed "
f"({follows.outcome})"
if follows
else ""
)
)
)
text = f"{issue.title}\n\n{issue.body}".strip()
if issue.author in self._gitea_operators:
log.as_actor("operator").append(
UserMessage(text=text, source=f"gitea:issue-body:{issue.number}")
)
else:
# L11: someone else's text is data; the operator's assignment is
# the instruction.
log.append(Fact(text=f"the issue as written by {issue.author}:\n{text}"))
# -- messages (L5) -------------------------------------------------------
def on_message(self, msg: InboundMessage) -> None:
"""On the transport's thread: stamp, then hand to the worker."""
if msg.sender not in self._operators:
return
command = command_form(msg.text)
thread = msg.thread
named = _NAMED.fullmatch(command)
if named is not None:
command = named.group(1)
project, issue = named.group(2), int(named.group(3))
threads = [
t
for (p, n), t in list(self._threads.items())
if n == issue and project in (None, p)
]
# Ambiguous across projects: no stamp; handle asks which.
thread = threads[0] if len(threads) == 1 else thread
if (
command == "stop"
and self._running
# Unnamed at top level, it stops the active item (L5).
and (not thread or thread == self._active_thread)
):
# Straight to the flag: the run is what it has to reach (L2).
self._cancel.set()
about = self._shown.get(thread) if thread else None
if self._running and command not in ("stop", "status"):
# L5: say what happens to a message that can't be read yet.
here = thread and thread == self._active_thread
self._transport.post(
"Noted; I'll read it when this turn ends."
if here
else f"Noted; I'm in the middle of {self._active_name}, and I'll "
"read it when that turn ends.",
thread=thread or msg.event_id or None,
)
self._worker.submit(
self._settled("your message", lambda: self.handle(msg, about))
)
def handle(self, msg: InboundMessage, about: int | None) -> None:
item = item_for_thread(self._store, msg.thread) if msg.thread else None
text = (msg.original or msg.text).strip()
command = command_form(msg.text)
order = _NEXT.fullmatch(command)
if order is not None:
self._store.transcript(item.id if item else None, "operator").append(
UserMessage(text=text, source=f"transport:{msg.sender}")
)
self._transport.post(
self._put_first(order.group(1), int(order.group(2))),
thread=msg.thread,
)
self._schedule()
return
named = _NAMED.fullmatch(command)
if named is not None:
# L5: a command that names its item applies to it.
command = named.group(1)
project, issue = named.group(2), int(named.group(3))
found = self._named(project, issue)
if len(found) != 1:
self._store.transcript(None, "operator").append(
UserMessage(text=text, source=f"transport:{msg.sender}")
)
self._transport.post(
self._not_one(found, project, issue, f"{command} {{}}"),
thread=msg.thread,
)
return
[item] = found
if item is None and command not in ("status", "stop", *_NEEDS_ITEM):
# L5: any other message is a conversation, from its first message.
self._converse(msg, text)
self._schedule()
return
if item is None:
self._store.transcript(None, "operator").append(
UserMessage(text=text, source=f"transport:{msg.sender}")
)
if command == "status":
self._transport.post(self.status(), thread=msg.thread)
elif command in _NEEDS_ITEM:
self._transport.post(self._which(command), thread=msg.thread)
elif not self._store.active_item():
self._transport.post("Nothing is running.", thread=msg.thread)
return
if command in _NEAR_ABANDON:
# L5: a near-miss for a command is asked about, never read as a
# reply (the driver can't stop or close anything itself).
self._store.transcript(None, "operator").append(
UserMessage(text=text, source=f"transport:{msg.sender}")
)
self._transport.post(
f"Did you mean `abandon`? That closes {self._label(item)} and its "
"issue; say abandon to confirm. (`stop` ends a run in progress.)",
thread=loop_of(item).thread,
)
return
if command in ("status", "details"):
# About the item, not a reply to it: nothing changes, a
# countdown runs on (L5). Recorded at the agent's level.
self._store.transcript(None, "operator").append(
UserMessage(text=text, source=f"transport:{msg.sender}")
)
said = (
self._details(item)
if command == "details"
else self._status_line(item, [i.id for i in self._queue()])
)
self._transport.post(said, thread=loop_of(item).thread)
return
self._store.transcript(item.id, "operator").append(
UserMessage(text=text, source=f"transport:{msg.sender}")
)
# Any reply stops a grace period's countdown: the operator is here.
loop = replace(
loop_of(item),
asked_on_pr=0,
asked_where="",
asked_text="",
grace_until=0.0,
talked=True,
idle_since=self._clock(),
)
if loop != loop_of(item):
save(self._store, item.id, item.position, loop, cause="operator spoke")
item = self._store.work_item(item.id) or item
if command == "abandon":
self._close(item, loop, "abandoned", f"abandoned by {msg.sender}")
elif command == "done" and loop.new_project:
self._provisioned(item, loop)
elif (
item.position == WAITING
and item.scope != GENERAL
and is_question(text)
and loop.wait in (GO_AHEAD, ANSWER, MERGE)
):
# L5: questions never change state. Answered, then back to this
# very wait: a restatement up stays up, a go still binds to it.
save(
self._store,
item.id,
READY,
replace(
loop,
question=True,
back_to=loop.wait,
back_note=loop.note,
wait="",
note="",
),
cause="the operator asked a question",
)
elif command == "trust new host key":
project = self._at(item)
if loop.host_key_changed and project.workspace is not None:
project.workspace.host.forget_host_key()
self._fact(
item,
f"the operator ({msg.sender}) trusted the build VM's new "
"host key: it will be set up again",
)
save(
self._store,
item.id,
READY,
replace(loop, wait="", note="", host_key_changed=False),
cause=f"new host key trusted by {msg.sender}",
)
else:
self._transport.post(
f"Nothing on {self._label(item)} is waiting for that.",
thread=loop.thread,
)
elif command == "allow":
if item.position == WAITING and loop.pending_action:
action = json.loads(loop.pending_action)
self._fact(
item,
f"the operator ({msg.sender}) allowed this once: "
f"{action['tool']} {json.dumps(action['arguments'])}. Run it "
"again exactly as before.",
)
save(
self._store,
item.id,
READY,
replace(
loop,
wait="",
note="",
allowed=loop.pending_action,
pending_action="",
),
cause=f"allowed by {msg.sender}",
)
else:
self._transport.post(
f"Nothing on {self._label(item)} is waiting for `allow`.",
thread=loop.thread,
)
elif command == "stop":
# Stopping a run took effect on arrival; a stop is never an answer.
if not loop.note.startswith("stopped ("):
self._transport.post(
f"{self._label(item)} isn't running; nothing to stop.",
thread=loop.thread,
)
elif item.position == BLOCKED:
self._transport.post(
f"{self._label(item)} is blocked by {', '.join(loop.blocked_by)}; "
"I'll start on it when they close, or when you remove them.",
thread=loop.thread,
)
elif item.position == WAITING and loop.wait == GO_AHEAD and command in _GO:
if about != loop.rev:
# L3: a go-ahead is about what was on screen when it arrived.
self._transport.post(
f"that go was for rev-{about}, but rev-{loop.rev} is up now; "
"say go again if it still stands.",
thread=loop.thread,
)
return
self._go(item, loop, msg.sender)
elif (
item.position == WAITING
and loop.wait == GO_AHEAD
and about == loop.rev
and self._plain_go(item, text)
):
# L5, D1: a plain-language yes, said so before acting.
self._transport.post(
f"Taking that as a go-ahead on rev-{loop.rev}.", thread=loop.thread
)
self._go(item, loop, f"{msg.sender} (in plain language)")
elif item.position == WAITING:
# An answer, changes to a restatement, or a remark: the driver
# reads it, and may reply and go back to this wait (L5).
save(
self._store,
item.id,
READY,
replace(loop, wait="", note="", back_to=loop.wait, back_note=loop.note),
cause="the operator answered",
)
self._schedule()
def _converse(self, msg: InboundMessage, text: str) -> None:
"""A conversation (L1): its own item in the `general` scope, in a
thread under the operator's message, or in the thread of one that
closed, which it continues."""
thread = msg.thread or msg.event_id
before = (
item_for_thread(self._store, thread, status=None) if msg.thread else None
)
title = text.splitlines()[0][:60] if text else "conversation"
item_id = f"{GENERAL}-{uuid.uuid4().hex[:8]}"
loop = Loop(
thread=thread,
idle_since=self._clock(),
talked=True,
follows=before.id if before else "",
)
self._store.open_work_item(
item_id,
title=title,
scope=GENERAL,
position=READY,
data={"loop": loop.to_data(), "roles": self._roles},
cause="the operator started a conversation",
)
log = self._store.transcript(item_id, "orchestrator")
if before is not None:
closing = [
e.event.text
for e in self._store.transcript(before.id, "orchestrator").read()
if isinstance(e.event, Fact) and e.event.text.startswith("work item ")
]
log.append(
Fact(
text=f'this continues the conversation "{before.title}" '
f"({before.id}), "
+ (closing[-1] if closing else "which is still open")
)
)
log.as_actor("operator").append(
UserMessage(text=text, source=f"transport:{msg.sender}")
)
def _go(self, item: WorkItem, loop: Loop, by: str) -> None:
self._fact(item, f"go-ahead on rev-{loop.rev} from {by}")
if item.scope == GENERAL and loop.proposal_kind == "project":
self._checklist(item, loop)
return
if item.scope == GENERAL:
self._file_work(item, loop)
return
save(
self._store,
item.id,
READY,
replace(loop, started=True, wait="", grace_until=0.0),
cause=f"go-ahead on rev-{loop.rev}",
)
def _file_work(self, item: WorkItem, loop: Loop) -> None:
"""D1: agreed work from a conversation gets a record issue in its
project, carrying the new item's ID from the moment it exists, and
an item of its own that starts as agreed. Under an intent (L8): a
crash between the issue and the item is settled at boot."""
project = self._projects.get(loop.proposal_project)
summary = self._latest_proposal(item)
if project is None or not summary:
note = "that proposal can't be filed any more; what next?"
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
return
new_id = f"{project.name}-c{uuid.uuid4().hex[:6]}"
body = (
f"{summary}\n\nFrom a conversation with the operator, agreed there as "
f"rev-{loop.rev}.\n\n<!-- nisse-item: {new_id} -->"
)
intent = self._store.begin_intent(
FILE_ISSUE,
{"repo": project.repo, "item": new_id, "title": loop.proposal_title},
cause=f"filing {loop.proposal_title}",
work_item=item.id,
)
try:
number = self._gitea.create_issue(project.repo, loop.proposal_title, body)
self._gitea.assign_issue(project.repo, number, self._account)
except GiteaError as exc:
self._store.abort_intent(intent, cause=f"Gitea refused: {exc}")
note = f"I couldn't open the issue: {exc}. Say go to try again."
self._wait(item, replace(loop, wait=GO_AHEAD, note=note), note)
return
self._start_filed(item, project, new_id, number, loop.proposal_title, summary)
self._store.commit_intent(intent, cause=f"opened {project.repo}#{number}")
def _start_filed(
self,
conversation: WorkItem,
project: Project,
new_id: str,
number: int,
title: str,
summary: str,
) -> None:
"""The filed issue's item, agreed already, and the conversation told
where the work went."""
root = self._transport.post(
f"Queued {project.repo}#{number}: {title} (from a conversation)"
)
new = Loop(
thread=root,
issue=number,
labels=project.labels,
started=True, # agreed in the conversation
follows=conversation.id,
)
self._threads[(project.name, number)] = root
self._store.open_work_item(
new_id,
title=title,
scope=project.name,
position=READY,
data={"loop": new.to_data(), "roles": self._roles},
cause=f"filed {project.repo}#{number} from {conversation.id}",
)
self._store.transcript(new_id, "orchestrator").append(
Fact(
text=f"work item opened from {project.repo}#{number} ({title}), "
f"filed from the conversation {conversation.id}, where the operator "
f"agreed to this brief:\n{summary}"
)
)
loop = loop_of(self._store.work_item(conversation.id) or conversation)
loop = replace(
loop,
became=(*loop.became, f"{project.repo}#{number}"),
wait=CHAT,
note="",
proposal_project="",
proposal_title="",
idle_since=self._clock(),
)
self._fact(conversation, f"filed {project.repo}#{number} as {new_id}")
self._wait(
conversation,
loop,
f"Opened {project.repo}#{number} and started on it in its own thread. "
"We can keep talking here.",
)
# -- new projects (D1, D8) --------------------------------------------------
def _checklist(self, item: WorkItem, loop: Loop) -> None:
"""An agreed new project: what it needs, for the operator to provide
(later nisse-ops, O6), with its project entry to add."""
name = loop.proposal_project
entry = (
"projects:\n"
f" {name}:\n"
f" repo: <owner>/{name}\n"
" base_branch: main\n"
f" workspace: /root/work/{name}\n"
" build_host:\n"
" address: <the build VM>\n"
" key: <my SSH key for it>\n"
" push_key: <the deploy key's private half>"
)
self._fact(item, f"provisioning checklist posted for {name}")
self._wait(
item,
replace(
loop,
wait=PROVISION,
note=f"{name}'s setup: say done when it's ready",
new_project=name,
idle_since=self._clock(),
),
f"To set up `{name}`, I need:\n"
f"1. A repository for it, e.g. <owner>/{name}, with a README so its "
"default branch exists.\n"
"2. Protection on the default branch: none of my pushes, and an "
"approval to merge.\n"
f"3. Me ({self._account}) as a collaborator with write access.\n"
"4. A build VM, with the toolchain from the decisions above.\n"
"5. A deploy key with write access, its private half where I run.\n"
"6. This entry in my configuration, with the <...> parts filled "
f"in:\n\n```yaml\n{entry}\n```\n\n"
"Say done when it's ready, and I'll check each of them.",
)
def _provisioned(self, item: WorkItem, loop: Loop) -> None:
""" "done": the entry is read again and the new project checked
(D4: verified, not assumed). What fails is said, per item; when all
is well the tracking issue is opened, and the project's item starts
with its architecture."""
name = loop.new_project
try:
added = list(self._reload(set(self._projects))) if self._reload else []
except Exception as exc: # noqa: BLE001 -- said; the operator fixes it
self._transport.post(
f"I couldn't read the new entry: {exc}. Fix it and say done again.",
thread=loop.thread,
)
return
project = next((p for p in added if p.name == name), None)
if project is None:
self._transport.post(
f"I don't see `{name}` in my configuration yet (item 6). Add its "
"entry and say done again.",
thread=loop.thread,
)
return
failures = (
project.milestones.verify_access(self._account)
if project.milestones is not None
else ["the project has no workspace"]
)
if failures:
self._fact(item, f"{name} checked: " + "; ".join(failures))
self._transport.post(
"Not ready yet: " + "; ".join(failures) + ". Say done again when "
"they're fixed.",
thread=loop.thread,
)
return
self._projects[name] = project
self._fact(item, f"{name} set up and checked: {project.repo}")
brief = self._latest_proposal(item)
new_id = f"{name}-c{uuid.uuid4().hex[:6]}"
title = f"Implement the build plan for {name}"
try:
number = self._gitea.create_issue(
project.repo,
title,
f"{brief}\n\nFrom a conversation with the operator, agreed there "
f"as rev-{loop.rev}. First: ARCHITECTURE.md and the build plan, "
"in a PR of documents only; code after it has merged."
f"\n\n<!-- nisse-item: {new_id} -->",
)
self._gitea.assign_issue(project.repo, number, self._account)
except GiteaError as exc:
self._transport.post(
f"{name} checks out, but I couldn't open its tracking issue: "
f"{exc}. Say done to try again.",
thread=loop.thread,
)
return
# D1: the architecture is agreed in its PR, as `design` steers.
design = replace(
project, labels=tuple(dict.fromkeys(("design", *project.labels)))
)
self._start_filed(item, design, new_id, number, title, brief)
loop = loop_of(self._store.work_item(item.id) or item)
save(
self._store,
item.id,
WAITING,
replace(loop, new_project="", proposal_kind=""),
cause=f"{name} provisioned",
)
def _latest_proposal(self, item: WorkItem) -> str:
proposals = [
e.event.plan
for e in self._store.transcript(item.id, "driver").read()
if isinstance(e.event, PlanSubmitted)
]
return proposals[-1] if proposals else ""
def _plain_go(self, item: WorkItem, text: str) -> bool:
"""Whether a reply that isn't a command agrees to the restatement up.
Questions never do (L5)."""
if self._reader is None or is_question(text):
return False
proposals = [
e.event.plan
for e in self._store.transcript(item.id, "driver").read()
if isinstance(e.event, PlanSubmitted)
]
if not proposals:
return False
classifier = self._roles_of(item).resolve("classifier")
agreed = self._reader.agrees(proposals[-1], text, model=classifier)
# L9 evaluation capture: the reading, as an input/output pair.
log = self._store.transcript(item.id, "classifier")
log.append(
DelegationStarted(
brief=f"proposal:\n{proposals[-1]}\n\nreply:\n{text}",
role="classifier",
model=classifier,
caps={},
)
)
log.append(
DelegationCompleted(
handback="a go-ahead" if agreed else "not a go-ahead",
artifacts=[],
tokens=0,
duration_s=0.0,
exit_status="answered",
)
)
return agreed
def _which(self, command: str) -> str:
"""L5: an unnamed command at top level is ambiguous: ask."""
items = [i for i in self._store.work_items("open") if i.scope != GENERAL]
if not items:
return f"{command} what? I have no open work items."
listed = ", ".join(f"{self._label(i)} ({i.title})" for i in items)
return (
f"Which item? Say it in the item's thread, or name it "
f'("{command} {self._label(items[0])}"). Open: {listed}.'
)
def _not_one(
self, found: list[WorkItem], project: str | None, issue: int, say: str
) -> str:
"""A named item that is none, or more than one (the same number in
two projects). ``say`` is the command, with {} for the item."""
name = f"{project}#{issue}" if project else f"#{issue}"
if not found:
return f"{name} isn't one of my open work items."
options = " or ".join(say.format(self._label(i)) for i in found)
return f"{name} is open in more than one project: say {options}."
def _put_first(self, project: str | None, issue: int) -> str:
"""The operator's "do #N next" (L1): it runs before every other
ready item, the latest such order first. It doesn't unblock."""
found = self._named(project, issue)
if len(found) != 1:
return self._not_one(found, project, issue, "do {} next")
[item] = found
name = self._label(item)
items = self._store.work_items("open")
first = max(loop_of(i).first for i in items) + 1
loop = replace(loop_of(item), first=first)
save(self._store, item.id, item.position, loop, cause=f"{name} put first")
self._fact(item, "the operator put this item first in the queue")
if item.position == BLOCKED:
return (
f"{name} goes first once it's unblocked "
f"(blocked by {', '.join(loop.blocked_by)})."
)
if item.position == WAITING:
return f"{name} goes first, but it's waiting for {loop.wait}: {loop.note}"
return f"{name} is next."
def _queue(self) -> list[WorkItem]:
"""Ready items in the order they run (L1): the operator's order
first, then the oldest."""
items = self._store.work_items("open") # oldest first
ready = [i for i in items if i.position == READY]
return sorted(ready, key=lambda i: -loop_of(i).first)
def status(self) -> str:
items = self._store.work_items("open")
if not items:
return (
"idle: no open work items. I take issues assigned to me in "
f"{self._repos()}."
)
queue = [i.id for i in self._queue()]
return "\n".join(self._status_line(item, queue) for item in items)
def _status_line(self, item: WorkItem, queue: list[str]) -> str:
loop = loop_of(item)
what = item.position
if item.position == WAITING and loop.wait == CHAT:
what = "talking; your turn"
elif item.position == WAITING:
what = f"waiting for {loop.wait} ({loop.note or f'rev-{loop.rev}'})"
if loop.grace_until:
left = max(0, int(loop.grace_until - self._clock()) // 60)
what += f", going ahead in {left} min unless you reply"
elif item.position == BLOCKED:
what = f"blocked by {', '.join(loop.blocked_by)}"
elif item.position == READY:
what = f"ready, {queue.index(item.id) + 1} of {len(queue)} in line"
name = (
f'conversation "{item.title}"'
if item.scope == GENERAL
else f"{self._label(item)} {item.title}"
)
return f"{name}: {what}"
# -- running (L1, L2) ------------------------------------------------------
def _schedule(self) -> None:
"""Run ready items until none is left. Only on the worker."""
if self._running:
return
while True:
ready = self._queue()
if not ready:
self._store.set_active(None, cause="nothing ready")
return
self._run(ready[0])
def _keep(self, project: str) -> tuple[str, ...]:
"""Branch prefixes of a project's open items: parked or not, they
are never set aside."""
return tuple(
f"{i.id}-" for i in self._store.work_items("open") if i.scope == project
)
def _park_elsewhere(self, here: str) -> None:
"""L1: switching to another project's item parks the one on the
project worked on last, so nothing lives only on its build VM. After
a boot nobody knows which that was, so every other is parked."""
last = self._last_project
others = [p for p in self._projects.values() if p.name != here]
for other in others if last is None else [self._projects[last]]:
if other.name == here or other.milestones is None:
continue
try:
other.milestones.park(keep=self._keep(other.name))
except Exception as exc: # noqa: BLE001 -- reported; the run goes on
self._oplog.emit(
"loop.park_failed", cause=repr(exc), project=other.name
)
self._last_project = here
# -- help (L9) -------------------------------------------------------------
def _roles_of(self, item: WorkItem) -> RoleMap:
"""The roles pinned when the item opened, so a configuration change
never splits one item across models (L9)."""
default = (
self._helpers.roles
if self._helpers
else RoleMap(driver=self._roles.get("driver", ""), helper="", expert="")
)
pinned = item.data.get("roles") or self._roles
return RoleMap.from_pinned(dict(pinned), default)
def _delegate(
self, item_id: str, task: str, *, trigger: str = ""
) -> tuple[str, bool]:
"""A helper takes a sub-task on a branch off the milestone's: merged
back on a handback, discarded otherwise."""
item = self._store.work_item(item_id)
assert item is not None
loop = loop_of(item)
milestones = self._at(item).milestones
workspace = self._at(item).workspace
if self._helpers is None or milestones is None or workspace is None:
return "there's no helper here", False
if not loop.branch:
return "start a milestone first: a helper works on its branch", False
if not task:
return "say what the sub-task is", False
roles = self._roles_of(item)
number = loop.helps + 1
branch = f"{loop.branch}-help-{number}"
save(
self._store,
item.id,
item.position,
replace(loop, helps=number),
cause=f"helper {number} called in",
)
helper = roles.resolve("helper")
delegation = Delegation(
self._helpers.delegate_client(helper),
self._store,
workspace,
milestones.git,
model=helper,
backing_model=(
self._helpers.backing(helper) if self._helpers.backing else ""
),
caps=self._helpers.caps,
cancel=self._cancel,
reader=self._helpers.reader(roles),
on_turn=lambda said: self._progress(
f"{item.id}:helper", loop.thread, said, prefix="helper: "
),
screen=lambda call: self._screen_helper(item.id, str(workspace.root), call),
)
git = milestones.git
before = git.head_sha() # where the milestone is, uncommitted work aside
brief = (
f'{task}\n\nThe work item: issue #{loop.issue}, "{item.title}", '
f"milestone {loop.milestone}." + (f" {trigger}" if trigger else "")
)
result = delegation.run(
self._store.transcript(item.id, "helper"), brief=brief, branch=branch
)
try:
# The milestone takes the helper's work (or, on failure, only
# its own back) as uncommitted changes: harness branches never
# hold commits only this clone has (publishing makes them), so
# parking, a later milestone and a follow-up PR all carry it.
git.reset_to(loop.branch, branch if result.usable else loop.branch)
git._git("reset", "-q", "--soft", before)
if branch in git.local_branches():
git._git("branch", "-q", "-D", branch)
except GitError as exc:
self._fact(item, f"after helper {number}: {exc}")
for intent in self._store.pending_intents():
# Cut off rather than finished: discarded all the same, never
# resumed blindly (L8, L9).
if (
intent.kind == DELEGATION_INTENT
and intent.payload.get("branch") == branch
):
self._store.abort_intent(intent.id, cause=f"delegation {result.status}")
if result.usable and result.handback is not None:
self._fact(item, f"helper {number} finished: {result.handback.render()}")
return f"the helper finished:\n{result.handback.render()}", True
self._fact(
item, f"helper {number} didn't finish ({result.status}): {result.note}"
)
said = (
f"the helper didn't finish ({result.status}: {result.note}); its work "
"was discarded and the milestone is as it was"
)
return said, False
# -- the fast path (P4, D6) -------------------------------------------------
def _screen(self, item_id: str, root: str, call: ToolCall) -> Screening:
"""The driver's execution call, judged before it runs (P4): by the
fast path's rules, then the classifier for what they don't know
(L4). A dangerous one runs only when the operator allowed exactly
this one, or the classifier found it plainly in scope."""
item = self._store.work_item(item_id)
assert item is not None
loop = loop_of(item)
action = Action(call.tool, dict(call.arguments))
key = action.key()
verdict, judged = self._judge(item, "driver", call, action, root)
if verdict.effect == REFUSED:
decision, screening = "refused", Screening("refuse", verdict.reason)
elif judged is not None and verdict.rule == IN_SCOPE:
decision, screening = "ran", Screening("run")
self._in_scope(item, action, verdict.reason)
elif verdict.effect == DANGEROUS and loop.allowed == key:
decision, screening = "allowed", Screening("run")
# Once: a second identical call is asked about again (L3).
save(
self._store,
item_id,
item.position,
replace(loop, allowed=""),
cause="allowed action ran",
)
elif verdict.effect == DANGEROUS:
decision = "asked"
why = (
verdict.reason
if judged is None
else f"the classifier says: {verdict.reason}"
)
screening = Screening("ask", self._ask_text(item, action, why), key)
else:
decision, screening = "ran", Screening("run")
if screening.decision == "run" and commits(action):
# #129: a local commit looked like publishing to the model.
screening = replace(screening, note=_COMMITTED)
self._verdict(item_id, "driver", call, verdict, decision, judged)
return screening
def _screen_helper(self, item_id: str, root: str, call: ToolCall) -> str | None:
"""A helper's execution call: a helper can't ask the operator, so a
dangerous one is refused with what to do instead."""
item = self._store.work_item(item_id)
assert item is not None
action = Action(call.tool, dict(call.arguments))
verdict, judged = self._judge(item, "helper", call, action, root)
if verdict.effect == REFUSED:
self._verdict(item_id, "helper", call, verdict, "refused", judged)
return verdict.reason
if judged is not None and verdict.rule == IN_SCOPE:
self._verdict(item_id, "helper", call, verdict, "ran", judged)
self._in_scope(item, action, verdict.reason)
return None
if verdict.effect == DANGEROUS:
self._verdict(item_id, "helper", call, verdict, "refused", judged)
return (
f"{verdict.reason}, which needs the operator's agreement; only the "
"driver can ask for it, so hand back and say what you need"
)
self._verdict(item_id, "helper", call, verdict, "ran", judged)
return None
def _judge(
self, item: WorkItem, actor: str, call: ToolCall, action: Action, root: str
) -> tuple[Verdict, tuple[str, str] | None]:
"""The fast path's verdict, or for what it doesn't know, the
classifier's, with (model, what it was shown)."""
verdict = classify(action, root)
if verdict.effect != UNKNOWN or self._classifier is None:
return verdict, None
self._verdict(item.id, actor, call, verdict, "classified")
loop = loop_of(item)
cached = (item.id, loop.rev, action.key())
if cached not in self._judged:
scope = Scope(
project=item.scope or "",
repo=self._at(item).repo,
issue=loop.issue,
agreed=self._agreed(item),
)
model = self._roles_of(item).resolve("classifier")
judged, by = self._classifier.judge(action, scope, root, model=model)
self._judged[cached] = (
judged,
(by, self._classifier.input_for(action, scope, root)),
)
judged, seen = self._judged[cached]
return judged, seen
def _agreed(self, item: WorkItem) -> str:
"""The restatement the operator said go to: the latest one put up."""
events = self._store.transcript(item.id, "driver").read()
plans = [e.event.plan for e in events if isinstance(e.event, PlanSubmitted)]
return plans[-1] if plans else item.title
def _in_scope(self, item: WorkItem, action: Action, reason: str) -> None:
"""P3: dangerous but in scope runs, and is reported."""
what = (
action.arguments.get("command") or f"{action.tool} {dict(action.arguments)}"
)
self._transport.post(
f"Ran on {self._label(item)}, judged in scope: `{what}` ({reason})",
thread=loop_of(item).thread,
)
def _verdict(
self,
item_id: str,
actor: str,
call: ToolCall,
verdict: Verdict,
decision: str,
judged: tuple[str, str] | None = None,
) -> None:
# L4, L7: every verdict logged, for the operator and the evaluation set.
model, shown = judged or ("", "")
source = "classifier" if judged is not None else "fast_path"
self._store.transcript(item_id, actor).append(
ActionVerdict(
call_id=call.call_id,
tool=call.tool,
effect=verdict.effect,
rule=verdict.rule,
reason=verdict.reason,
decision=decision,
source=source,
model=model,
input=shown,
)
)
self._oplog.emit(
"action.verdict",
cause=f"{actor} called {call.tool}",
work_item=item_id,
source=source,
model=model,
effect=verdict.effect,
rule=verdict.rule,
decision=decision,
)
def _ask_text(self, item: WorkItem, action: Action, reason: str) -> str:
"""Rendered from the action itself, never from a model's account of
it (L3, L11)."""
what = (
f"run `{action.arguments.get('command', '')}`"
if action.tool == "run_command"
else f"call {action.tool} {dict(action.arguments)}"
)
return (
f"{self._label(item)} wants to {what}: {reason}. Say `allow` to let "
"exactly this run once, or tell me what to do instead."
)
def _ask_expert(self, item_id: str, question: str) -> tuple[str, bool]:
item = self._store.work_item(item_id)
assert item is not None
if self._helpers is None:
return "there's no expert here", False
if not question:
return "ask a question", False
desk = ExpertDesk(
self._helpers.client,
self._roles_of(item),
budget=self._helpers.expert_budget,
)
try:
answer = desk.ask(self._store.transcript(item.id, "expert"), question)
except BudgetExhausted as exc:
return str(exc), False
except LLMError as exc:
return f"the expert couldn't answer: {exc}", False
return answer.render(), True
def _help_when_stuck(self, item: WorkItem, loop: Loop, outcome: RunOutcome) -> bool:
"""L9's observable triggers: the driver stuck (commands failing in a
row) or out of turns mid-milestone. A helper is called in, once per
milestone; True when it finished and the driver goes on."""
if (
self._helpers is None
or outcome.kind not in ("stuck", "capped")
or not loop.branch
or loop.helped_at == loop.milestone
):
return False
save(
self._store,
item.id,
RUNNING,
replace(loop, helped_at=loop.milestone),
cause="helper called in",
)
self._transport.post(
f"I got stuck ({outcome.note[:200]}), so a helper is taking over the "
f"rest of milestone {loop.milestone}.",
thread=loop.thread,
)
self._running = True
try:
_, ok = self._delegate(
item.id,
f"Finish milestone {loop.milestone}, so its tests pass.",
trigger=f"The driver got stuck: {outcome.note}",
)
finally:
self._running = False
self._cancel.clear()
if not ok:
return False
item = self._store.work_item(item.id) or item
save(self._store, item.id, READY, loop_of(item), cause="a helper finished")
return True
def _changed_problem(self, item_id: str, args: dict[str, object]) -> str | None:
"""L5: a revision says what changed from the one before."""
item = self._store.work_item(item_id)
if item is None or loop_of(item).rev == 0:
return None
if str(args.get("changed", "")).strip():
return None
return (
f"not posted: this is a revision of rev-{loop_of(item).rev}, so say "
"in changed what's different and why"
)
def _progress(self, item_id: str, thread: str, said: str, prefix: str = "") -> None:
"""A turn's own words, posted at a readable rate (L5, #42): at most
one a minute per item, the latest when the minute is up."""
now = self._clock()
if now - self._progressed.get(item_id, float("-inf")) < PROGRESS_EVERY_S:
return
self._progressed[item_id] = now
text = said if len(said) <= PROGRESS_CHARS else said[:PROGRESS_CHARS] + " …"
self._transport.post(f"{prefix}{text}", thread=thread)
def _details(self, item: WorkItem) -> str:
"""L5: raw tool activity, on request: the item's last tool calls."""
events = self._store.transcript(item.id, "driver").read()
results = {
e.event.call_id: e.event for e in events if isinstance(e.event, ToolResult)
}
lines = []
for e in events:
if not isinstance(e.event, ToolCall):
continue
call, result = e.event, results.get(e.event.call_id)
target = call.arguments.get("path") or call.arguments.get("command") or ""
outcome = (
"no result yet"
if result is None
else ("ok" if result.success else "failed")
+ (f": {result.result.strip()[:120]}" if result.result.strip() else "")
)
who = "" if e.actor == "driver" else f"{e.actor}: "
lines.append(f"- {who}{call.tool}({str(target)[:80]}) → {outcome}")
if not lines:
return f"No tool calls yet on {self._label(item)}."
return (
f"The last tool calls on {self._label(item)}:\n"
+ "\n".join(lines[-DETAILS:])
+ self._verdict_summary(events)
)
@staticmethod
def _verdict_summary(events: Sequence[StoredEvent]) -> str:
"""The fast path's verdicts on the item (P4): counts, and every
action that wasn't simply run."""
verdicts = [e.event for e in events if isinstance(e.event, ActionVerdict)]
if not verdicts:
return ""
counts: dict[str, int] = {}
for v in verdicts:
counts[v.effect] = counts.get(v.effect, 0) + 1
order = ("harmless", "contained", "unknown", "dangerous", "refused")
said = ", ".join(f"{counts[e]} {e}" for e in order if e in counts)
notable = [
f"- {v.decision}{' by the classifier' if v.source == 'classifier' else ''}"
f": {v.tool} ({v.rule}: {v.reason})"
for v in verdicts
if v.decision not in ("ran", "classified")
or (v.source == "classifier" and v.rule == IN_SCOPE)
]
return f"\n\nVerdicts: {said}." + (
"\n" + "\n".join(notable[-DETAILS:]) if notable else ""
)
def _design_problem(self, item_id: str) -> str | None:
"""D1, D2: a `design` item agrees its design before any code. Until
a PR of it has merged, a milestone holds documents only."""
item = self._store.work_item(item_id)
if item is None:
return None
loop = loop_of(item)
milestones = self._at(item).milestones
if "design" not in loop.labels or loop.design_merged or milestones is None:
return None
try:
changed = milestones.changed()
except GitError:
return None # publishing reports what's wrong with the workspace
code = [
p for p in changed if p != "ARCHITECTURE.md" and not p.startswith("docs/")
]
if not code:
return None
return (
"not published: this milestone is the design, so it holds documents "
"only (ARCHITECTURE.md, docs/); code starts in a later milestone, "
"after this one's PR has merged. Take these out of it (they can "
"wait in the plan): " + ", ".join(code)
)
def _converse_run(self, item: WorkItem) -> None:
"""A conversation's turn (L1, L5): it talks, and may propose work on
a project. No workspace, nothing to run."""
loop = loop_of(item)
self._store.set_active(item.id, cause="ready")
save(self._store, item.id, RUNNING, loop, cause="driver run")
projects = "\n".join(f"- {p.name}: {p.repo}" for p in self._projects.values())
system = CONVERSATION.format(projects=f"The projects:\n{projects}")
self._active_thread = loop.thread
self._running = True
try:
outcome = self._driver.run(
self._store.transcript(item.id, "driver"),
system=system,
tools=[REPLY, ASK_HUMAN, PROPOSE_WORK, PROPOSE_PROJECT],
workspace=None,
checks={
"propose_work": self._proposal_problem,
"propose_project": self._project_problem,
},
)
finally:
self._running = False
self._cancel.clear()
self._settle(item.id, outcome)
def _project_problem(self, args: dict[str, object]) -> str | None:
name = str(args.get("name", "")).strip()
if not re.fullmatch(r"[a-z0-9][a-z0-9-]*", name) or name == GENERAL:
return "a project's name is lowercase letters, digits and -, not general"
if name in self._projects:
return f"{name} is a project already: propose_work for work on it"
if not str(args.get("purpose", "")).strip():
return "say what the project is for"
return _restatement_problem(
{"text": f"{args.get('purpose', '')}\n{args.get('decisions', '')}"}
)
def _proposal_problem(self, args: dict[str, object]) -> str | None:
project = str(args.get("project", "")).strip()
if project not in self._projects:
names = ", ".join(self._projects)
return f"no project {project!r}: the projects are {names}"
if not str(args.get("title", "")).strip():
return "the issue needs a title"
summary = str(args.get("summary", "")).strip()
if not summary:
return "the issue needs a summary: what was decided"
return _restatement_problem({"text": summary})
def _run(self, item: WorkItem) -> None:
if item.scope == GENERAL:
self._converse_run(item)
return
loop = loop_of(item)
project = self._at(item)
milestones = project.milestones
self._store.set_active(item.id, cause="ready")
self._park_elsewhere(project.name)
checked = self._preflight and project.workspace is not None
marker: dict[str, Any] | None = None
starting = not loop.prepared
if checked and milestones is not None:
ready = self._readiness(item, project, loop)
if ready is None:
return # waiting on the operator
loop, marker = ready
starting = not loop.prepared
if milestones is not None:
keep = self._keep(project.name)
try:
if not loop.prepared:
# D3: a fresh workspace when a work item starts.
notes = milestones.prepare(f"{item.id}-work", keep=keep)
else:
# D4: back to its branch, parking whoever was there.
notes = []
milestones.resume(
loop.branch or f"{item.id}-work", keep=keep, mine=f"{item.id}-"
)
except Exception as exc: # noqa: BLE001 -- the item waits, not the loop
note = f"I couldn't prepare the workspace: {exc}. Reply to retry."
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
self._store.set_active(None, cause="workspace failed")
return
if not loop.prepared:
self._fact(item, "workspace prepared from the latest default branch")
if notes:
said = "Set aside from the workspace: " + "; ".join(notes)
self._fact(item, said)
self._transport.post(said, thread=loop.thread)
loop = replace(loop, prepared=True)
save(self._store, item.id, RUNNING, loop, cause="driver run")
if checked and milestones is not None:
toolchain = self._toolchain(item, project, loop, marker, starting)
if toolchain is None:
return # preflight was stopped: settled
loop = toolchain
save(self._store, item.id, RUNNING, loop, cause="toolchain checked")
# D1: a `discuss` item talks before it restates.
hold = (
"discuss" in loop.labels and not loop.talked
) or loop.started # agreed: a restatement is behind it (D1)
if loop.question:
hold = True # nothing but answering (below)
tools = [
*([] if hold else [RESTATE]),
ASK_HUMAN,
*(
_LOOK
if not loop.started
else [
START_MILESTONE,
FINISH_MILESTONE,
CLOSE_ITEM,
*([WAIT_FOR_REVIEW] if loop.pr else []),
*([DELEGATE, ASK_EXPERT] if self._helpers else []),
*([REPLY] if loop.back_to else []),
*EXECUTION_TOOLS,
]
),
]
if loop.question:
tools = [REPLY, *_LOOK]
elif loop.back_to and not loop.started:
tools = [*tools, REPLY] # before the go-ahead, a remark answered
roles = self._roles_of(item)
workspace = project.workspace
system = SYSTEM.format(
repo=project.repo,
issue=loop.issue,
title=item.title,
labels="".join(f"{LABELS[x]}\n\n" for x in loop.labels if x in LABELS),
helpers=(
f"{self._helpers.briefing(roles)}\n\n"
if self._helpers and loop.started
else ""
),
workspace=workspace.describe() if workspace else "",
) + (QUESTION if loop.question else "")
self._active_thread = loop.thread
self._active_name = self._label(item)
self._running = True
try:
outcome = self._driver.run(
self._store.transcript(item.id, "driver"),
system=system,
tools=tools,
workspace=workspace,
handlers={
"start_milestone": lambda args: self._start(item.id, args),
"delegate": lambda args: self._delegate(
item.id, str(args.get("task", "")).strip()
),
"ask_expert": lambda args: self._ask_expert(
item.id, str(args.get("question", "")).strip()
),
},
checks={
"restate": lambda args: (
_restatement_problem(args)
or self._changed_problem(item.id, args)
),
"finish_milestone": lambda args: self._design_problem(item.id),
},
model=roles.resolve("driver"),
on_turn=lambda said: self._progress(item.id, loop.thread, said),
screen=(
(lambda call: self._screen(item.id, str(workspace.root), call))
if workspace
else None
),
)
finally:
self._running = False
self._cancel.clear()
self._settle(item.id, outcome)
def _settle(self, item_id: str, outcome: RunOutcome) -> None:
item = self._store.work_item(item_id)
assert item is not None
loop = loop_of(item)
args = outcome.arguments or {}
# An allow covers one run; a pending ask survives only a run that
# goes back to waiting on it (a question, or a reply to a remark).
keeps = loop.question or (
outcome.kind == "wait" and outcome.tool == "reply" and bool(loop.back_to)
)
loop = replace(
loop, allowed="", pending_action=loop.pending_action if keeps else ""
)
if outcome.kind == "wait" and outcome.tool == ASK_ALLOW:
question = str(args.get("question", ""))
self._fact(item, f"asked the operator: {question}")
self._transport.post(question, thread=loop.thread)
self._wait(
item,
replace(
loop,
wait=ANSWER,
note=question,
pending_action=str(args.get("key", "")),
),
None,
)
return
if loop.question:
# Back to the wait the question interrupted, as it was (L5).
if outcome.kind == "wait" and outcome.tool == "reply":
self._transport.post(
str(args.get("text", "")).strip(), thread=loop.thread
)
else:
self._transport.post(
f"I couldn't answer that ({outcome.note or outcome.kind}).",
thread=loop.thread,
)
self._wait(
item,
replace(
loop,
wait=loop.back_to,
note=loop.back_note,
question=False,
back_to="",
back_note="",
),
None,
)
self._store.set_active(None, cause="question answered")
return
if outcome.kind == "wait" and outcome.tool == "restate":
text = str(args.get("text", "")).strip()
rev = loop.rev + 1
self._store.transcript(item.id, "driver").append(
PlanSubmitted(plan=text, model=self._roles.get("driver", ""))
)
self._fact(item, f"restatement rev-{rev} put up for a go-ahead")
self._shown[loop.thread] = rev
grace = self._clock() + self._grace if "bug" in loop.labels else 0.0
after = "Say go to start, or reply with changes."
if grace:
after = (
f"It's a bug, so I'll go ahead in {round(self._grace / 60)} min "
"unless you reply; say go to start now, or reply with changes."
)
changed = str(args.get("changed", "")).strip()
since = f"\n\nChanged from rev-{rev - 1}: {changed}" if rev > 1 else ""
self._transport.post(
f"rev-{rev}: {text}{since}\n\n{after}", thread=loop.thread
)
self._wait(
item,
replace(loop, rev=rev, wait=GO_AHEAD, note="", grace_until=grace),
None,
)
elif outcome.kind == "wait" and outcome.tool == "reply" and loop.back_to:
# A project item's reply: answered where asked (the thread or the
# PR), then back to the very wait the message interrupted (L5).
self._answer_where_asked(item, loop, str(args.get("text", "")).strip())
self._wait(
item, replace(loop, wait=loop.back_to, note=loop.back_note), None
)
elif outcome.kind == "wait" and outcome.tool == "reply":
text = str(args.get("text", "")).strip()
self._transport.post(text, thread=loop.thread)
self._wait(
item,
replace(loop, wait=CHAT, note="", idle_since=self._clock()),
None,
)
elif outcome.kind == "wait" and outcome.tool == "propose_work":
loop = replace(loop, proposal_kind="work")
project = self._projects[str(args.get("project", "")).strip()]
title = str(args.get("title", "")).strip()
summary = str(args.get("summary", "")).strip()
rev = loop.rev + 1
self._store.transcript(item.id, "driver").append(
PlanSubmitted(plan=summary, model=self._roles.get("driver", ""))
)
self._fact(item, f"work proposed as rev-{rev}: {project.repo}, {title}")
self._shown[loop.thread] = rev
self._transport.post(
f"rev-{rev}: I'd open an issue in {project.repo}, "
f'"{title}":\n\n{summary}\n\nSay go to open it and start, or '
"reply with changes.",
thread=loop.thread,
)
self._wait(
item,
replace(
loop,
rev=rev,
wait=GO_AHEAD,
note="",
proposal_project=project.name,
proposal_title=title,
idle_since=self._clock(),
),
None,
)
elif outcome.kind == "wait" and outcome.tool == "propose_project":
name = str(args.get("name", "")).strip()
purpose = str(args.get("purpose", "")).strip()
decisions = str(args.get("decisions", "")).strip()
brief = f"{purpose}\n\nDecisions:\n{decisions}" if decisions else purpose
rev = loop.rev + 1
self._store.transcript(item.id, "driver").append(
PlanSubmitted(plan=brief, model=self._roles.get("driver", ""))
)
self._fact(item, f"new project proposed as rev-{rev}: {name}")
self._shown[loop.thread] = rev
self._transport.post(
f"rev-{rev}: a new project, `{name}`. {brief}\n\nSay go and I'll "
"list what it needs set up, or reply with changes.",
thread=loop.thread,
)
self._wait(
item,
replace(
loop,
rev=rev,
wait=GO_AHEAD,
note="",
proposal_project=name,
proposal_title="",
proposal_kind="project",
idle_since=self._clock(),
),
None,
)
elif outcome.kind == "wait" and outcome.tool == "ask_human":
question = str(args.get("question", "")).strip()
self._answer_where_asked(item, loop, question)
self._wait(item, replace(loop, wait=ANSWER, note=question), None)
elif outcome.kind == "wait" and outcome.tool == "wait_for_review":
note = str(args.get("note", "")).strip() or f"waiting on PR #{loop.pr}"
if loop.pr:
self._answer_where_asked(item, loop, note)
self._wait(item, replace(loop, wait=MERGE, note=f"PR #{loop.pr}"), None)
else:
say = f"{note} (there's no open PR, so I need you to tell me what next)"
self._wait(item, replace(loop, wait=ANSWER, note=say), say)
elif outcome.kind == "wait" and outcome.tool == "finish_milestone":
self._finish(
item,
loop,
str(args.get("summary", "")).strip(),
str(args.get("title", "")).strip(),
# Unsaid, it's final: the PR closes the issue, as before.
str(args.get("final", "yes")).strip().lower() != "no",
)
elif outcome.kind == "wait" and outcome.tool == "close_item" and loop.pr:
# Closing leaves nothing open (L2): an open PR is waited out first.
self._fact(
item,
f"close_item waits: PR #{loop.pr} is still open; the item closes "
"after it merges",
)
self._wait(
item,
replace(loop, wait=MERGE, note=f"PR #{loop.pr}"),
f"I'll close this once PR #{loop.pr} merges: {loop.pr_url}",
)
elif outcome.kind == "wait" and outcome.tool == "close_item":
summary = str(args.get("summary", "")).strip()
self._transport.post(f"Done: {summary}", thread=loop.thread)
self._close(item, loop, "completed", summary)
elif outcome.kind == "stopped":
note = f"stopped ({outcome.note}). Why did you stop it? Reply to continue."
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
elif self._help_when_stuck(item, loop, outcome):
pass # ready again: the driver goes on from the helper's handback
else:
note = f"I stopped short: {outcome.note}. Reply to continue."
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
self._store.set_active(None, cause="item waiting")
# -- milestones (D4) -----------------------------------------------------
def _start(self, item_id: str, args: dict[str, object]) -> tuple[str, bool]:
"""start_milestone, mid-run: cut the milestone's branch."""
item = self._store.work_item(item_id)
assert item is not None
loop = loop_of(item)
project = self._at(item)
if loop.pr:
# D4: the next milestone waits for the merge, read from Gitea now,
# not from the last watch round.
try:
pr = self._gitea.pull_request(project.repo, loop.pr)
except GiteaError as exc:
return f"couldn't check PR #{loop.pr}: {exc}; try again", False
if pr.state == "open":
wait = (
f"milestone {loop.milestone}'s PR #{loop.pr} hasn't merged yet; "
"the next milestone starts after it merges. If you're only "
"waiting for that, call wait_for_review."
)
return wait, False
how = "merged" if pr.merged else "was closed without merging"
self._fact(item, f"PR #{loop.pr} {how}")
loop = replace(loop, pr=0, pr_url="", branch="")
if loop.branch:
busy = f"milestone {loop.milestone} is still open; finish it first"
return busy, False
if project.milestones is None:
return "milestones aren't available here", False
number = loop.milestone + 1
branch = f"{item.id}-milestone-{number}-{uuid.uuid4().hex[:4]}"
try:
project.milestones.start(branch)
except GitError as exc:
return f"couldn't start the milestone: {exc}", False
name = str(args.get("name", "")).strip()
self._fact(item, f"milestone {number} started ({name}) on branch {branch}")
save(
self._store,
item.id,
RUNNING,
replace(loop, milestone=number, branch=branch),
cause=f"milestone {number} started",
)
return f"started milestone {number} on branch {branch}", True
def _answer_where_asked(self, item: WorkItem, loop: Loop, text: str) -> None:
"""D5: answer where the operator asked. A PR question is answered on
the PR, quoting the line it was about (an inline thread can't be
replied to through the API), with a pointer in the Matrix thread."""
if not loop.asked_on_pr:
self._transport.post(text, thread=loop.thread)
return
quoted = [f"> on `{loop.asked_where}`:"] if loop.asked_where else []
quoted += [f"> {line}" for line in loop.asked_text.splitlines()]
body = "\n".join([*quoted, "", text])
try:
self._gitea.post_comment(self._at(item).repo, loop.asked_on_pr, body)
except GiteaError:
self._transport.post(text, thread=loop.thread) # the PR is unreachable
return
url = loop.pr_url or f"PR #{loop.asked_on_pr}"
self._transport.post(
f"I answered your comment on PR #{loop.asked_on_pr}: {url}",
thread=loop.thread,
)
def _finish(
self,
item: WorkItem,
loop: Loop,
summary: str,
title: str = "",
final: bool = True,
) -> None:
"""finish_milestone: publish the milestone, or push a review fix."""
project = self._at(item)
milestones = project.milestones
if milestones is None:
note = "milestones aren't available here"
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
return
if loop.pr:
fixed = milestones.push_fix(
item_id=item.id,
branch=loop.branch,
pr=loop.pr,
message=summary or "Address review comments",
attempt=loop.fixes + 1,
)
self._fact(item, f"review fix: {fixed.note}")
if fixed.kind in ("pushed", "nothing"):
self._gitea_comment(project.repo, loop.pr, fixed.note)
self._wait(
item,
replace(loop, wait=MERGE, note=f"PR #{loop.pr}"),
f"PR #{loop.pr}: {fixed.note}",
)
elif fixed.kind == "new-pr" and fixed.pr is not None:
# Later fixes go to the new PR's branch, never the merged one.
loop = replace(
loop,
pr=fixed.pr.number,
pr_url=fixed.pr.url,
branch=fixed.branch,
fixes=loop.fixes + 1,
)
self._wait(
item,
replace(loop, wait=MERGE, note=f"PR #{fixed.pr.number}"),
f"The PR merged before the fix landed, so it's a new PR: "
f"#{fixed.pr.number} {fixed.pr.url}",
)
else:
self._wait(
item,
replace(loop, wait=ANSWER, note=fixed.note),
f"{fixed.note}. What next?",
)
return
if not loop.branch:
# D4: finishing without a start cuts the branch now.
number = loop.milestone + 1
branch = f"{item.id}-milestone-{number}-{uuid.uuid4().hex[:4]}"
try:
milestones.start(branch)
except GitError as exc:
note = f"I couldn't start milestone {number}: {exc}"
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
return
loop = replace(loop, milestone=number, branch=branch)
tested = preflight.tools_of(loop.toolchain)
published = milestones.publish(
item_id=item.id,
branch=loop.branch,
title=title or f"{item.title} (milestone {loop.milestone})",
summary=(
f"{summary}\n\nTested with: {preflight.said(tested)}"
if tested
else summary
),
issue=loop.issue,
final=final,
)
left = (
f" (left out {len(published.left_out)} build artifact(s))"
if published.left_out
else ""
)
if published.pr is not None and published.kind in ("opened", "adopted"):
pr = published.pr
self._fact(
item,
f"milestone {loop.milestone} published: PR #{pr.number} "
f"({pr.url}){left}",
)
self._wait(
item,
replace(
loop,
pr=pr.number,
pr_url=pr.url,
wait=MERGE,
note=f"PR #{pr.number}",
),
f"Milestone {loop.milestone} is up for review: {pr.url}{left}",
)
else:
self._fact(
item, f"milestone {loop.milestone} not published: {published.note}"
)
self._wait(
item,
replace(loop, wait=ANSWER, note=published.note),
f"Milestone {loop.milestone}: {published.note}. What next?",
)
def _gitea_comment(self, repo: str, pr: int, text: str) -> None:
try:
self._gitea.post_comment(repo, pr, text)
except GiteaError:
pass # courtesy only; the pushed commit is the answer
def _wait(self, item: WorkItem, loop: Loop, say: str | None) -> None:
if say:
self._transport.post(say, thread=loop.thread)
# Waiting again: whatever woke the item is settled.
loop = replace(loop, question=False, back_to="", back_note="")
save(self._store, item.id, WAITING, loop, cause=f"waiting for {loop.wait}")
def _close(self, item: WorkItem, loop: Loop, outcome: str, note: str) -> None:
self._fact(item, f"work item {outcome}: {note}")
project = self._at(item)
if loop.pr and outcome == "abandoned":
# L2: abandoning closes the item's open PR with a note.
try:
self._gitea.post_comment(project.repo, loop.pr, f"abandoned: {note}")
self._gitea.close_pull_request(project.repo, loop.pr)
except GiteaError as exc:
self._transport.post(
f"could not close PR #{loop.pr}: {exc}", thread=loop.thread
)
save(self._store, item.id, CLOSED, loop, cause=outcome)
self._store.close_work_item(item.id, outcome=outcome, cause=note)
if project.milestones is not None:
try:
project.milestones.drop_parked(f"{item.id}-")
except GitError:
pass # a leftover side branch; nothing depends on it
if loop.issue:
try:
self._gitea.post_comment(project.repo, loop.issue, f"{outcome}: {note}")
self._gitea.close_issue(project.repo, loop.issue)
except GiteaError as exc:
try:
blockers = self._gitea.open_dependencies(project.repo, loop.issue)
except GiteaError:
blockers = []
why = (
f"Gitea won't close an issue while it depends on open ones "
f"({', '.join(blockers)}): close it when they're done, or "
"remove the dependency"
if blockers
else str(exc)
)
self._transport.post(
f"closed here, but could not close #{loop.issue}: {why}",
thread=loop.thread,
)
if outcome == "abandoned":
self._transport.post(f"abandoned: {note}", thread=loop.thread)
def _fact(self, item: WorkItem, text: str) -> None:
self._store.transcript(item.id, "orchestrator").append(Fact(text=text))
def _instant(stamp: str) -> datetime:
"""An ISO 8601 time from Gitea or the state store, comparable either way."""
when = datetime.fromisoformat(stamp)
return when if when.tzinfo else when.replace(tzinfo=UTC)
Reference in new issue View Git Blame Copy Permalink