dev/nisse
2
1 Fork 0
Code Issues 23 Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

M4.4: an unreachable build VM blocks its projects until it's back (#137)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m43s
Details
CI / e2e (pull_request) Successful in 3m53s
Details

Browse Source 
M4.3's block, with new triggers and a way out. A VM out of reach at the
check before a run, in a run (which ends there: the driver, a toolchain
step and a requested step), or while preparing the workspace blocks every
project on it; the item stays ready, waiting on no one. On the intake round
it is tried again every tuning.build_vm.retry_s (300), and once it answers
it must still answer settle_s (180) later; then the work a lost run left
there is parked (pushed), the VM unblocked, and that said once.

A changed host key, at the check or on a retry, blocks the VM too, until
the operator's `vm ready`, which trusts the new key: `trust new host key`
is retired, and the message says it's a key problem that may be an
impersonation. Questions are answered while a VM is blocked, without it.
Starting the agent with a VM down no longer fails.

The walk breaks the VM's SSH, now or partway through a run, and checks that
no model works on a VM that is down; loop.qnt models the VM going and
coming back (unblocked_only_when_up). Together they found: a lost run's
work set aside by the next item there; a question left behind a block; a
lost VM swallowed while the toolchain step checked or recorded it; and a
remark waiting to be answered that outlived its PR's merge. All fixed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-10-08 21:06:05 +00:00
1 parent 1091178a81
commit a921b73d34
19 files changed
+935 -158
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

nisse-dev/src/nisse_dev/loop/agent.py
+265 -107
View File
@@ -56,7 +56,13 @@ from nisse_dev.llm import LLMError, RoleMap
from nisse_dev.loop import preflight
from nisse_dev.loop.actions import classify, commits
from nisse_dev.loop.classifier import IN_SCOPE, ActionClassifier, Scope
from nisse_dev.loop.driver import ASK_ALLOW, Driver, RunOutcome, Screening
from nisse_dev.loop.driver import (
ASK_ALLOW,
Driver,
RunOutcome,
Screening,
VMUnreachable,
)
from nisse_dev.loop.help import ASK_EXPERT, DELEGATE, Helpers
from nisse_dev.loop.items import (
ANSWER,
@@ -112,7 +118,7 @@ _COMMITTED = (
)
# The operator's word that a blocked build VM is ready again (D3).
_VM_READY = re.compile(r"vm ready(?: ([a-z0-9][a-z0-9-]*))?")
_NEEDS_ITEM = frozenset({"go", "approve", "abandon", "allow", "trust new host key"})
_NEEDS_ITEM = frozenset({"go", "approve", "abandon", "allow"})
# Near-misses for abandon: asked about, never taken for it or for a reply.
_NEAR_ABANDON = frozenset(
{
@@ -129,7 +135,19 @@ _NEAR_ABANDON = frozenset(
BUG_GRACE_S = 600.0
GENERAL_IDLE_S = 86400.0
_SAVED = "The work in progress is pushed; anything else on it will be lost."
BROKEN = "broken" # a VM block's cause: the driver found it broken (D3)
# A build VM block's causes (D3): the driver found it broken (M4.3); it
# can't be reached, and is tried again; it answered with a new host key.
BROKEN = "broken"
UNREACHABLE = "unreachable"
HOST_KEY = "host key"
# How each reads in a sentence about the VM.
_BLOCKED_BY = {
BROKEN: "needs reimaging",
UNREACHABLE: "is out of reach",
HOST_KEY: "has a new host key",
}
VM_RETRY_S = 300.0
VM_SETTLE_S = 180.0
PROGRESS_EVERY_S = 60.0 # progress in prose at a readable rate (L5, #42)
PROGRESS_CHARS = 400
DETAILS = 10 # tool calls shown by `details`
@@ -213,6 +231,9 @@ The operator has asked a question (the last message). Answer it with reply,
from what you know and what you can look at; don't change anything. The item
then goes back to waiting exactly as it was, so a restatement up stays up."""
OFFLINE = """ The build VM is out of reach just now, so you can't look at
anything: answer from what you know, and say what you'd check."""
CONVERSATION = """You are nisse-dev, a software agent, in a conversation with
the operator in a Matrix thread. It isn't about a project yet: it may be a
question, thoughts on feasibility or hardware, or the start of some work.
@@ -263,6 +284,11 @@ class Project:
merge: str = "operator" # the trust dial (L3): who merges its PRs
def _minutes(seconds: float) -> str:
m = max(1, round(seconds / 60))
return f"{m} minute{'s' if m != 1 else ''}"
def _restatement_problem(args: dict[str, object]) -> str | None:
"""L5, D1: a restatement is agreed to as it stands, so it asks nothing.
One that ends in a question is sent back: questions go by ask_human."""
@@ -313,6 +339,8 @@ class LoopAgent:
markers: str = preflight.MARKERS,
clock: Callable[[], float] = time.time,
intake_interval_s: float = INTAKE_INTERVAL_S,
vm_retry_s: float = VM_RETRY_S,
vm_settle_s: float = VM_SETTLE_S,
) -> None:
self._store = store
self._transport = transport
@@ -348,6 +376,11 @@ class LoopAgent:
self._judged: dict[tuple[str, int, str], tuple[Verdict, tuple[str, str]]] = {}
self._clock = clock
self._interval = intake_interval_s
# Unreachable build VMs (#137): when each is next tried, and since
# when it has answered. Volatile: after a boot, tried at once.
self._vm_retry = vm_retry_s
self._vm_settle = vm_settle_s
self._probes: dict[str, tuple[float, float | None]] = {}
# thread root -> revision on screen there, read on the transport's
# thread to stamp replies on arrival (L3); (project, issue) -> thread
# root, for a command that names its item.
@@ -471,21 +504,12 @@ class LoopAgent:
root = str(project.workspace.root)
try:
ready = preflight.check(host, project.name, root, self._markers)
except HostKeyChanged:
note = (
f"the build VM for {project.name} ({host.describe()}) answered "
"with a different host key than the one I pinned. If you "
"rebuilt it, say `trust new host key` and I'll set it up "
"again; if you didn't, something is impersonating it."
)
self._wait(
item, replace(loop, wait=ANSWER, note=note, host_key_changed=True), note
)
except HostKeyChanged as exc:
self._block_vm(host.describe(), HOST_KEY, str(exc), item)
self._store.set_active(None, cause="host key changed")
return None
except HostUnavailable as exc:
note = f"I can't reach the build VM: {exc}. Reply to retry."
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
self._block_vm(host.describe(), UNREACHABLE, str(exc), item)
self._store.set_active(None, cause="build VM unreachable")
return None
if ready.missing:
@@ -540,8 +564,9 @@ class LoopAgent:
)
try:
now = preflight.versions(host, before)
except HostUnavailable:
return loop # the run will find out, and say so
except HostUnavailable as exc:
self._settle(item.id, RunOutcome("unreachable", note=str(exc)))
return None
changed = preflight.drift(before, now)
if changed:
said = "the toolchain changed since it was checked: " + "; ".join(changed)
@@ -590,9 +615,13 @@ class LoopAgent:
if requested
else "toolchain step: setting the build VM up from the repository",
)
before = self._changed_files(project)
roles = self._roles_of(item)
outer = self._running # a requested step runs inside the driver's run
try:
before = self._changed_files(project)
except HostUnavailable as exc:
self._lost_in_step(item, outer, exc)
return None
roles = self._roles_of(item)
if not outer:
self._active_thread = loop.thread
self._active_name = self._label(item)
@@ -620,6 +649,11 @@ class LoopAgent:
self._cancel.clear()
self._settle(item.id, outcome)
return None
if outcome.kind == "unreachable":
if outer:
raise VMUnreachable(outcome.note) # ends the driver's run too
self._settle(item.id, outcome)
return None
host = workspace.host
now = loop_of(self._store.work_item(item.id) or item)
if outcome.kind == "wait" and outcome.tool == "toolchain_ready":
@@ -646,8 +680,10 @@ class LoopAgent:
"at": datetime.now(UTC).isoformat(timespec="seconds"),
},
)
except HostUnavailable:
tools = report.tools
except HostUnavailable as exc:
# #137: no marker, so no work: the VM is blocked instead.
self._lost_in_step(item, outer, exc)
return None
missing = [f"{name} ({why})" for name, why in report.missing]
parts = [f"toolchain: {preflight.said(tools)}"]
if installed:
@@ -658,7 +694,11 @@ class LoopAgent:
parts.append(report.summary)
if not requested:
# D3: the harness's step edits nothing in the repository.
edited = sorted(set(self._changed_files(project)) - set(before))
try:
edited = sorted(set(self._changed_files(project)) - set(before))
except HostUnavailable as exc:
self._lost_in_step(item, outer, exc)
return None
if edited:
parts.append(
"it changed files it shouldn't have, left for the work: "
@@ -671,16 +711,20 @@ class LoopAgent:
loop = replace(now, toolchain=tuple(tuple(t.to_data()) for t in tools))
return loop, said
def _lost_in_step(self, item: WorkItem, outer: bool, exc: Exception) -> None:
"""A toolchain step lost its VM (#137): a requested one ends the
driver's run with it; the harness's own blocks the VM, settled."""
if outer:
raise VMUnreachable(str(exc)) from exc
self._settle(item.id, RunOutcome("unreachable", note=str(exc)))
def _changed_files(self, project: Project) -> list[str]:
"""The workspace's changes, as git sees them (for the setup step's
promise to edit nothing)."""
assert project.workspace is not None
try:
result = project.workspace.host.execute(
"git status --porcelain", cwd=str(project.workspace.root), timeout_s=30
)
except HostUnavailable:
return []
result = project.workspace.host.execute(
"git status --porcelain", cwd=str(project.workspace.root), timeout_s=30
) # HostUnavailable: the step's caller handles a lost VM (#137)
return [line[3:] for line in result.stdout.splitlines() if line.strip()]
def _requested_step(self, item_id: str, reason: str) -> tuple[str, bool]:
@@ -943,6 +987,7 @@ class LoopAgent:
for project in self._projects.values():
self._intake(project)
self.watch()
self.check_vms()
self.check_blocked()
self.check_grace()
self.check_idle()
@@ -1042,6 +1087,10 @@ class LoopAgent:
and self._at(item).merge == "approved"
):
pr, loop = self._merge_if_approved(item, loop, repo, pr)
if pr.state != "open" and loop.back_to == MERGE:
# A remark on it waited to be answered (a later run): the
# merge it would go back to waiting for is over.
loop = replace(loop, back_to="", back_note="")
if pr.merged:
self._fact(item, f"PR #{loop.pr} merged")
loop, position = replace(loop, pr=0, pr_url="", branch=""), READY
@@ -1392,27 +1441,6 @@ class LoopAgent:
),
cause="the operator asked a question",
)
elif command == "trust new host key":
project = self._at(item)
if loop.host_key_changed and project.workspace is not None:
project.workspace.host.forget_host_key()
self._fact(
item,
f"the operator ({msg.sender}) trusted the build VM's new "
"host key: it will be set up again",
)
save(
self._store,
item.id,
READY,
replace(loop, wait="", note="", host_key_changed=False),
cause=f"new host key trusted by {msg.sender}",
)
else:
self._transport.post(
f"Nothing on {self._label(item)} is waiting for that.",
thread=loop.thread,
)
elif command == "allow":
if item.position == WAITING and loop.pending_action:
action = json.loads(loop.pending_action)
@@ -1838,7 +1866,9 @@ class LoopAgent:
ready = [
i
for i in items
if i.position == READY and self._host_of(i.scope) not in blocks
if i.position == READY
# L5: a question is answered even while its VM is blocked.
and (self._host_of(i.scope) not in blocks or loop_of(i).question)
]
return sorted(ready, key=lambda i: -loop_of(i).first)
@@ -1873,8 +1903,11 @@ class LoopAgent:
what = f"blocked by {', '.join(loop.blocked_by)}"
elif item.position == READY and item.id not in queue:
block = self._store.vm_blocks().get(self._host_of(item.scope) or "")
why = block.reason if block else "unknown"
what = f"blocked: its build VM needs recovery ({why})"
what = (
f"blocked: its build VM {_BLOCKED_BY[block.cause]} ({block.reason})"
if block
else "blocked: its build VM is blocked"
)
elif item.position == READY:
what = f"ready, {queue.index(item.id) + 1} of {len(queue)} in line"
name = (
@@ -2415,13 +2448,20 @@ class LoopAgent:
checked = self._preflight and project.workspace is not None
marker: dict[str, Any] | None = None
starting = not loop.prepared
if checked and milestones is not None:
# L5: a question doesn't wait for a blocked build VM: it is answered
# without one (#137).
offline = loop.question and self._vm_blocked(project.name)
if checked and milestones is not None and not offline:
ready = self._readiness(item, project, loop)
if ready is None:
return # waiting on the operator
loop, marker = ready
starting = not loop.prepared
if milestones is not None:
if ready is None and loop.question and self._vm_blocked(project.name):
offline = True
self._store.set_active(item.id, cause="a question, without the VM")
elif ready is None:
return # waiting on the operator, or on the VM
else:
loop, marker = ready
starting = not loop.prepared
if milestones is not None and not offline:
keep = self._keep(project.name)
try:
if not loop.prepared:
@@ -2434,6 +2474,12 @@ class LoopAgent:
loop.branch or f"{item.id}-work", keep=keep, mine=f"{item.id}-"
)
except Exception as exc: # noqa: BLE001 -- the item waits, not the loop
host = self._host_of(project.name)
if isinstance(exc.__cause__, HostUnavailable) and host is not None:
# #137: the VM went between the check and this.
self._block_vm(host, UNREACHABLE, str(exc.__cause__), item)
self._store.set_active(None, cause="build VM unreachable")
return
note = f"I couldn't prepare the workspace: {exc}. Reply to retry."
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
self._store.set_active(None, cause="workspace failed")
@@ -2446,10 +2492,10 @@ class LoopAgent:
self._transport.post(said, thread=loop.thread)
loop = replace(loop, prepared=True)
save(self._store, item.id, RUNNING, loop, cause="driver run")
if checked and milestones is not None:
if checked and milestones is not None and not offline:
toolchain = self._toolchain(item, project, loop, marker, starting)
if toolchain is None:
return # preflight was stopped: settled
return # stopped, or the VM went: settled
loop = toolchain
save(self._store, item.id, RUNNING, loop, cause="toolchain checked")
# D1: a `discuss` item talks before it restates.
@@ -2478,23 +2524,27 @@ class LoopAgent:
),
]
if loop.question:
tools = [REPLY, *_LOOK]
tools = [REPLY] if offline else [REPLY, *_LOOK]
elif loop.back_to and not loop.started:
tools = [*tools, REPLY] # before the go-ahead, a remark answered
roles = self._roles_of(item)
workspace = project.workspace
system = SYSTEM.format(
repo=project.repo,
issue=loop.issue,
title=item.title,
labels="".join(f"{LABELS[x]}\n\n" for x in loop.labels if x in LABELS),
helpers=(
f"{self._helpers.briefing(roles)}\n\n"
if self._helpers and loop.started
else ""
),
workspace=workspace.describe() if workspace else "",
) + (QUESTION if loop.question else "")
workspace = None if offline else project.workspace
system = (
SYSTEM.format(
repo=project.repo,
issue=loop.issue,
title=item.title,
labels="".join(f"{LABELS[x]}\n\n" for x in loop.labels if x in LABELS),
helpers=(
f"{self._helpers.briefing(roles)}\n\n"
if self._helpers and loop.started
else ""
),
workspace=workspace.describe() if workspace else "",
)
+ (QUESTION if loop.question else "")
+ (OFFLINE if offline else "")
)
self._active_thread = loop.thread
self._active_name = self._label(item)
self._running = True
@@ -2728,6 +2778,13 @@ class LoopAgent:
summary = str(args.get("summary", "")).strip()
self._transport.post(f"Done: {summary}", thread=loop.thread)
self._close(item, loop, "completed", summary)
elif outcome.kind == "unreachable":
# #137: not the work's fault. The item goes on when the VM is back.
save(self._store, item.id, READY, loop, cause="build VM unreachable")
project = self._at(item)
host = self._host_of(project.name)
if host is not None:
self._block_vm(host, UNREACHABLE, outcome.note, item)
elif outcome.kind == "stopped":
note = f"stopped ({outcome.note}). Why did you stop it? Reply to continue."
self._wait(item, replace(loop, wait=ANSWER, note=note), note)
@@ -2772,36 +2829,138 @@ class LoopAgent:
self._block_vm(host, BROKEN, reason, item, risk)
def _block_vm(
self, host: str, cause: str, reason: str, by: WorkItem, risk: str = ""
self,
host: str,
cause: str,
reason: str,
by: WorkItem | None,
risk: str = "",
) -> None:
"""Block a build VM (D3): every project on it waits until it is
recovered. Said once at top level, and in each waiting item's thread.
The recovery is the operator's for now; nisse-ops' later (M7.6)."""
self._store.block_vm(host, cause=cause, reason=reason, work_item=by.id)
names = sorted(p for p in self._projects if self._host_of(p) == host)
blocked = [
i
for i in self._store.work_items("open")
if i.scope in names and i.scope != GENERAL
]
for other in blocked:
line = (
f"Blocked: the build VM ({host}) needs reimaging. "
f"Waiting for `vm ready {other.scope}`."
The recovery is the operator's for now, or the VM's coming back;
nisse-ops' later (M7.6)."""
if host in self._store.vm_blocks():
return # already said
self._store.block_vm(
host, cause=cause, reason=reason, work_item=by.id if by else None
)
self._probes.pop(host, None)
names = self._on_host(host)
for other in self._store.work_items("open"):
if other.scope not in names:
continue
line = f"Blocked: the build VM ({host}) {_BLOCKED_BY[cause]}. " + (
"I'll go on once it's back."
if cause == UNREACHABLE
else f"Waiting for `vm ready {other.scope}`."
)
self._fact(other, line)
self._transport.post(line, thread=loop_of(other).thread)
self._transport.post(
f"The build VM {host} seems broken, found working on "
f"{self._label(by)}: {reason}\n\nThe projects on it are blocked "
f"until it's fixed: {', '.join(names)}. Could you reimage it from "
f"its template? {risk or _SAVED} If its "
self._transport.post(self._blocked_message(host, cause, reason, by, risk))
def _blocked_message(
self, host: str, cause: str, reason: str, by: WorkItem | None, risk: str
) -> str:
names = self._on_host(host)
ready = " or ".join(f"`vm ready {n}`" for n in names)
found = f", found working on {self._label(by)}" if by else ""
blocked = f"The projects on it are blocked: {', '.join(names)}."
if cause == UNREACHABLE:
return (
f"I can't reach the build VM {host}{found}: {reason}\n\n"
f"{blocked} I'll try again every {_minutes(self._vm_retry)} and "
f"go on once it has answered for {_minutes(self._vm_settle)}. If "
f"it's gone for good, reimage it and say {ready}; anything not "
"pushed on it will be lost."
)
if cause == HOST_KEY:
return (
f"The build VM {host} answered with a different host key than "
f"the one I pinned{found}. {blocked}\n\nIf you rebuilt it, say "
f"{ready}: that trusts its new key, and I'll set it up again. "
"If you didn't, something may be impersonating it, so don't."
)
return (
f"The build VM {host} seems broken{found}: {reason}\n\n{blocked} "
f"Could you reimage it from its template? {risk or _SAVED} If its "
"address changes, update `build_host` in my configuration and "
"restart me first.\n\nOnce it's ready, say "
+ " or ".join(f"`vm ready {n}`" for n in names)
+ "; that also trusts its new host key."
f"restart me first.\n\nOnce it's ready, say {ready}; that also "
"trusts its new host key."
)
def _vm_blocked(self, project: str) -> bool:
return self._host_of(project) in self._store.vm_blocks()
def _on_host(self, host: str) -> list[str]:
return sorted(p for p in self._projects if self._host_of(p) == host)
def _park_on(self, host: str) -> None:
"""A run cut off by its VM going away couldn't park its work: park
each project's workspace on the VM before anything else is taken up
there, which would set that work aside (#137). Raises when the VM
can't be reached."""
for name in self._on_host(host):
milestones = self._projects[name].milestones
if milestones is not None:
milestones.park(keep=self._keep(name))
def _unblock_vm(self, host: str, cause: str, said: str) -> None:
self._store.unblock_vm(host, cause=cause)
self._probes.pop(host, None)
for other in self._store.work_items("open"):
if other.scope in self._on_host(host):
self._fact(other, said)
def check_vms(self) -> None:
"""#137: each unreachable build VM, tried again when due. Once it
answers, it must still answer a settle period later before work goes
on (SSH is up before the VM has finished booting). A new host key
is the operator's to trust; a broken VM, theirs to reimage."""
now = self._clock()
for host, block in self._store.vm_blocks().items():
if block.cause != UNREACHABLE:
continue
due, answered = self._probes.get(host, (now, None))
if now < due:
continue
names = self._on_host(host)
if not names:
continue # no project on it any more
workspace = self._projects[names[0]].workspace
assert workspace is not None
try:
workspace.host.execute("true", timeout_s=30)
except HostKeyChanged as exc:
self._store.reblock_vm(host, cause=HOST_KEY, reason=str(exc))
self._probes.pop(host, None)
self._transport.post(
self._blocked_message(host, HOST_KEY, str(exc), None, "")
)
continue
except HostUnavailable as exc:
self._oplog.emit("vm.unreachable", cause=str(exc), host=host)
self._probes[host] = (now + self._vm_retry, None)
continue
if answered is None or now - answered < self._vm_settle:
since = now if answered is None else answered
self._probes[host] = (since + self._vm_settle, since)
continue
try:
self._park_on(host)
except Exception as exc: # noqa: BLE001 -- not back after all
self._oplog.emit("vm.unreachable", cause=repr(exc), host=host)
self._probes[host] = (now + self._vm_retry, None)
continue
self._unblock_vm(
host,
"it answered again",
f"the build VM ({host}) is back: work goes on",
)
self._transport.post(
f"The build VM {host} is back: {', '.join(names)} can go on."
)
def _vm_ready(self, name: str | None, sender: str) -> str:
"""The operator's `vm ready`: the build VM's block lifts and its new
host key is trusted (D3). A reimaged VM has no marker, so its first
@@ -2824,18 +2983,17 @@ class LoopAgent:
if host is None or workspace is None or host not in blocks:
return f"Nothing is blocking {name}'s build VM."
workspace.host.forget_host_key()
block = self._store.unblock_vm(host, cause=f"vm ready from {sender}")
names = sorted(p for p in self._projects if self._host_of(p) == host)
for other in self._store.work_items("open"):
if other.scope in names:
self._fact(
other,
f"the operator ({sender}) said the build VM ({host}) is "
"ready: its new host key is trusted, and it is set up again "
"if it was reimaged",
)
assert block is not None # checked above, on the worker
return f"{host} is unblocked: {', '.join(names)} can go on."
try:
self._park_on(host)
except Exception as exc: # noqa: BLE001 -- reimaged: nothing left there
self._oplog.emit("vm.park_failed", cause=repr(exc), host=host)
self._unblock_vm(
host,
f"vm ready from {sender}",
f"the operator ({sender}) said the build VM ({host}) is ready: its "
"new host key is trusted, and it is set up again if it was reimaged",
)
return f"{host} is unblocked: {', '.join(self._on_host(host))} can go on."
# -- milestones (D4) -----------------------------------------------------