M4.4: an unreachable build VM blocks its projects until it's back (#137)

M4.3's block, with new triggers and a way out. A VM out of reach at the
check before a run, in a run (which ends there: the driver, a toolchain
step and a requested step), or while preparing the workspace blocks every
project on it; the item stays ready, waiting on no one. On the intake round
it is tried again every tuning.build_vm.retry_s (300), and once it answers
it must still answer settle_s (180) later; then the work a lost run left
there is parked (pushed), the VM unblocked, and that said once.

A changed host key, at the check or on a retry, blocks the VM too, until
the operator's `vm ready`, which trusts the new key: `trust new host key`
is retired, and the message says it's a key problem that may be an
impersonation. Questions are answered while a VM is blocked, without it.
Starting the agent with a VM down no longer fails.

The walk breaks the VM's SSH, now or partway through a run, and checks that
no model works on a VM that is down; loop.qnt models the VM going and
coming back (unblocked_only_when_up). Together they found: a lost run's
work set aside by the next item there; a question left behind a block; a
lost VM swallowed while the toolchain step checked or recorded it; and a
remark waiting to be answered that outlived its PR's merge. All fixed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

docs/Architecture.md

@@ -709,8 +709,12 @@ driver asks for the VM to be reimaged from its template. The harness pushes the
 work in progress first, then blocks the VM: nothing runs on any project on it,
 across restarts, until the operator says `vm ready` (which also trusts its new
 host key), and a reimaged VM is set up again by the toolchain step. A VM that
-can't be reached blocks the same way (#137). The operator recovers it for now;
-`nisse-ops` later (O5).
+can't be reached blocks the same way, whether the check before a run finds it
+or a run loses it (which ends that run there); it is tried again in the
+background, and work goes on by itself once it has answered for a while. A VM
+that answers with a new host key blocks until the operator's `vm ready`.
+Questions are still answered while a VM is blocked, without it. The operator
+recovers a VM for now; `nisse-ops` later (O5).
 
 ### D4. Git, branches and PRs
 

docs/Development-Plan.md

@@ -377,7 +377,8 @@ agent merging behind protection it can't satisfy, verified by a test (D4, D5, #6
 
 ### Phase 4 — The environment
 
-**M4.1 Toolchain preflight, M4.2 the toolchain step, M4.3 environment repair.**
+**M4.1 Toolchain preflight, M4.2 the toolchain step, M4.3 environment repair,
+M4.4 an unreachable build VM.**
 *Pre-approved.* All on the build VM (D3).
 
 **M4.1** is **done** (this PR). A model works out and sets up what a project
@@ -397,7 +398,7 @@ needs; code makes sure that happens, bounds it and checks the result.
 - Later runs check the versions again; a change is a fact and a thread line,
   and the PR body says what the work was tested with.
 - A changed host key is its own failure: the item waits for the operator's
-  `trust new host key`, then the VM is set up again.
+  `trust new host key`, then the VM is set up again (since M4.4, `vm ready`).
 - Decided with the operator: the model judges toolchains (the harness can't
   without it), preflight runs on the driver's model, gaps don't block. Turning
   off unattended upgrades and the template's baseline are template choices.
@@ -434,9 +435,34 @@ past the usual fixes (D3).
 - `loop.qnt` gained the block (`blocked_vm_runs_nothing`); it found that work in
   progress was lost on a reimage, hence parking first. The walk requests
   reimages and recovers them, reimaged or not.
-- Next: an unreachable VM blocks through the same mechanism, unblocked by a slow
-  background retry once it is back (#137). Later, nisse-ops reimages a VM itself
-  (M6.2, M7.6); a changed address means editing `build_host` and restarting.
+- Later, nisse-ops reimages a VM itself (M6.2, M7.6); a changed address means
+  editing `build_host` and restarting.
+
+**M4.4** *(#137, with the operator).* **Done** (this PR). A VM that can't be
+reached blocks through M4.3's mechanism; only the triggers and the way out
+differ.
+- Triggers: the check before a run, a run that loses the VM (it ends there, so
+  no model works against a missing VM: the toolchain step and a requested step
+  too), and preparing the workspace. The item stays ready, waiting on no one.
+- Tried again every `tuning.build_vm.retry_s` (5 min) on the intake round, said
+  only in the operations log; once it answers, it must still answer
+  `settle_s` (3 min) later, so a VM that has just booted is left to finish.
+  Then the work a lost run left on it is parked (pushed), before any item can
+  set it aside, and it is unblocked and said once. `vm ready` clears it too;
+  the message warns that a reimage loses what isn't pushed.
+- A changed host key, at the check or on a retry, blocks the VM as well, and is
+  cleared only by `vm ready`, which trusts the new key: `trust new host key` is
+  retired, and the message says it's a key problem and that it may be an
+  impersonation.
+- Questions are answered while a VM is blocked, without it (L5: a question
+  changes nothing). Starting the agent with a VM down no longer fails.
+- The walk breaks the VM's SSH, now or partway through a run, and checks that
+  no model works on a VM that is down. With `loop.qnt`, it found: work a lost
+  run left behind set aside by the next item there (hence parking first); a
+  question left unanswered behind a block (hence answering without the VM);
+  a lost VM swallowed while the toolchain step wrote its marker; and a remark
+  waiting to be answered that outlived its PR's merge, sending the item back
+  to wait for a merge that had happened. All fixed.
 
 ### Phase 5 — Memory
 
@@ -496,7 +522,7 @@ vision; one local model for every agent; several forges (#26); a room per projec
 | #69 inline review comments | M2.1 |
 | #70 watch stall | M2.1 |
 | #77 conversational agent | M2.3 (questions), M2.7 |
-| #137 block on an unreachable build VM | After M4.3, which built the block |
+| #137 block on an unreachable build VM | M4.4 |
 
 ## 6. Implementation notes
 

docs/Tuning-Guide.md

@@ -565,3 +565,25 @@ reply in its thread starts a new conversation linked to it.
 - **Too low:** a conversation you come back to after lunch has to be picked up
   again from its summary.
 - **Too high:** idle conversations pile up in `status` and in the queue.
+
+### `tuning.build_vm.retry_s` — default 300
+
+How often a build VM that can't be reached is tried again (architecture D3).
+While it's out of reach, every project on it is blocked; each try is one SSH
+connection that gives up after 15 seconds.
+
+- **Too low:** a VM that is down for long is tried constantly, and each try
+  holds up the agent's other duties for up to 15 seconds.
+- **Too high:** work waits longer than it needs to after the VM is back.
+- **Deciding:** about how long a reboot of the VM's host takes. A VM you know
+  is back can be cleared at once with `vm ready <project>`.
+
+### `tuning.build_vm.settle_s` — default 180
+
+Once an unreachable VM answers again, how long it must keep answering before
+work goes on: SSH comes up before a VM has finished booting.
+
+- **Too low:** work starts on a VM still running cloud-init or starting
+  services, and fails in confusing ways.
+- **Too high:** work waits after the VM is ready.
+- **Deciding:** how long the template takes from SSH answering to fully up.