M4.4: an unreachable build VM blocks its projects until it's back (#137)

M4.3's block, with new triggers and a way out. A VM out of reach at the
check before a run, in a run (which ends there: the driver, a toolchain
step and a requested step), or while preparing the workspace blocks every
project on it; the item stays ready, waiting on no one. On the intake round
it is tried again every tuning.build_vm.retry_s (300), and once it answers
it must still answer settle_s (180) later; then the work a lost run left
there is parked (pushed), the VM unblocked, and that said once.

A changed host key, at the check or on a retry, blocks the VM too, until
the operator's `vm ready`, which trusts the new key: `trust new host key`
is retired, and the message says it's a key problem that may be an
impersonation. Questions are answered while a VM is blocked, without it.
Starting the agent with a VM down no longer fails.

The walk breaks the VM's SSH, now or partway through a run, and checks that
no model works on a VM that is down; loop.qnt models the VM going and
coming back (unblocked_only_when_up). Together they found: a lost run's
work set aside by the next item there; a question left behind a block; a
lost VM swallowed while the toolchain step checked or recorded it; and a
remark waiting to be answered that outlived its PR's merge. All fixed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

nisse-dev/tests/test_loop_toolchain.py

@@ -66,7 +66,7 @@ def markers(world: dict[str, Any]) -> str:
 
 
 def agent_for(
-    world: dict[str, Any], project: str, model: ScriptedModel
+    world: dict[str, Any], project: str, model: ScriptedModel, **more: Any
 ) -> tuple[LoopAgent, InMemoryTransport]:
     transport = InMemoryTransport()
     cancel = Cancel()
@@ -92,6 +92,7 @@ def agent_for(
         preflight=True,
         preflight_turns=5,
         markers=markers(world),
+        **more,
     )
     transport.on_message(loop.on_message)
     return loop, transport
@@ -236,7 +237,7 @@ def test_a_missing_baseline_waits_on_the_operator(
     assert loop.wait == ANSWER and "lacks no-such-tool-xyz" in loop.note
 
 
-def test_a_changed_host_key_waits_for_the_operators_word(
+def test_a_changed_host_key_blocks_the_vm_until_the_operators_word(
     world: dict[str, Any], project: str, monkeypatch: pytest.MonkeyPatch
 ) -> None:
     real = preflight.check
@@ -254,13 +255,15 @@ def test_a_changed_host_key_waits_for_the_operators_word(
     )
     agent, transport = agent_for(world, project, ScriptedModel(ready(), RESTATE))
     agent.intake()
-    loop = loop_of(the_item(world))
-    assert loop.host_key_changed and "trust new host key" in loop.note
+    [block] = world["store"].vm_blocks().values()
+    assert block.cause == "host key"
+    [asked] = [t for t in said(transport) if "different host key" in t]
+    assert f"`vm ready {project}`" in asked and "impersonating" in asked
+    assert "has a new host key" in agent.status()
     say(transport, world, "it's fine")  # an answer isn't the operator's word
-    still = loop_of(the_item(world))
-    assert (still.wait, still.host_key_changed, forgotten) == (ANSWER, True, [])
+    assert world["store"].vm_blocks() and forgotten == []
     changed["on"] = False
-    say(transport, world, "trust new host key")
+    transport.receive(DON, f"vm ready {project}")
     assert forgotten == ["yes"]
     assert loop_of(the_item(world)).wait == GO_AHEAD  # set up, then planned
 
@@ -386,7 +389,7 @@ def test_a_broken_vm_blocks_its_project_until_the_operator_says_ready(
     # Neither runs while the VM is blocked, even after a restart.
     again, _ = agent_for(world, project, ScriptedModel())
     assert again._queue() == []
-    assert "its build VM needs recovery" in again.status()
+    assert "its build VM needs reimaging" in again.status()
     # The operator reimages it: the clone and the marker are gone.
     shutil.rmtree(world["repo"].root)
     world["host"].execute(f'rm -f "{preflight.marker_path(project, markers(world))}"')