Closes#137. M4.4: a build VM that can't be reached blocks every project on it through M4.3's block. Only the triggers and the way out differ. As agreed, a changed host key now blocks the VM too, cleared by vm ready.
Triggers
The check before a run finds the VM out of reach. The item stays ready and isn't waiting on you.
A run loses the VM. The run ends right there, so no model works against a missing VM. ToolOutcome.unreachable gives the driver a new outcome, unreachable. A toolchain step the driver requested raises VMUnreachable, which ends the driver's own run.
Preparing the workspace loses the VM, in the gap after the check.
A changed host key, found at the check or on a retry, blocks the VM until you say vm ready.
The way out
Background retries on the intake round, every tuning.build_vm.retry_s (300 s). Each retry goes only to the operations log.
A settle period. Once the VM answers, it must still answer settle_s (180 s) later.
Then:
the work a lost run left on the VM is parked (pushed), before anything else can set it aside;
the VM is unblocked;
you hear once: "The build VM … is back: … can go on."
vm ready also clears it. The message warns that a reimage loses whatever isn't pushed.
trust new host key is retired. The host-key message says it's a key problem and that it may be an impersonation.
Also
Questions are answered while a VM is blocked, without the VM (reply only), as L5 requires.
nisse-dev now starts even when a VM is down. Before, Workspace construction raised an error.
What the walk and loop.qnt found (all fixed)
The walk now breaks the VM's SSH, either straight away or partway through a run, and checks that no model works on a VM that is down. loop.qnt now models the VM going and coming back (unblocked_only_when_up). Between them they found five problems:
A lost run's work was set aside by the next item on that VM. Fixed by parking it before unblocking.
A question sat unanswered behind a block.
A lost VM was swallowed while the toolchain step wrote its marker, so the driver started without one.
The same happened with the step's git status.
A remark waiting to be answered outlived its PR's merge, and sent the item back to waiting for a merge that had already happened. This one exists without a block too, whenever that run is delayed. It has its own test in test_loop_live_fixes.py.
The walk check parked_work_safe now counts only writes that ran.
Tests
test_loop_vm.py, 11 tests:
an unreachable VM blocks its project, with retries only when due, a settle period, and a failure during settling resetting the wait;
losing the VM mid-run, and inside a requested step;
a new key on return;
vm ready clears an unreachable block;
a broken block is never probed;
a restart retries at once;
a question answered offline;
the VM lost at the marker and at git status;
a lost run's work is parked before the VM is reused.
Mutant checks: the tests fail without the settle period, without the driver's stop, without the park, and without the back_to fix.
Locally: the unit tier (697 tests), 96 walks (two batches of 48 seeds), and 200,000 loop.qnt traces pass. A mutant model without the probe's guard fails unblocked_only_when_up.
Not covered
A helper (delegation) runs its own loop and isn't stopped mid-way. The driver's next call to the VM catches it.
A changed VM address is still "edit build_host and restart" (M6.2, M7.6).
Closes #137. M4.4: a build VM that can't be reached blocks every project on it through M4.3's block. Only the triggers and the way out differ. As agreed, a changed host key now blocks the VM too, cleared by `vm ready`.
## Triggers
- **The check before a run** finds the VM out of reach. The item stays ready and isn't waiting on you.
- **A run loses the VM.** The run ends right there, so no model works against a missing VM. `ToolOutcome.unreachable` gives the driver a new outcome, `unreachable`. A toolchain step the driver requested raises `VMUnreachable`, which ends the driver's own run.
- **Preparing the workspace** loses the VM, in the gap after the check.
- **A changed host key**, found at the check or on a retry, blocks the VM until you say `vm ready`.
## The way out
- **Background retries** on the intake round, every `tuning.build_vm.retry_s` (300 s). Each retry goes only to the operations log.
- **A settle period.** Once the VM answers, it must still answer `settle_s` (180 s) later.
- **Then:**
1. the work a lost run left on the VM is parked (pushed), before anything else can set it aside;
2. the VM is unblocked;
3. you hear once: "The build VM … is back: … can go on."
- **`vm ready`** also clears it. The message warns that a reimage loses whatever isn't pushed.
- **`trust new host key` is retired.** The host-key message says it's a key problem and that it may be an impersonation.
## Also
- **Questions are answered while a VM is blocked**, without the VM (`reply` only), as L5 requires.
- **nisse-dev now starts even when a VM is down.** Before, `Workspace` construction raised an error.
## What the walk and `loop.qnt` found (all fixed)
The walk now breaks the VM's SSH, either straight away or partway through a run, and checks that no model works on a VM that is down. `loop.qnt` now models the VM going and coming back (`unblocked_only_when_up`). Between them they found five problems:
1. A lost run's work was set aside by the next item on that VM. Fixed by parking it before unblocking.
2. A question sat unanswered behind a block.
3. A lost VM was swallowed while the toolchain step wrote its marker, so the driver started without one.
4. The same happened with the step's `git status`.
5. A remark waiting to be answered outlived its PR's merge, and sent the item back to waiting for a merge that had already happened. This one exists without a block too, whenever that run is delayed. It has its own test in `test_loop_live_fixes.py`.
The walk check `parked_work_safe` now counts only writes that ran.
## Tests
- **`test_loop_vm.py`, 11 tests:**
- an unreachable VM blocks its project, with retries only when due, a settle period, and a failure during settling resetting the wait;
- losing the VM mid-run, and inside a requested step;
- a new key on return;
- `vm ready` clears an unreachable block;
- a `broken` block is never probed;
- a restart retries at once;
- a question answered offline;
- the VM lost at the marker and at `git status`;
- a lost run's work is parked before the VM is reused.
- **Mutant checks:** the tests fail without the settle period, without the driver's stop, without the park, and without the `back_to` fix.
- **Locally:** the unit tier (697 tests), 96 walks (two batches of 48 seeds), and 200,000 `loop.qnt` traces pass. A mutant model without the probe's guard fails `unblocked_only_when_up`.
## Not covered
- A helper (delegation) runs its own loop and isn't stopped mid-way. The driver's next call to the VM catches it.
- A changed VM address is still "edit `build_host` and restart" (M6.2, M7.6).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
M4.3's block, with new triggers and a way out. A VM out of reach at the
check before a run, in a run (which ends there: the driver, a toolchain
step and a requested step), or while preparing the workspace blocks every
project on it; the item stays ready, waiting on no one. On the intake round
it is tried again every tuning.build_vm.retry_s (300), and once it answers
it must still answer settle_s (180) later; then the work a lost run left
there is parked (pushed), the VM unblocked, and that said once.
A changed host key, at the check or on a retry, blocks the VM too, until
the operator's `vm ready`, which trusts the new key: `trust new host key`
is retired, and the message says it's a key problem that may be an
impersonation. Questions are answered while a VM is blocked, without it.
Starting the agent with a VM down no longer fails.
The walk breaks the VM's SSH, now or partway through a run, and checks that
no model works on a VM that is down; loop.qnt models the VM going and
coming back (unblocked_only_when_up). Together they found: a lost run's
work set aside by the next item there; a question left behind a block; a
lost VM swallowed while the toolchain step checked or recorded it; and a
remark waiting to be answered that outlived its PR's merge. All fixed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
dongar
Closes #137. M4.4: a build VM that can't be reached blocks every project on it through M4.3's block. Only the triggers and the way out differ. As agreed, a changed host key now blocks the VM too, cleared by
vm ready.Triggers
ToolOutcome.unreachablegives the driver a new outcome,unreachable. A toolchain step the driver requested raisesVMUnreachable, which ends the driver's own run.vm ready.The way out
tuning.build_vm.retry_s(300 s). Each retry goes only to the operations log.settle_s(180 s) later.vm readyalso clears it. The message warns that a reimage loses whatever isn't pushed.trust new host keyis retired. The host-key message says it's a key problem and that it may be an impersonation.Also
replyonly), as L5 requires.Workspaceconstruction raised an error.What the walk and
loop.qntfound (all fixed)The walk now breaks the VM's SSH, either straight away or partway through a run, and checks that no model works on a VM that is down.
loop.qntnow models the VM going and coming back (unblocked_only_when_up). Between them they found five problems:git status.test_loop_live_fixes.py.The walk check
parked_work_safenow counts only writes that ran.Tests
test_loop_vm.py, 11 tests:vm readyclears an unreachable block;brokenblock is never probed;git status;back_tofix.loop.qnttraces pass. A mutant model without the probe's guard failsunblocked_only_when_up.Not covered
build_hostand restart" (M6.2, M7.6).🤖 Generated with Claude Code
https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk