M4.4: an unreachable build VM blocks its projects until it's back (#137)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m43s
CI / e2e (pull_request) Successful in 3m53s

M4.3's block, with new triggers and a way out. A VM out of reach at the
check before a run, in a run (which ends there: the driver, a toolchain
step and a requested step), or while preparing the workspace blocks every
project on it; the item stays ready, waiting on no one. On the intake round
it is tried again every tuning.build_vm.retry_s (300), and once it answers
it must still answer settle_s (180) later; then the work a lost run left
there is parked (pushed), the VM unblocked, and that said once.

A changed host key, at the check or on a retry, blocks the VM too, until
the operator's `vm ready`, which trusts the new key: `trust new host key`
is retired, and the message says it's a key problem that may be an
impersonation. Questions are answered while a VM is blocked, without it.
Starting the agent with a VM down no longer fails.

The walk breaks the VM's SSH, now or partway through a run, and checks that
no model works on a VM that is down; loop.qnt models the VM going and
coming back (unblocked_only_when_up). Together they found: a lost run's
work set aside by the next item there; a question left behind a block; a
lost VM swallowed while the toolchain step checked or recorded it; and a
remark waiting to be answered that outlived its PR's merge. All fixed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-10-08 21:06:05 +00:00
1 parent 1091178a81
commit a921b73d34
19 files changed
+935 -158

No files matched your search

+32 -6
View File
@@ -377,7 +377,8 @@ agent merging behind protection it can't satisfy, verified by a test (D4, D5, #6
### Phase 4 — The environment
**M4.1 Toolchain preflight, M4.2 the toolchain step, M4.3 environment repair.**
**M4.1 Toolchain preflight, M4.2 the toolchain step, M4.3 environment repair,
M4.4 an unreachable build VM.**
*Pre-approved.* All on the build VM (D3).
**M4.1** is **done** (this PR). A model works out and sets up what a project
@@ -397,7 +398,7 @@ needs; code makes sure that happens, bounds it and checks the result.
- Later runs check the versions again; a change is a fact and a thread line,
and the PR body says what the work was tested with.
- A changed host key is its own failure: the item waits for the operator's
`trust new host key`, then the VM is set up again.
`trust new host key`, then the VM is set up again (since M4.4, `vm ready`).
- Decided with the operator: the model judges toolchains (the harness can't
without it), preflight runs on the driver's model, gaps don't block. Turning
off unattended upgrades and the template's baseline are template choices.
@@ -434,9 +435,34 @@ past the usual fixes (D3).
- `loop.qnt` gained the block (`blocked_vm_runs_nothing`); it found that work in
progress was lost on a reimage, hence parking first. The walk requests
reimages and recovers them, reimaged or not.
- Next: an unreachable VM blocks through the same mechanism, unblocked by a slow
background retry once it is back (#137). Later, nisse-ops reimages a VM itself
(M6.2, M7.6); a changed address means editing `build_host` and restarting.
- Later, nisse-ops reimages a VM itself (M6.2, M7.6); a changed address means
editing `build_host` and restarting.
**M4.4** *(#137, with the operator).* **Done** (this PR). A VM that can't be
reached blocks through M4.3's mechanism; only the triggers and the way out
differ.
- Triggers: the check before a run, a run that loses the VM (it ends there, so
no model works against a missing VM: the toolchain step and a requested step
too), and preparing the workspace. The item stays ready, waiting on no one.
- Tried again every `tuning.build_vm.retry_s` (5 min) on the intake round, said
only in the operations log; once it answers, it must still answer
`settle_s` (3 min) later, so a VM that has just booted is left to finish.
Then the work a lost run left on it is parked (pushed), before any item can
set it aside, and it is unblocked and said once. `vm ready` clears it too;
the message warns that a reimage loses what isn't pushed.
- A changed host key, at the check or on a retry, blocks the VM as well, and is
cleared only by `vm ready`, which trusts the new key: `trust new host key` is
retired, and the message says it's a key problem and that it may be an
impersonation.
- Questions are answered while a VM is blocked, without it (L5: a question
changes nothing). Starting the agent with a VM down no longer fails.
- The walk breaks the VM's SSH, now or partway through a run, and checks that
no model works on a VM that is down. With `loop.qnt`, it found: work a lost
run left behind set aside by the next item there (hence parking first); a
question left unanswered behind a block (hence answering without the VM);
a lost VM swallowed while the toolchain step wrote its marker; and a remark
waiting to be answered that outlived its PR's merge, sending the item back
to wait for a merge that had happened. All fixed.
### Phase 5 — Memory
@@ -496,7 +522,7 @@ vision; one local model for every agent; several forges (#26); a room per projec
| #69 inline review comments | M2.1 |
| #70 watch stall | M2.1 |
| #77 conversational agent | M2.3 (questions), M2.7 |
| #137 block on an unreachable build VM | After M4.3, which built the block |
| #137 block on an unreachable build VM | M4.4 |
## 6. Implementation notes