Block projects on an unreachable build VM, and unblock when it's back #137

New Issue

2026-10-08T18:51:22Z

dev-agent commented

2026-10-08 18:51:22 +00:00

A build VM that can't be reached is probably fine once it's back (rebooting, a network blip, the host down), but today nisse-dev doesn't treat it as a VM-level problem:

At the check before a run (_readiness, HostUnavailable): only that item waits ("I can't reach the build VM… Reply to retry"). Other items on the same project are still picked, and each fails the same way.
During a run: each command returns "the build VM is unreachable" as an ordinary failed result (tools.py). The model keeps going and spends turns on it until the failure limit stops it. That leaves recovery to the model noticing, which P6 rules out.

Wanted

An unreachable VM blocks every project on it, using the blocking added in M4.3 for a VM the driver reports broken (request_reimage). Only the trigger differs.
Unreachable mid-run ends the run in code, so the model never works against a missing VM.
Keep trying in the background and unblock automatically once the VM answers. Retry slowly, and wait a while after the first answer before unblocking, so a VM that has just come back has time to finish booting before work starts on it.
Tell the operator once when the VM goes out of reach and once when it comes back, not on every retry.
A changed host key on return still goes through the operator's trust new host key.

Tests

An unreachable VM blocks the other items on its project.
Unreachable mid-run ends the run and blocks.
A successful probe clears the block, but only after the grace period.
The block survives a restart.

A build VM that can't be reached is probably fine once it's back (rebooting, a network blip, the host down), but today nisse-dev doesn't treat it as a VM-level problem: - **At the check before a run** (`_readiness`, `HostUnavailable`): only that item waits ("I can't reach the build VM… Reply to retry"). Other items on the same project are still picked, and each fails the same way. - **During a run:** each command returns "the build VM is unreachable" as an ordinary failed result (`tools.py`). The model keeps going and spends turns on it until the failure limit stops it. That leaves recovery to the model noticing, which P6 rules out. ## Wanted - An unreachable VM **blocks every project on it**, using the blocking added in M4.3 for a VM the driver reports broken (`request_reimage`). Only the trigger differs. - Unreachable mid-run **ends the run** in code, so the model never works against a missing VM. - **Keep trying in the background** and **unblock automatically** once the VM answers. Retry slowly, and wait a while after the first answer before unblocking, so a VM that has just come back has time to finish booting before work starts on it. - Tell the operator once when the VM goes out of reach and once when it comes back, not on every retry. - A changed host key on return still goes through the operator's `trust new host key`. ## Tests - An unreachable VM blocks the other items on its project. - Unreachable mid-run ends the run and blocks. - A successful probe clears the block, but only after the grace period. - The block survives a restart.

dev-agent referenced this issue

2026-10-08 19:27:28 +00:00

M4.3: a broken build VM blocks its projects until it's ready #138

dev-agent referenced this issue from a commit

2026-10-08 19:27:28 +00:00

M4.3: a broken build VM blocks its projects until it's ready

dev-agent referenced this issue from a commit

2026-10-08 21:06:06 +00:00

M4.4: an unreachable build VM blocks its projects until it's back (#137)

dev-agent referenced a pull request that will close this issue

2026-10-08 21:06:26 +00:00

M4.4: an unreachable build VM blocks its projects until it's back (#137) #139