M4.1: toolchain preflight #133

2026-10-08T15:43:31Z

dev-agent commented

2026-10-08 15:43:31 +00:00

M4.1, toolchain preflight (D3). A model works out and sets up what a project needs on its build VM; code makes sure that happens, bounds it and checks the result. This is the shape we settled on in discussion.

The harness checks the build VM before each driver run on a project, in one round trip:

The baseline it relies on itself (git, base64, realpath, ssh-agent…). If any is missing, the item waits: it's a template problem.
The workspace clone. If it's missing, it's cloned again and the item hears so. The item resumes from what it pushed; unpushed work is lost, as D3 accepts.
A marker per project, ~/.nisse/<project>.ready.

No marker means preflight first. That covers a new project, a new VM, or a wiped one.

Who runs it: the item's driver model, in a bounded run with its own prompt. The prompt says:
- the project's toolchain is in scope;
- the template's basics, and tools that act on other systems, are not;
- never credentials.
Its tools: the execution tools, screened by the fast path and classifier as ever, plus toolchain_ready.
The harness verifies the report: it runs each reported version command itself, and only version-shaped ones. That includes commands read back from the marker, which the model's commands could edit. Then it writes the marker.
Gaps and an unfinished preflight become facts, and work goes on. The thread only hears about installs and gaps.
A rebuilt VM starts from the last toolchain.

Drift: later runs check the versions again. A change becomes a fact and a thread line, e.g. "rustc 1.80.1 → 1.82.0". The PR body says what the work was tested with: "Tested with: python3 Python 3.12.3, pytest pytest 9.1.1".

Host keys: a changed key is its own failure (HostKeyChanged), not just "unreachable". The item waits for the operator's trust new host key, which the model can't give. That drops the pin, and the VM is set up again.

Also:

a prompt line saying to install what the work needs on the build VM, say why, and put lasting needs in the project's own pins;
tuning.preflight (max_turns 20, command_timeout_s 900), in the Tuning Guide and the example config.

Tests

test_preflight.py: version-shaped commands (accepted and refused), drift, the report check, and on the real throwaway host: the marker round trip, the baseline probe, and versions read by the harness (a missing tool shows as "not found"; a non-version command is never run).
test_execution.py: a real changed host key gives HostKeyChanged, and only forgetting the pin clears it.
test_loop_preflight.py:
- preflight comes before planning, then is quiet;
- it runs once, then versions are checked, with drift said and the PR line written;
- only version commands are accepted;
- installs and gaps are said;
- an unfinished preflight is said and work goes on;
- a wiped VM is cloned again and set up;
- a missing baseline waits;
- a changed host key waits for the operator's word.
The property walk gains a VM-wipe step and ready_before_work: the driver never works on a project whose VM has no marker. It catches a mutant that skips preflight. parked_work_safe now accepts that a wipe loses unpushed work.
E2E: the session's first test gets a real preflight on the local model. Here it reported "python3 Python 3.12.3, pytest pytest 9.1.1", verified by the harness. The slice test asserts the PR's "Tested with" line.

Results, run locally:

Unit: 756 passed.
Integration: 18 passed.
Full e2e: 8 of 9 passed. The new-project test failed once: the local driver kept "restating" through ask_human after the work was agreed. It then passed 2 of 2 on reruns, with the preflight running in both, so I read it as the model drifting. I'll watch it in CI. If it recurs, the preflight's turns in the driver's context are the first thing to look at.

Deferred:

M4.2: recording every install for rebuilds.
M4.3: repair.
Template choices (unattended upgrades, the baseline): yours.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

**M4.1, toolchain preflight** (D3). A model works out and sets up what a project needs on its build VM; code makes sure that happens, bounds it and checks the result. This is the shape we settled on in discussion. **The harness checks the build VM before each driver run on a project**, in one round trip: - **The baseline** it relies on itself (`git`, `base64`, `realpath`, `ssh-agent`…). If any is missing, the item waits: it's a template problem. - **The workspace clone.** If it's missing, it's cloned again and the item hears so. The item resumes from what it pushed; unpushed work is lost, as D3 accepts. - **A marker per project**, `~/.nisse/<project>.ready`. **No marker means preflight first.** That covers a new project, a new VM, or a wiped one. - **Who runs it:** the item's driver model, in a bounded run with its own prompt. The prompt says: - the project's toolchain is in scope; - the template's basics, and tools that act on other systems, are not; - never credentials. - **Its tools:** the execution tools, screened by the fast path and classifier as ever, plus `toolchain_ready`. - **The harness verifies the report:** it runs each reported version command itself, and only version-shaped ones. That includes commands read back from the marker, which the model's commands could edit. Then it writes the marker. - **Gaps and an unfinished preflight** become facts, and work goes on. The thread only hears about installs and gaps. - **A rebuilt VM** starts from the last toolchain. **Drift:** later runs check the versions again. A change becomes a fact and a thread line, e.g. *"rustc 1.80.1 → 1.82.0"*. The PR body says what the work was tested with: *"Tested with: python3 Python 3.12.3, pytest pytest 9.1.1"*. **Host keys:** a changed key is its own failure (`HostKeyChanged`), not just "unreachable". The item waits for the operator's **`trust new host key`**, which the model can't give. That drops the pin, and the VM is set up again. **Also:** - a prompt line saying to install what the work needs on the build VM, say why, and put lasting needs in the project's own pins; - `tuning.preflight` (`max_turns` 20, `command_timeout_s` 900), in the Tuning Guide and the example config. **Tests** - **`test_preflight.py`:** version-shaped commands (accepted and refused), drift, the report check, and on the real throwaway host: the marker round trip, the baseline probe, and versions read by the harness (a missing tool shows as "not found"; a non-version command is never run). - **`test_execution.py`:** a real changed host key gives `HostKeyChanged`, and only forgetting the pin clears it. - **`test_loop_preflight.py`:** - preflight comes before planning, then is quiet; - it runs once, then versions are checked, with drift said and the PR line written; - only version commands are accepted; - installs and gaps are said; - an unfinished preflight is said and work goes on; - a wiped VM is cloned again and set up; - a missing baseline waits; - a changed host key waits for the operator's word. - **The property walk** gains a VM-wipe step and `ready_before_work`: the driver never works on a project whose VM has no marker. It catches a mutant that skips preflight. `parked_work_safe` now accepts that a wipe loses unpushed work. - **E2E:** the session's first test gets a real preflight on the local model. Here it reported *"python3 Python 3.12.3, pytest pytest 9.1.1"*, verified by the harness. The slice test asserts the PR's "Tested with" line. **Results, run locally:** - **Unit:** 756 passed. - **Integration:** 18 passed. - **Full e2e:** 8 of 9 passed. The new-project test failed once: the local driver kept "restating" through `ask_human` after the work was agreed. It then passed 2 of 2 on reruns, with the preflight running in both, so I read it as the model drifting. I'll watch it in CI. If it recurs, the preflight's turns in the driver's context are the first thing to look at. **Deferred:** - **M4.2:** recording every install for rebuilds. - **M4.3:** repair. - **Template choices** (unattended upgrades, the baseline): yours. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dev-agent added the e2e:full label 2026-10-08 15:43:31 +00:00

dev-agent added 1 commit 2026-10-08 15:43:31 +00:00

M4.1: toolchain preflight

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 2m59s

Details

CI / e2e (pull_request) Successful in 10m33s

Details

9968a0922a

A model works out and sets up what a project needs on its build VM;
code makes sure that happens, bounds it and checks the result (D3).

- Before each driver run on a project, one round trip checks the build
  VM: the harness's own baseline (missing: the item waits, a template
  problem), the workspace clone (missing: cloned again and said; the
  item resumes from what it pushed) and a per-project marker.
- No marker (a new project, a new or wiped VM): the item's driver model
  runs a bounded preflight first, with its own prompt, the execution
  tools (screened as ever) and toolchain_ready. The harness runs each
  reported version command itself (version-shaped commands only, even
  from the marker, which the model's commands could edit) and writes
  the marker. Gaps and an unfinished preflight are facts and work goes
  on; the thread hears only of installs and gaps. A rebuilt VM starts
  from the last toolchain.
- Later runs check the versions again; a change is a fact and a thread
  line. The PR body says what the work was tested with.
- A changed host key is its own failure (HostKeyChanged): the item
  waits for the operator's `trust new host key`, which drops the pin;
  then the VM is set up again.
- A prompt line: install what the work needs on the build VM, say why,
  and put lasting needs in the project's own pins.
- tuning.preflight: max_turns, command_timeout_s.
- The property walk gains a VM wipe and ready_before_work (checked
  against a mutant that skips preflight); parked_work_safe accepts that
  a wipe loses what wasn't pushed (D3).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dongar approved these changes

2026-10-08 16:29:37 +00:00

dongar merged commit b6eaceefa2 into main

2026-10-08 16:29:41 +00:00

dongar deleted branch preflight

2026-10-08 16:29:41 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#133