Add the delegation harness (A13) — step 7b #21

2026-09-08T00:14:43Z

dev-agent commented

2026-09-08 00:14:43 +00:00

The substance of step 7. A frontier model receives a scoped sub-task, works autonomously with full tool access, and hands back; the local loop reads the handback note and the resulting file state, never the reasoning.

Branch at start, abort discards it. This turned out to require committing each turn's work onto the branch — without commits, deleting a branch discards nothing, and my first version left the files sitting in the working tree. "Abort discards the work" was a comforting phrase rather than a fact until the test caught it. The caller's own in-flight changes are committed to their branch first, so an abort cannot destroy work nobody delegated.
Hard caps on turns and wall clock. A breach returns usable == False, so the caller must gate rather than resume on a half-edited tree, per A13's defined failure return.
Checkpointed as it happens, never batched at handback, so a crash leaves the branch at a known commit and a truthful transcript — the precondition for A3's resume rule.
Handback validated locally rather than trusted: blank fields or a non-boolean success flag are refused and the delegation continues.

Two failure modes only a real model found

1. Success claimed with nothing changed. The handback plus file state is the entire interface back, so a note describing work that does not exist is worse than no note. Success now requires at least one successful mutating tool call. Reporting failure still requires none — "this sub-task is impossible" is a legitimate finding.

2. The delegate invented /workspace. It wrote slugify.py there, ran its tests there, saw them pass, and handed back an accurate report — while the repo stayed empty and git saw nothing. The milestone would have published an empty PR with a confident description of work that was not in it. Writes outside the workspace are now refused with a message naming the real root, and both the executor and delegate prompts state where the workspace is rather than leaving the model to guess.

That second one is worth dwelling on: it is not a sandboxing question (the VM is the trust boundary, unchanged). It is that work outside the repo does not exist as far as git, the PR, and the operator are concerned, while the model has every reason to believe it succeeded.

18 new unit tests, 2 live (a real delegate writing and testing a slugify module, and a cap breach discarding its work). 216 unit + 15 integration green locally, and the live delegation passed 3/3 consecutive runs after the fixes.

One thing needs your hands: the delegate's stray writes created /workspace/ on this VM before the fix. I tried to inspect and remove it and both commands were blocked by the permission classifier, so I have left it alone — please delete it when convenient.

🤖 Generated with Claude Code

The substance of step 7. A frontier model receives a scoped sub-task, works autonomously with full tool access, and hands back; the local loop reads the handback note and the resulting file state, never the reasoning. - **Branch at start, abort discards it.** This turned out to require committing each turn's work onto the branch — without commits, deleting a branch discards *nothing*, and my first version left the files sitting in the working tree. "Abort discards the work" was a comforting phrase rather than a fact until the test caught it. The caller's own in-flight changes are committed to *their* branch first, so an abort cannot destroy work nobody delegated. - **Hard caps** on turns and wall clock. A breach returns `usable == False`, so the caller must gate rather than resume on a half-edited tree, per A13's defined failure return. - **Checkpointed as it happens**, never batched at handback, so a crash leaves the branch at a known commit and a truthful transcript — the precondition for A3's resume rule. - **Handback validated locally** rather than trusted: blank fields or a non-boolean success flag are refused and the delegation continues. ### Two failure modes only a real model found **1. Success claimed with nothing changed.** The handback plus file state is the entire interface back, so a note describing work that does not exist is worse than no note. Success now requires at least one successful mutating tool call. Reporting *failure* still requires none — "this sub-task is impossible" is a legitimate finding. **2. The delegate invented `/workspace`.** It wrote `slugify.py` there, ran its tests there, saw them pass, and handed back an accurate report — while the repo stayed empty and git saw nothing. The milestone would have published an empty PR with a confident description of work that was not in it. Writes outside the workspace are now refused with a message naming the real root, and both the executor and delegate prompts state where the workspace is rather than leaving the model to guess. That second one is worth dwelling on: it is not a sandboxing question (the VM is the trust boundary, unchanged). It is that **work outside the repo does not exist** as far as git, the PR, and the operator are concerned, while the model has every reason to believe it succeeded. 18 new unit tests, 2 live (a real delegate writing and testing a `slugify` module, and a cap breach discarding its work). 216 unit + 15 integration green locally, and the live delegation passed 3/3 consecutive runs after the fixes. **One thing needs your hands:** the delegate's stray writes created `/workspace/` on this VM before the fix. I tried to inspect and remove it and both commands were blocked by the permission classifier, so I have left it alone — please delete it when convenient. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-08 00:14:44 +00:00

Add the delegation harness (A13) — step 7b

CI / checks (push) Failing after 53s

Details

CI / checks (pull_request) Failing after 43s

Details

53ed11f9c4

A frontier model receives a scoped sub-task, works autonomously with
full tool access, and hands back; the local loop reads the handback
note and the file state, never the reasoning.

- Branch at delegation start, and abort discards it. That required
  committing each turn's work onto the branch: without commits,
  deleting a branch discards nothing, so "abort discards the work"
  was a comforting phrase rather than a fact. The caller's own
  in-flight changes are committed to their branch first, so an abort
  cannot destroy work nobody delegated
- Hard caps on turns and wall clock; a breach is a defined failure
  return, and its result is not usable, so the caller gates rather
  than resuming on a half-edited tree
- Every event is checkpointed as it arrives, so a crash mid-delegation
  leaves the branch at a known commit and a truthful transcript
- Handback is validated locally: blank fields or a non-boolean success
  flag are refused and the delegation continues rather than accepting
  a useless note

Two failure modes found by running it against a real model:

- A success claim with nothing changed was accepted. The handback plus
  file state is the entire interface back, so a note describing work
  that does not exist is worse than no note. Success now requires a
  successful mutating tool call; reporting failure still does not,
  since "this is impossible" is a legitimate finding
- The delegate invented the absolute path /workspace, wrote there, ran
  its tests there, and handed back truthfully — with the repo empty
  and git seeing nothing. Writes outside the workspace are now refused
  with a message naming the real root, and both prompts state where
  the workspace is instead of leaving the model to guess. This is not
  sandboxing: work outside the repo is invisible to git, so the
  milestone would publish an empty PR

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dev-agent added 1 commit 2026-09-08 00:18:34 +00:00

Stop CI running two suites at once, and retry rate limits

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 1m5s

Details

af9bdaac8d

CI failed on the delegation branch with a 429 from the proxy: token
limit 200000, 17399 remaining. Not a defect — the live delegation
tests are token-hungry, and every push to a PR branch fired two
identical suites concurrently, since the push and pull_request events
both matched.

- Push now only triggers on main. A PR branch is already covered by
  the pull_request event, so this halves both the wall time and the
  tokens spent per branch
- A 429 is retried once before being surfaced. Losing a ten-turn
  delegation to a transient limit wastes far more than waiting out
  the window, and retrying forever would hide a real ceiling. The
  sleep is injectable, so the existing 429 test no longer waits 20
  seconds for real

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dongar approved these changes

2026-09-08 00:47:04 +00:00

dongar merged commit d8d37b5441 into main

2026-09-08 00:47:08 +00:00

dongar deleted branch delegation

2026-09-08 00:47:08 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#21