Roles, capability profiles and help on the new loop (M2.6) #113

2026-10-07T16:33:41Z

dev-agent commented

2026-10-07 16:33:41 +00:00

M2.6 of the Development Plan (L9), as planned.

What changes

Roles. roles.driver (unset: the planner's model) and roles.classifier (unset: the expert's) join helper and expert. The mapping is pinned when an item opens, and the driver and the go-ahead reader run on the item's own models, so a config change never splits one item across models. executor and planner remain only for the old loop, until M2.8.
Capability profiles. A profiles: section in config, keyed by model alias: tools, strengths, weaknesses. The example config is seeded for qwen3.8-27b, Haiku and Opus from PR #55's benchmark. Profiles are used in briefings only:
- the driver is told what its helper and expert are good at;
- the helper is told who will act on its handback, replacing "a materially weaker model";
- never a model's name.
Help for the driver:
- delegate(task): the existing harness runs the helper on a branch off the milestone's, under an intent. On a handback, its work comes back to the milestone as uncommitted changes. Otherwise it's discarded and the milestone is as it was.
- ask_expert(question): keeps its per-item budget.
Triggers. tuning.executor.failure_limit failed commands in a row (the driver's new stuck outcome), or the turn cap, mid-milestone, call a helper in automatically, once per milestone. If the helper fails too, the item waits on you as before. A delegation cut off by a crash is discarded at boot.
Evaluation capture. Delegations and ask_expert as before, and now the reader's verdicts as classifier pairs.

Found by the walk

The problem: a helper's work first came back as commits only the local clone had. With a review fix delegated and the PR merging meanwhile, the next milestone, cut fresh from main, stranded those commits: never pushed, in no PR.
The fix: the helper's work is returned uncommitted, which restores the invariant that harness branches never hold local-only commits. Parking, a later milestone and a follow-up PR then all carry it.
The property: the walk now also checks delegations_settle: no helper branch or delegation intent outlives a step.

Tests

test_loop_help.py (9 tests):
- merge-back and discard;
- delegating needs a milestone;
- the stuck trigger, once per milestone;
- the expert and its budget;
- pinning;
- briefings by profile, with no model names;
- boot discarding a cut-off delegation.
Config tests for roles and profiles.
The walk delegates, with a scripted helper that fails about 40% of the time.
No new e2e test, as planned: with every e2e role on one local model, delegation would be qwen helping qwen. This PR's CI runs the e2e suite for regressions.
Results: unit (721) and integration (21) tiers pass.

Deferred

The context-won't-fit trigger (L6).
The action classifier (M3.2).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

M2.6 of the Development Plan (L9), as planned. ## What changes - **Roles.** `roles.driver` (unset: the planner's model) and `roles.classifier` (unset: the expert's) join `helper` and `expert`. The mapping is pinned when an item opens, and the driver and the go-ahead reader run on the item's own models, so a config change never splits one item across models. `executor` and `planner` remain only for the old loop, until M2.8. - **Capability profiles.** A `profiles:` section in config, keyed by model alias: `tools`, `strengths`, `weaknesses`. The example config is seeded for qwen3.8-27b, Haiku and Opus from PR #55's benchmark. Profiles are used in briefings only: - the driver is told what its helper and expert are good at; - the helper is told who will act on its handback, replacing "a materially weaker model"; - never a model's name. - **Help for the driver:** - **`delegate(task)`:** the existing harness runs the helper on a branch off the milestone's, under an intent. On a handback, its work comes back to the milestone as **uncommitted** changes. Otherwise it's discarded and the milestone is as it was. - **`ask_expert(question)`:** keeps its per-item budget. - **Triggers.** `tuning.executor.failure_limit` failed commands in a row (the driver's new `stuck` outcome), or the turn cap, mid-milestone, call a helper in automatically, once per milestone. If the helper fails too, the item waits on you as before. A delegation cut off by a crash is discarded at boot. - **Evaluation capture.** Delegations and `ask_expert` as before, and now the reader's verdicts as `classifier` pairs. ## Found by the walk - **The problem:** a helper's work first came back as commits only the local clone had. With a review fix delegated and the PR merging meanwhile, the next milestone, cut fresh from `main`, stranded those commits: never pushed, in no PR. - **The fix:** the helper's work is returned uncommitted, which restores the invariant that harness branches never hold local-only commits. Parking, a later milestone and a follow-up PR then all carry it. - **The property:** the walk now also checks `delegations_settle`: no helper branch or delegation intent outlives a step. ## Tests - **`test_loop_help.py` (9 tests):** - merge-back and discard; - delegating needs a milestone; - the stuck trigger, once per milestone; - the expert and its budget; - pinning; - briefings by profile, with no model names; - boot discarding a cut-off delegation. - **Config tests** for roles and profiles. - **The walk** delegates, with a scripted helper that fails about 40% of the time. - **No new e2e test,** as planned: with every e2e role on one local model, delegation would be qwen helping qwen. This PR's CI runs the e2e suite for regressions. - **Results:** unit (721) and integration (21) tiers pass. ## Deferred - The context-won't-fit trigger (L6). - The action classifier (M3.2). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dev-agent added 1 commit 2026-10-07 16:33:41 +00:00

Roles, capability profiles and help on the new loop (M2.6)

CI / checks (push) Skipped

CI / checks (pull_request) Failing after 5m23s

Details

CI / e2e (pull_request) Canceled after 21m42s

Details

59fd00ce8c

- roles.driver (unset: the planner's model) and roles.classifier (unset:
  the expert's); the mapping is pinned when an item opens, and the driver
  and the go-ahead reader run on the item's own (L9).
- profiles: in configuration (tools, strengths, weaknesses), used in
  briefings: the driver's instructions say what its helper and expert are
  good at, the helper's brief who reads its handback; never a model name.
- delegate(task) and ask_expert(question) for the driver. A helper works
  on a branch off the milestone's; on a handback its work comes back as
  uncommitted changes, otherwise it is discarded, so harness branches
  never hold commits only the clone has. ask_expert keeps its budget.
- Triggers: failure_limit failed commands in a row (Driver's new "stuck"
  outcome) or the turn cap, mid-milestone, call a helper in once per
  milestone; if it fails too, the item waits on the operator. A
  delegation cut off by a crash is discarded at boot.
- Evaluation capture: delegations and ask_expert as before, and the
  reader's verdicts as classifier pairs.

Tests: help tests (merge-back, discard, needs a milestone, the stuck
trigger and its once-per-milestone limit, the expert and its budget,
pinning, briefings by profile, boot discarding a cut-off delegation),
config tests for roles and profiles, and the walk delegating with a
scripted helper that sometimes fails, with a delegations_settle property.
The walk found helper commits stranded on a merged PR's branch, fixed by
returning a helper's work uncommitted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dev-agent added 1 commit 2026-10-07 16:40:50 +00:00

CI: run the e2e job only after the checks pass

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 5m48s

Details

CI / e2e (pull_request) Successful in 21m52s

Details

ba001a9cec

A PR that's already red doesn't spend ~25 minutes of the RTX 4090.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPQ14L3LikyefCMFBqEbKE

dongar approved these changes

2026-10-07 17:40:30 +00:00

dev-agent merged commit 5967d022b6 into main

2026-10-07 17:40:46 +00:00

dev-agent deleted branch roles

2026-10-07 17:40:46 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#113