Add LLM-backed planning agent (A4/A11/A13) — step 5a #11

Merged

dongar merged 3 commits from llm-planning into main 2026-09-07 16:15:47 +00:00

Conversation 1 Commits 3 Files Changed 8

+523 -5

dev-agent commented 2026-09-07 14:47:30 +00:00

Collaborator

Copy link

Copy source

First half of build-order step 5: the stub agent's seam now has a real LLM behind it. The executor tool loop (5b) follows.

• planner.py — LLMPlanningAgent offers the model exactly two tools, submit_plan and ask_human, so A4's gating is structural at the tool layer rather than enforced by prompt discipline. There is no proceed tool, so the model has no path to advance past a gate. The system prompt demands explicit, low-inference plans, because A13 says a materially weaker model executes them.
• Truthful transcript — model output is appended to the canonical log before being interpreted, so a crash or a rejected tool call still leaves an honest record.
• Bad turns gate, never guess — prose instead of a tool call, malformed arguments, or a failed request retries once with a nudge, then raises PlanningError; the orchestrator moves to AWAIT_INPUT and asks. It never invents a plan or loops.
• Orchestrator now appends every operator message, plan_submitted, and gate_opened to the transcript. Rejection feedback therefore reaches the next planning turn as conversation rather than as an out-of-band parameter — a test asserts the re-plan sees "too vague, name the files". The A11 role mapping is pinned at task start into durable state.
• llm.py hardening — non-JSON or non-object tool arguments now raise LLMError instead of escaping as a raw JSONDecodeError.

Finding: llama3.2 does not hold the tool-call contract

I ran the live planning turn against the local model and it failed. Probing the raw response shows why — it does emit a tool call, but with truncated arguments:

"arguments": "{\"type\": \"application\", \"content\": \"{\"}"

so the "plan" is the single character {; other attempts returned no tool call at all. This is exactly what A15 anticipates ("constrained decoding for tool-call JSON", "reduced tool schema for the executor role") and is consistent with A12's phase policy, which escalates planning to a frontier model unconditionally — planning was never meant to run locally.

Consequences in this PR: the client hardening above, and the live planning test is opt-in on NISSE_LIVE_PLANNER_MODEL (documented in .env.example) rather than on the usual NISSE_LLM_BASE_URL sentinel. That is a deliberate skip awaiting a frontier route, not the accidental kind we just fixed — the LLM and Matrix live tests keep their sentinels and still run.

What I'd like from you eventually: a LiteLLM route to a frontier model plus a virtual key scoped to reach it. Then set NISSE_LIVE_PLANNER_MODEL and this test proves the contract end to end. Not blocking 5b, which targets the executor role where llama3.2 belongs.

18 new unit tests (tool-call interpretation, retry/nudge, malformed arguments, no-proceed-tool invariant, transcript shape, role pinning, failure gating, feedback round-trip). 90 unit + 7 integration green locally, 1 deliberate skip.

🤖 Generated with Claude Code

First half of build-order step 5: the stub agent's seam now has a real LLM behind it. The executor tool loop (5b) follows. - **`planner.py`** — `LLMPlanningAgent` offers the model exactly two tools, `submit_plan` and `ask_human`, so A4's gating is structural at the tool layer rather than enforced by prompt discipline. There is no proceed tool, so the model has no path to advance past a gate. The system prompt demands explicit, low-inference plans, because A13 says a materially weaker model executes them. - **Truthful transcript** — model output is appended to the canonical log *before* being interpreted, so a crash or a rejected tool call still leaves an honest record. - **Bad turns gate, never guess** — prose instead of a tool call, malformed arguments, or a failed request retries once with a nudge, then raises `PlanningError`; the orchestrator moves to `AWAIT_INPUT` and asks. It never invents a plan or loops. - **Orchestrator** now appends every operator message, `plan_submitted`, and `gate_opened` to the transcript. Rejection feedback therefore reaches the next planning turn *as conversation* rather than as an out-of-band parameter — a test asserts the re-plan sees "too vague, name the files". The A11 role mapping is pinned at task start into durable state. - **`llm.py` hardening** — non-JSON or non-object tool arguments now raise `LLMError` instead of escaping as a raw `JSONDecodeError`. ### Finding: llama3.2 does not hold the tool-call contract I ran the live planning turn against the local model and it failed. Probing the raw response shows why — it does emit a tool call, but with truncated arguments: ``` "arguments": "{\"type\": \"application\", \"content\": \"{\"}" ``` so the "plan" is the single character `{`; other attempts returned no tool call at all. This is exactly what A15 anticipates ("constrained decoding for tool-call JSON", "reduced tool schema for the executor role") and is consistent with A12's phase policy, which escalates planning to a frontier model unconditionally — planning was never meant to run locally. Consequences in this PR: the client hardening above, and the live planning test is **opt-in on `NISSE_LIVE_PLANNER_MODEL`** (documented in `.env.example`) rather than on the usual `NISSE_LLM_BASE_URL` sentinel. That is a deliberate skip awaiting a frontier route, not the accidental kind we just fixed — the LLM and Matrix live tests keep their sentinels and still run. **What I'd like from you eventually**: a LiteLLM route to a frontier model plus a virtual key scoped to reach it. Then set `NISSE_LIVE_PLANNER_MODEL` and this test proves the contract end to end. Not blocking 5b, which targets the executor role where llama3.2 belongs. 18 new unit tests (tool-call interpretation, retry/nudge, malformed arguments, no-proceed-tool invariant, transcript shape, role pinning, failure gating, feedback round-trip). 90 unit + 7 integration green locally, 1 deliberate skip. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-07 14:47:31 +00:00

Add LLM-backed planning agent (A4/A11/A13) — step 5a ... cf85648719

- planner.py: LLMPlanningAgent offers exactly two tools, submit_plan
  and ask_human, so gating stays structural at the tool layer; no
  proceed tool exists. System prompt requires explicit, low-inference
  plans because a weaker model executes them (A13)
- Model output is appended to the canonical log as it arrives, before
  being interpreted, so a crash mid-planning leaves a truthful
  transcript
- A bad turn (prose instead of a tool call, malformed arguments, or a
  failed call) retries once, then raises PlanningError; the
  orchestrator gates to AWAIT_INPUT rather than inventing a plan
- Orchestrator: appends every operator message, plan_submitted, and
  gate_opened to the transcript, so re-planning sees rejection
  feedback as conversation rather than out-of-band parameters; pins
  the A11 role mapping at task start
- llm.py: non-JSON or non-object tool arguments now raise LLMError
  instead of escaping as JSONDecodeError — observed from llama3.2,
  which emits truncated argument strings
- Live planning test is opt-in on NISSE_LIVE_PLANNER_MODEL: planning
  escalates unconditionally (A12) and the local model does not hold
  the tool-call contract, so it awaits a frontier route

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dev-agent added 1 commit 2026-09-07 16:01:21 +00:00

Enable the live planning test against claude-haiku-4-5 ... 6c17f2bf57

The virtual key now reaches a Haiku-class route, which holds the
tool-call contract llama3.2 could not: a real planning turn returns a
well-formed submit_plan call with an explicit, low-inference plan.

Wires NISSE_LIVE_PLANNER_MODEL through the CI workflow (add the
matching Actions secret to run it there) and records the finding in
.env.example.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dev-agent added 1 commit 2026-09-07 16:02:58 +00:00

Trigger CI to pick up the NISSE_LIVE_PLANNER_MODEL secret ... fb35ca774d

Empty commit: Gitea 1.27 exposes no rerun API endpoint, and Actions
secrets are read at run start, so a new run is needed to exercise the
live planning test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dongar approved these changes 2026-09-07 16:15:44 +00:00

dongar merged commit 47b2bdfdbd into main 2026-09-07 16:15:47 +00:00

dongar deleted branch llm-planning 2026-09-07 16:15:47 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

dongar

Labels

Clear labels

bug

Reproduce first; the fix goes ahead after a grace period unless the operator replies

design

Agree the design (ARCHITECTURE.md + build plan PR) before any code

discuss

Talk it through with the operator before restating

e2e:full

CI runs all 48 property walks and the whole end-to-end suite on this PR

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

dev-agent dongar (Don Garrett) nisse-review

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/nisse#11