dev/nisse
2
1 Fork 0
Code Issues 23 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Enforce the tool-call contract, asking servers only where it works
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 56s
Details

Browse Source 
The executor ended a milestone on any reply without a tool call, and
nothing held a model to the offered tools. Measured on the 4070:
tool_choice "required" is enforced by vLLM (local-fast) and Anthropic,
but qwen3-14b (local-coder) under it cannot stop and writes until the
output ceiling, a known model bug. So enforcement is asked for per
model, and the loop holds the contract itself everywhere.

- The catalog reads supports_tool_choice from LiteLLM's model_info;
  BudgetedClient sends tool_choice only to models declared true, as
  "required" on the normalized route and {"type": "any"} on the
  Anthropic passthrough, and only when tools are offered
- The executor gains an explicit finish tool, the only way to end a
  milestone. Finishing with nothing changed is refused once; a prose
  reply gets one nudge, then escalates; a call to a tool not offered
  counts toward failure_limit, like a failed command; calls after
  finish are recorded as not run
- Clients record the model that actually answered. ModelFacts.answered
  recognises a proxy fallback, which once credited Claude's answers to
  a local model that could not call tools at all
- The context probe reports a fallback instead of trusting it; rerun,
  both local models now refuse over-long prompts, as they should
- The capability matrix checks every answer came from the model asked,
  that models call only offered tools, and that a declared
  supports_tool_choice is true
- Spec A15 and the tuning guide say how enforcement is requested,
  declared and backed up

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-09-24 22:54:49 +00:00
1 parent 389b71a5d1
commit b934c46714
16 files changed
+568 -86
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

nisse-dev/src/nisse_dev/executor.py
+86 -31
View File
@@ -27,20 +27,42 @@ EXECUTOR_SYSTEM = """You are nisse-dev's executor, doing mechanical work.
Follow the approved plan below exactly. It was written to be executed
verbatim; do not redesign it, expand its scope, or skip steps.
Work by calling tools. Verify your work by running the project's tests.
When every step of the plan is done and the tests pass, reply with a short
plain-text summary and no tool call — that ends the milestone."""
Every reply must be a tool call. Verify your work by running the project's
tests. When every step of the plan is done and the tests pass, call finish
with a short summary — that, and nothing else, ends the milestone."""
DEFAULT_MAX_TURNS = 24
DEFAULT_FAILURE_LIMIT = 2
# A turn with no tool call means "the plan is done". Believing that before
# any real work has happened would record an empty milestone as a success, so
# the claim is challenged once first.
_NUDGE = (
"\n\nYou have not made any change yet, so the plan cannot be complete. "
"Carry out the next step of the plan with a tool call, or run the tests "
"to show the work is already done."
FINISH_TOOL: dict[str, Any] = {
"type": "function",
"function": {
"name": "finish",
"description": (
"End the milestone: every step of the plan is done and the tests pass."
),
"parameters": {
"type": "object",
"properties": {
"summary": {"type": "string", "description": "What was done."}
},
"required": ["summary"],
},
},
}
# Finishing before any real work has happened would record an empty
# milestone as a success, so the claim is refused once first.
_NOT_YET = (
"You have not made any change yet, so the plan cannot be complete. Carry "
"out the next step of the plan, or run the tests to show the work is "
"already done, before calling finish."
)
# A reply in prose breaks the tool contract. A server that enforces
# tool_choice never allows one; on one that doesn't, the model is told once.
_PROSE_NUDGE = (
"\n\nYour last reply was prose. Every reply must be a tool call: carry "
"out the next step, or call finish if the plan is complete."
)
@@ -128,8 +150,8 @@ class LLMExecutor:
# A12: ask_expert exists in addition to the observable escalation
# triggers, never as a replacement for them.
if self._expert is None:
return EXECUTOR_TOOLS
return [*EXECUTOR_TOOLS, ASK_EXPERT_TOOL]
return [*EXECUTOR_TOOLS, FINISH_TOOL]
return [*EXECUTOR_TOOLS, ASK_EXPERT_TOOL, FINISH_TOOL]
def _consult(self, arguments: dict[str, Any]) -> ToolOutcome:
assert self._expert is not None
@@ -173,9 +195,12 @@ class LLMExecutor:
if plan is not None:
# In the system prompt, so the context budget never drops it.
system += f"\n\nThe approved plan:\n{plan}"
tools = self._tools()
offered = {tool["function"]["name"] for tool in tools}
consecutive_failures = 0
did_work = False
challenged = False
finish_refused = False
prose_nudged = False
for turn in range(self._max_turns):
# A4: `stop` takes effect at the next tool boundary. The start of
@@ -185,7 +210,7 @@ class LLMExecutor:
try:
produced = self._client.complete(
self._store.read_events(after=after),
tools=self._tools(),
tools=tools,
model=model,
system=system,
)
@@ -199,19 +224,12 @@ class LLMExecutor:
calls.append(event)
if not calls:
if did_work:
return ExecutorOutcome(
"completed", "executor reported the plan done"
)
if not challenged:
# Claiming completion without having touched anything is
# not evidence of completion; ask once before believing it.
challenged = True
system += _NUDGE
if not prose_nudged:
prose_nudged = True
system += _PROSE_NUDGE
continue
return ExecutorOutcome(
"escalate",
"executor claimed the milestone was done without making any change",
"escalate", "executor kept replying in prose instead of tools"
)
self._report(turn, calls)
@@ -219,11 +237,47 @@ class LLMExecutor:
if self._cancel.requested:
skip_calls(self._store, calls[index:])
return ExecutorOutcome("stopped", f"during turn {turn + 1}")
if call.tool in MUTATING_TOOLS:
did_work = True
if call.tool == "ask_expert":
if call.tool == "finish":
if not did_work and not finish_refused:
finish_refused = True
outcome = ToolOutcome(_NOT_YET, success=False)
elif not did_work:
return ExecutorOutcome(
"escalate",
"executor claimed the milestone was done without "
"making any change",
)
else:
self._store.append_event(
ToolResult(
call_id=call.call_id,
result="milestone finished",
success=True,
)
)
skip_calls(
self._store,
calls[index + 1 :],
"not run: finish had already ended the milestone",
)
summary = str(call.arguments.get("summary", "")).strip()
return ExecutorOutcome(
"completed", summary or "executor reported the plan done"
)
elif call.tool not in offered:
# A model inventing tools is not doing the work, and with
# no server enforcement nothing else would stop it, so it
# counts toward escalation like a failed command (A12).
outcome = ToolOutcome(
f"no such tool: {call.tool}; the tools are "
+ ", ".join(sorted(offered)),
success=False,
)
elif call.tool == "ask_expert":
outcome = self._consult(call.arguments)
else:
if call.tool in MUTATING_TOOLS:
did_work = True
outcome = self._workspace.dispatch(call.tool, call.arguments)
self._store.append_event(
ToolResult(
@@ -232,9 +286,10 @@ class LLMExecutor:
success=outcome.success,
)
)
# Only commands count toward escalation: a failed read is the
# model looking around, a failed command is work not working.
if call.tool == "run_command":
# Commands and invented tools count toward escalation: a
# failed read is the model looking around, a failed command
# is work not working.
if call.tool == "run_command" or call.tool not in offered:
consecutive_failures = (
0 if outcome.success else consecutive_failures + 1
)