Count only mutating tools as evidence of work
The previous fix challenged a completion claim made before any tool call, but treated every tool as work — so a model that ran one list_dir and then declared the milestone done was believed, and CI caught the same empty milestone again. Reading a file or listing a directory is the model orienting itself. Only write_file and run_command now count, so a completion claim must be backed by something that changed or verified the tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8f365263c4
commit
886cb80583
2 files changed
+26
-3
No files matched your search
@@ -30,8 +30,13 @@ plain-text summary and no tool call — that ends the milestone."""
|
||||
DEFAULT_MAX_TURNS = 24
|
||||
DEFAULT_FAILURE_LIMIT = 2
|
||||
|
||||
# Tools that change or verify something. Reading a file or listing a
|
||||
# directory is the model looking around, not doing the work, so neither
|
||||
# counts as evidence that a milestone was carried out.
|
||||
MUTATING_TOOLS = frozenset({"write_file", "run_command"})
|
||||
|
||||
# A turn with no tool call means "the plan is done". Believing that before
|
||||
# any tool has been called would record an empty milestone as a success, so
|
||||
# any real work has happened would record an empty milestone as a success, so
|
||||
# the claim is challenged once first.
|
||||
_NUDGE = (
|
||||
"\n\nYou have not made any change yet, so the plan cannot be complete. "
|
||||
@@ -105,7 +110,8 @@ class LLMExecutor:
|
||||
)
|
||||
|
||||
for call in calls:
|
||||
did_work = True
|
||||
if call.tool in MUTATING_TOOLS:
|
||||
did_work = True
|
||||
outcome = self._workspace.dispatch(call.tool, call.arguments)
|
||||
self._store.append_event(
|
||||
ToolResult(
|
||||
|
||||
Reference in new issue
Block a user