Fit every model request to its context window
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 55s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 55s
Every turn re-sent the whole transcript with nothing checking it against the model's window. On a local model that fails silently: Ollama drops the start of an over-long prompt, which is where the system prompt and the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with no error. - The model catalog reads max_input_tokens and max_output_tokens from LiteLLM's model_info, and startup refuses a role on a model with no declared window - BudgetedClient wraps both routes. It fits the canonical events before rendering: older tool results cut to head and tail, then the oldest turns dropped with a note, keeping calls with their results; a request that still cannot fit fails with ContextOverflow and is never sent. The output ceiling is the smaller of config and the model's limit - Tokens are estimated from characters, pessimistically, and the estimate is corrected from reported prompt sizes, only ever upward - The executor takes its scope from the log: the latest plan, which is the approved milestone plan, goes in its system prompt, and it sees only events after it - Tool output keeps its head and, mostly, its tail, so a test runner's failure summary survives truncation - tuning.context.keep_recent_turns and safety_margin are new knobs - scripts/probe_context.py checks a declared window against what the server does; it confirmed llama3.2 cuts at the declared 4096 - The tuning guide covers the budget, declaring a local model's limits, and the probe Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
31851b7396
commit
494715efa9
17 files changed
+1082
-45
No files matched your search
@@ -15,7 +15,7 @@ from dataclasses import dataclass
|
||||
from typing import Any, Literal
|
||||
|
||||
from nisse_dev.cancel import Cancel, skip_calls
|
||||
from nisse_dev.canonical import ToolCall, ToolResult
|
||||
from nisse_dev.canonical import PlanSubmitted, StoredEvent, ToolCall, ToolResult
|
||||
from nisse_dev.delegation import Delegation
|
||||
from nisse_dev.expert import ASK_EXPERT_TOOL, BudgetExhausted, ExpertDesk
|
||||
from nisse_dev.llm import LLMClient, LLMError, RoleMap
|
||||
@@ -24,7 +24,7 @@ from nisse_dev.tools import EXECUTOR_TOOLS, MUTATING_TOOLS, ToolOutcome, Workspa
|
||||
|
||||
EXECUTOR_SYSTEM = """You are nisse-dev's executor, doing mechanical work.
|
||||
|
||||
Follow the approved plan above exactly. It was written to be executed
|
||||
Follow the approved plan below exactly. It was written to be executed
|
||||
verbatim; do not redesign it, expand its scope, or skip steps.
|
||||
|
||||
Work by calling tools. Verify your work by running the project's tests.
|
||||
@@ -145,12 +145,34 @@ class LLMExecutor:
|
||||
except LLMError as exc:
|
||||
return ToolOutcome(f"the consultant did not answer: {exc}", success=False)
|
||||
|
||||
def _scope(self) -> tuple[str | None, int]:
|
||||
"""The approved plan, and where this milestone's events begin.
|
||||
|
||||
The most recent plan in the log is the approved milestone plan: no
|
||||
plan is submitted between approval and execution, or during review
|
||||
responses. The executor sees that plan and what came after it, and
|
||||
not the planning conversation or earlier milestones, which a small
|
||||
context window cannot afford and the plan already distils.
|
||||
"""
|
||||
latest: StoredEvent | None = None
|
||||
for stored in self._store.read_events():
|
||||
if isinstance(stored.event, PlanSubmitted):
|
||||
latest = stored
|
||||
if latest is None:
|
||||
return None, 0
|
||||
assert isinstance(latest.event, PlanSubmitted)
|
||||
return latest.event.plan, latest.id
|
||||
|
||||
def run(self, milestone: int) -> ExecutorOutcome:
|
||||
model = self._roles.resolve("executor")
|
||||
plan, after = self._scope()
|
||||
system = (
|
||||
f"{EXECUTOR_SYSTEM}\n\n{self._workspace.describe()}"
|
||||
f"\n\nYou are executing milestone {milestone}."
|
||||
)
|
||||
if plan is not None:
|
||||
# In the system prompt, so the context budget never drops it.
|
||||
system += f"\n\nThe approved plan:\n{plan}"
|
||||
consecutive_failures = 0
|
||||
did_work = False
|
||||
challenged = False
|
||||
@@ -162,7 +184,7 @@ class LLMExecutor:
|
||||
return ExecutorOutcome("stopped", f"before turn {turn + 1}")
|
||||
try:
|
||||
produced = self._client.complete(
|
||||
self._store.read_events(),
|
||||
self._store.read_events(after=after),
|
||||
tools=self._tools(),
|
||||
model=model,
|
||||
system=system,
|
||||
|
||||
Reference in new issue
Block a user