Fit every model request to its context window
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 55s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 55s
Every turn re-sent the whole transcript with nothing checking it against the model's window. On a local model that fails silently: Ollama drops the start of an over-long prompt, which is where the system prompt and the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with no error. - The model catalog reads max_input_tokens and max_output_tokens from LiteLLM's model_info, and startup refuses a role on a model with no declared window - BudgetedClient wraps both routes. It fits the canonical events before rendering: older tool results cut to head and tail, then the oldest turns dropped with a note, keeping calls with their results; a request that still cannot fit fails with ContextOverflow and is never sent. The output ceiling is the smaller of config and the model's limit - Tokens are estimated from characters, pessimistically, and the estimate is corrected from reported prompt sizes, only ever upward - The executor takes its scope from the log: the latest plan, which is the approved milestone plan, goes in its system prompt, and it sees only events after it - Tool output keeps its head and, mostly, its tail, so a test runner's failure summary survives truncation - tuning.context.keep_recent_turns and safety_margin are new knobs - scripts/probe_context.py checks a declared window against what the server does; it confirmed llama3.2 cuts at the declared 4096 - The tuning guide covers the budget, declaring a local model's limits, and the probe Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
31851b7396
commit
494715efa9
17 files changed
+1082
-45
No files matched your search
@@ -86,12 +86,19 @@ class ToolsTuning:
|
||||
max_output_chars: int = 4000
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ContextTuning:
|
||||
keep_recent_turns: int = 6
|
||||
safety_margin: float = 0.1
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Tuning:
|
||||
executor: ExecutorTuning = field(default_factory=ExecutorTuning)
|
||||
expert: ExpertTuning = field(default_factory=ExpertTuning)
|
||||
delegation: DelegationTuning = field(default_factory=DelegationTuning)
|
||||
tools: ToolsTuning = field(default_factory=ToolsTuning)
|
||||
context: ContextTuning = field(default_factory=ContextTuning)
|
||||
|
||||
|
||||
_TUNING_SECTIONS: dict[str, type] = {
|
||||
@@ -128,6 +135,9 @@ def _parse_tuning(raw: Any, errors: list[str]) -> Tuning:
|
||||
continue
|
||||
values[key] = number
|
||||
sections[name] = cls(**values)
|
||||
margin = sections.get("context", ContextTuning()).safety_margin
|
||||
if margin >= 1:
|
||||
errors.append("tuning.context.safety_margin: must be less than 1")
|
||||
return Tuning(**sections)
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user