Fit every model request to its context window
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 55s

Every turn re-sent the whole transcript with nothing checking it against
the model's window. On a local model that fails silently: Ollama drops
the start of an over-long prompt, which is where the system prompt and
the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with
no error.

- The model catalog reads max_input_tokens and max_output_tokens from
  LiteLLM's model_info, and startup refuses a role on a model with no
  declared window
- BudgetedClient wraps both routes. It fits the canonical events before
  rendering: older tool results cut to head and tail, then the oldest
  turns dropped with a note, keeping calls with their results; a request
  that still cannot fit fails with ContextOverflow and is never sent.
  The output ceiling is the smaller of config and the model's limit
- Tokens are estimated from characters, pessimistically, and the
  estimate is corrected from reported prompt sizes, only ever upward
- The executor takes its scope from the log: the latest plan, which is
  the approved milestone plan, goes in its system prompt, and it sees
  only events after it
- Tool output keeps its head and, mostly, its tail, so a test runner's
  failure summary survives truncation
- tuning.context.keep_recent_turns and safety_margin are new knobs
- scripts/probe_context.py checks a declared window against what the
  server does; it confirmed llama3.2 cuts at the declared 4096
- The tuning guide covers the budget, declaring a local model's limits,
  and the probe

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-09-24 00:26:30 +00:00
1 parent 31851b7396
commit 494715efa9
17 files changed
+1082 -45

No files matched your search

+28 -4
View File
@@ -15,6 +15,7 @@ from nisse_dev.approvals import PlanBoard
from nisse_dev.cancel import Cancel
from nisse_dev.catalog import ModelCatalog, select_client
from nisse_dev.config import AgentConfig
from nisse_dev.context import BudgetedClient
from nisse_dev.control import Orchestrator
from nisse_dev.delegation import Caps, Delegation
from nisse_dev.executor import LLMExecutor
@@ -50,6 +51,10 @@ def discover_models(config: AgentConfig, oplog: OpLog) -> ModelCatalog:
"""
catalog = ModelCatalog.discover(config.llm_base_url, config.llm_token)
catalog.validate(config.roles.pinned())
# Every role's context window must be known before a request is built
# against it (A10): a guess that is too large reaches a server that may
# silently cut the prompt.
catalog.require_limits(config.roles.pinned())
for role in TOOL_DRIVING_ROLES:
facts = catalog.facts(config.roles.resolve(role))
if facts.supports_tools is False:
@@ -156,14 +161,33 @@ def build_agent(
max_output_chars=tuning.tools.max_output_chars,
)
normalized = LiteLLMClient(
config.llm_base_url, config.llm_token, max_tokens=config.max_output_tokens
def limits_for(model: str) -> tuple[int, int | None]:
facts = catalog.facts(model)
assert facts.max_input_tokens is not None # require_limits checked
return facts.max_input_tokens, facts.max_output_tokens
def budgeted(client: object) -> BudgetedClient:
# Every request, on either route, is fitted to its model's window.
return BudgetedClient(
client,
limits_for,
max_output_tokens=config.max_output_tokens,
keep_recent_turns=tuning.context.keep_recent_turns,
safety_margin=tuning.context.safety_margin,
)
normalized = budgeted(
LiteLLMClient(
config.llm_base_url, config.llm_token, max_tokens=config.max_output_tokens
)
)
# Delegations prefer the vendor-native route; fidelity is cheap
# elsewhere (A5). Which route a given model gets is decided by
# select_client, not by where the caller happens to sit.
passthrough = AnthropicPassthroughClient(
config.llm_base_url, config.llm_token, max_tokens=config.max_output_tokens
passthrough = budgeted(
AnthropicPassthroughClient(
config.llm_base_url, config.llm_token, max_tokens=config.max_output_tokens
)
)
transport = MatrixTransport(config.matrix)