Fit every model request to its context window
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 55s
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 55s
Every turn re-sent the whole transcript with nothing checking it against the model's window. On a local model that fails silently: Ollama drops the start of an over-long prompt, which is where the system prompt and the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with no error. - The model catalog reads max_input_tokens and max_output_tokens from LiteLLM's model_info, and startup refuses a role on a model with no declared window - BudgetedClient wraps both routes. It fits the canonical events before rendering: older tool results cut to head and tail, then the oldest turns dropped with a note, keeping calls with their results; a request that still cannot fit fails with ContextOverflow and is never sent. The output ceiling is the smaller of config and the model's limit - Tokens are estimated from characters, pessimistically, and the estimate is corrected from reported prompt sizes, only ever upward - The executor takes its scope from the log: the latest plan, which is the approved milestone plan, goes in its system prompt, and it sees only events after it - Tool output keeps its head and, mostly, its tail, so a test runner's failure summary survives truncation - tuning.context.keep_recent_turns and safety_margin are new knobs - scripts/probe_context.py checks a declared window against what the server does; it confirmed llama3.2 cuts at the declared 4096 - The tuning guide covers the budget, declaring a local model's limits, and the probe Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
31851b7396
commit
494715efa9
17 files changed
+1082
-45
No files matched your search
@@ -3,7 +3,9 @@
|
||||
import pytest
|
||||
|
||||
from nisse_dev.canonical import AssistantText, StoredEvent, UserMessage
|
||||
from nisse_dev.catalog import ModelCatalog
|
||||
from nisse_dev.config import AgentConfig
|
||||
from nisse_dev.context import BudgetedClient
|
||||
from nisse_dev.llm import LiteLLMClient
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
@@ -25,3 +27,35 @@ def test_executor_model_completes_via_litellm(
|
||||
texts = [e.text for e in result if isinstance(e, AssistantText)]
|
||||
assert texts, f"no assistant text in {result!r}"
|
||||
assert "pong" in texts[0].lower()
|
||||
|
||||
|
||||
def test_declared_limits_reach_a_budgeted_request(
|
||||
litellm: LiteLLMClient, live_config: AgentConfig
|
||||
) -> None:
|
||||
"""The whole chain: LiteLLM's model_info, the catalog, the budget, and a
|
||||
real request through the proxy with the ceiling the budget chose."""
|
||||
catalog = ModelCatalog.discover(live_config.llm_base_url, live_config.llm_token)
|
||||
roles = live_config.roles.pinned()
|
||||
catalog.require_limits(roles) # every test role has a declared window
|
||||
|
||||
model = live_config.roles.executor
|
||||
facts = catalog.facts(model)
|
||||
client = BudgetedClient(
|
||||
litellm,
|
||||
lambda m: (
|
||||
catalog.facts(m).max_input_tokens,
|
||||
catalog.facts(m).max_output_tokens,
|
||||
),
|
||||
max_output_tokens=live_config.max_output_tokens,
|
||||
)
|
||||
assert client.budget_for(model).window == facts.max_input_tokens
|
||||
|
||||
events = [
|
||||
StoredEvent(
|
||||
id=1, ts="t", event=UserMessage(text="Reply with exactly: pong", source="t")
|
||||
)
|
||||
]
|
||||
produced = client.complete(events, None, model, system="Be terse.")
|
||||
texts = [e.text for e in produced if isinstance(e, AssistantText)]
|
||||
assert texts and "pong" in texts[0].lower()
|
||||
assert client.last_usage.get("input", 0) > 0
|
||||
Reference in new issue
Block a user