dongarandClaude Opus 5.5
494715efa9
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 55s
Every turn re-sent the whole transcript with nothing checking it against the model's window. On a local model that fails silently: Ollama drops the start of an over-long prompt, which is where the system prompt and the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with no error. - The model catalog reads max_input_tokens and max_output_tokens from LiteLLM's model_info, and startup refuses a role on a model with no declared window - BudgetedClient wraps both routes. It fits the canonical events before rendering: older tool results cut to head and tail, then the oldest turns dropped with a note, keeping calls with their results; a request that still cannot fit fails with ContextOverflow and is never sent. The output ceiling is the smaller of config and the model's limit - Tokens are estimated from characters, pessimistically, and the estimate is corrected from reported prompt sizes, only ever upward - The executor takes its scope from the log: the latest plan, which is the approved milestone plan, goes in its system prompt, and it sees only events after it - Tool output keeps its head and, mostly, its tail, so a test runner's failure summary survives truncation - tuning.context.keep_recent_turns and safety_margin are new knobs - scripts/probe_context.py checks a declared window against what the server does; it confirmed llama3.2 cuts at the declared 4096 - The tuning guide covers the budget, declaring a local model's limits, and the probe Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
54 lines
1.9 KiB
Python
54 lines
1.9 KiB
Python
import json
|
|
|
|
import httpx
|
|
|
|
from nisse_dev.probe import interpret, probe
|
|
|
|
|
|
def test_a_cut_at_the_declared_window_matches() -> None:
|
|
result = interpret(declared=4096, sent=6144, reported=4096, error=None)
|
|
assert result.verdict == "matches"
|
|
assert result.server_window == 4096
|
|
|
|
|
|
def test_a_cut_below_the_declaration_says_what_to_set() -> None:
|
|
# The dangerous case: prompts between the two would be silently cut.
|
|
result = interpret(declared=8192, sent=12288, reported=4096, error=None)
|
|
assert result.verdict == "declared_too_high"
|
|
assert "set max_input_tokens to 4096" in result.message
|
|
|
|
|
|
def test_a_cut_above_the_declaration_is_conservative() -> None:
|
|
result = interpret(declared=4096, sent=6144, reported=5000, error=None)
|
|
assert result.verdict == "declared_conservative"
|
|
|
|
|
|
def test_an_uncut_prompt_means_the_server_allows_more() -> None:
|
|
result = interpret(declared=4096, sent=6144, reported=6100, error=None)
|
|
assert result.verdict == "declared_conservative"
|
|
assert result.server_window is None
|
|
|
|
|
|
def test_a_refusal_is_the_safe_behaviour() -> None:
|
|
result = interpret(declared=4096, sent=6144, reported=None, error="400: too long")
|
|
assert result.verdict == "refuses"
|
|
|
|
|
|
def test_probe_calibrates_then_sends_half_again_the_window() -> None:
|
|
"""Against a fake Ollama: 2 tokens a word, silently cut at 4096."""
|
|
sizes: list[int] = []
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
words = len(json.loads(request.content)["messages"][0]["content"].split())
|
|
sizes.append(words * 2)
|
|
reported = min(words * 2, 4096)
|
|
return httpx.Response(200, json={"usage": {"prompt_tokens": reported}})
|
|
|
|
with httpx.Client(
|
|
base_url="http://litellm", transport=httpx.MockTransport(handler)
|
|
) as http:
|
|
result = probe(http, "llama3.2", declared=4096)
|
|
|
|
assert result.verdict == "matches"
|
|
assert sizes[1] >= 4096 * 1.4 # the probe really exceeded the window
|