Fit every model request to its context window

Every turn re-sent the whole transcript with nothing checking it against
the model's window. On a local model that fails silently: Ollama drops
the start of an over-long prompt, which is where the system prompt and
the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with
no error.

- The model catalog reads max_input_tokens and max_output_tokens from
  LiteLLM's model_info, and startup refuses a role on a model with no
  declared window
- BudgetedClient wraps both routes. It fits the canonical events before
  rendering: older tool results cut to head and tail, then the oldest
  turns dropped with a note, keeping calls with their results; a request
  that still cannot fit fails with ContextOverflow and is never sent.
  The output ceiling is the smaller of config and the model's limit
- Tokens are estimated from characters, pessimistically, and the
  estimate is corrected from reported prompt sizes, only ever upward
- The executor takes its scope from the log: the latest plan, which is
  the approved milestone plan, goes in its system prompt, and it sees
  only events after it
- Tool output keeps its head and, mostly, its tail, so a test runner's
  failure summary survives truncation
- tuning.context.keep_recent_turns and safety_margin are new knobs
- scripts/probe_context.py checks a declared window against what the
  server does; it confirmed llama3.2 cuts at the declared 4096
- The tuning guide covers the budget, declaring a local model's limits,
  and the probe

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

nisse-dev/src/nisse_dev/probe.py

@@ -0,0 +1,114 @@
+"""Check a model's declared context window against what its server does.
+
+LiteLLM's ``max_input_tokens`` for a local model is only a declaration; the
+real limit is whatever the model server allocates. This sends one prompt
+half again as long as the declared window and reads back what the server did
+with it:
+
+- **It refused.** Over-long prompts fail loudly, which is the safe behaviour.
+- **It cut the prompt.** The reported prompt size is where it cut, which is
+  the server's real window. Ollama does this, silently.
+- **It took the whole prompt.** The server allows more than is declared, so
+  the declaration is conservative.
+
+Run it with ``scripts/probe_context.py``. It spends one small and one large
+request on the model, so it refuses windows above a ceiling unless told
+otherwise: commercial models' limits are already known to LiteLLM.
+"""
+
+from dataclasses import dataclass
+
+import httpx
+
+# Of a probe's size, how much must come back for it to count as uncut. A
+# chat template adds a little, and a tokenizer counts a little differently.
+_UNCUT = 0.9
+# How close the server's cut must be to the declaration to count as a match.
+_MATCH = 0.02
+
+
+@dataclass(frozen=True)
+class ProbeResult:
+    verdict: str  # "matches", "declared_too_high", "declared_conservative", ...
+    message: str
+    server_window: int | None = None
+
+
+def interpret(
+    declared: int, sent: int, reported: int | None, error: str | None
+) -> ProbeResult:
+    """What one over-long probe says about the server's window.
+
+    ``sent`` is roughly how many tokens the probe held; ``reported`` is the
+    prompt size the server reported, if it answered; ``error`` its error
+    text, if it refused.
+    """
+    if error is not None:
+        return ProbeResult(
+            "refuses",
+            f"the server refused a ~{sent}-token prompt ({error[:200]}). Over-long "
+            "prompts fail loudly here, which is safe; the declared window of "
+            f"{declared} was not contradicted.",
+        )
+    assert reported is not None
+    if reported >= sent * _UNCUT:
+        return ProbeResult(
+            "declared_conservative",
+            f"the server took the whole ~{sent}-token prompt ({reported} reported), "
+            f"more than the declared {declared}. The declaration is safe; raise it "
+            "if you want the room.",
+            server_window=None,
+        )
+    if abs(reported - declared) <= declared * _MATCH:
+        return ProbeResult(
+            "matches",
+            f"the server cut the prompt at {reported} tokens, matching the declared "
+            f"{declared}.",
+            server_window=reported,
+        )
+    if reported < declared:
+        return ProbeResult(
+            "declared_too_high",
+            f"the server cut the prompt at {reported} tokens, but LiteLLM declares "
+            f"{declared}. Prompts between the two would be silently cut: set "
+            f"max_input_tokens to {reported}.",
+            server_window=reported,
+        )
+    return ProbeResult(
+        "declared_conservative",
+        f"the server cut the prompt at {reported} tokens, above the declared "
+        f"{declared}. The declaration is safe; it could be raised to {reported}.",
+        server_window=reported,
+    )
+
+
+def _prompt(words: int) -> str:
+    return " ".join(f"item{i}" for i in range(words)) + "\n\nReply with just: ok"
+
+
+def _send(http: httpx.Client, model: str, words: int) -> tuple[int | None, str | None]:
+    response = http.post(
+        "/chat/completions",
+        json={
+            "model": model,
+            "messages": [{"role": "user", "content": _prompt(words)}],
+            "max_tokens": 5,
+        },
+    )
+    if response.status_code != 200:
+        return None, f"{response.status_code}: {response.text}"
+    usage = response.json().get("usage") or {}
+    return int(usage.get("prompt_tokens") or 0), None
+
+
+def probe(http: httpx.Client, model: str, declared: int) -> ProbeResult:
+    """Measure how many tokens a word of the probe costs on this model, then
+    send a prompt half again as long as the declared window."""
+    calibration_words = 200
+    tokens, error = _send(http, model, calibration_words)
+    if error is not None or not tokens:
+        return ProbeResult("failed", f"a short calibration request failed: {error}")
+    per_word = tokens / calibration_words
+    sent = int(declared * 1.5)
+    reported, error = _send(http, model, int(sent / per_word))
+    return interpret(declared, sent, reported, error)