Fit every model request to its context window

Every turn re-sent the whole transcript with nothing checking it against
the model's window. On a local model that fails silently: Ollama drops
the start of an over-long prompt, which is where the system prompt and
the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with
no error.

- The model catalog reads max_input_tokens and max_output_tokens from
  LiteLLM's model_info, and startup refuses a role on a model with no
  declared window
- BudgetedClient wraps both routes. It fits the canonical events before
  rendering: older tool results cut to head and tail, then the oldest
  turns dropped with a note, keeping calls with their results; a request
  that still cannot fit fails with ContextOverflow and is never sent.
  The output ceiling is the smaller of config and the model's limit
- Tokens are estimated from characters, pessimistically, and the
  estimate is corrected from reported prompt sizes, only ever upward
- The executor takes its scope from the log: the latest plan, which is
  the approved milestone plan, goes in its system prompt, and it sees
  only events after it
- Tool output keeps its head and, mostly, its tail, so a test runner's
  failure summary survives truncation
- tuning.context.keep_recent_turns and safety_margin are new knobs
- scripts/probe_context.py checks a declared window against what the
  server does; it confirmed llama3.2 cuts at the declared 4096
- The tuning guide covers the budget, declaring a local model's limits,
  and the probe

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/Tuning-Guide.md

@@ -114,8 +114,9 @@ to look at when deciding.
 
 ### `llm.max_output_tokens` — default 16384
 
-The ceiling on a single response. It is sent with every request, for every
-role.
+The ceiling on a single response, for every role. The ceiling actually sent
+is the smaller of this and the model's own `max_output_tokens` in LiteLLM, so
+a local model can be held to less without affecting the frontier roles.
 
 - **Why it exists:** a response that hits the ceiling is truncated
   mid-sentence, or worse, mid-tool-call, where it can still parse. The agent
@@ -128,9 +129,8 @@ role.
   prompt plus ceiling exceeds the context.
 - **Too low:** truncated plans and tool calls.
 - **Deciding:** executor turns are short, so 2–4k is plenty for a local
-  executor. Frontier planning wants the large default. This one knob covers
-  every role for now; a per-model setting is coming with the context budget
-  (see *Coming next*).
+  executor. Frontier planning wants the large default. Leave this high, and
+  set a local model's smaller ceiling as its `max_output_tokens` in LiteLLM.
 
 ### `tuning.executor.max_turns` — default 24
 
@@ -224,19 +224,105 @@ How long one `run_command` may run before its whole process group is killed.
 ### `tuning.tools.max_output_chars` — default 4000
 
 How much of a tool's output (command output, file contents, directory
-listings) the model sees. Anything past this is cut, with a note saying how
-much was dropped.
+listings) the model sees. Longer output keeps its first quarter and last three
+quarters, with a note in between saying how much was left out. The end gets
+the larger share because test runners and compilers print their summary of
+what failed last; the start keeps the exit code.
 
 - **This is a context knob.** Every tool result stays in the transcript and is
   re-sent every turn. 4,000 characters is about 1,000 tokens, so 20 turns of
   maximal output is about 20k tokens, which is already most of a local
   model's context.
 - **Too low:** the model doesn't see the part of the output that matters.
-- **Too high:** the transcript outgrows the context window sooner.
-- **A known weakness:** truncation keeps the *start* of the output, but test
-  runners put their failure summary at the *end*. A long, failing pytest run
-  can lose exactly the lines the model needed. Keeping the head and the tail
-  would fix this, and is a candidate for the context-budget work.
+- **Too high:** the transcript outgrows the context window sooner. The
+  context budget copes (see below), but by cutting older output harder.
+
+### `tuning.context` — fitting each request into the window
+
+Before any request is sent, the agent checks it against the model's context
+window and makes it fit. A server doesn't always refuse a prompt that's too
+long. Ollama silently drops its start, which is where the system prompt and
+the plan are, so an unfitted request can fail in a way nobody sees.
+
+The window is each model's `max_input_tokens` in LiteLLM (see *Declaring a
+local model's limits* below). A model in use without one stops the agent at
+startup. From the window come the output ceiling and a safety margin, and
+the rest is for the prompt. When the prompt is too big, three steps apply in
+order, stopping at the first that fits:
+
+1. **Older tool results are cut** to about 600 characters, head and tail.
+   The most recent turns keep their output whole.
+2. **The oldest turns are dropped,** tool calls together with their results.
+   A note at the start tells the model how many events were left out. The
+   log keeps everything; only the request is shortened.
+3. **If the system prompt and the recent turns still don't fit,** the
+   request is not sent. The milestone stops and asks you.
+
+Tokens are estimated from characters (pessimistically, at 3 per token) and
+the estimate is corrected from the prompt sizes the server reports. The
+correction only ever makes the estimate larger.
+
+The executor also sends less in the first place: the approved milestone plan
+goes in its system prompt, which is never cut, and it sees only what happened
+since that plan. The planning conversation and earlier milestones stay in the
+log.
+
+| Knob | Default | |
+|---|---|---|
+| `keep_recent_turns` | 6 | model turns that are never cut or dropped |
+| `safety_margin` | 0.1 | fraction of the window held back for estimation error |
+
+- **`keep_recent_turns` too low:** the model loses sight of what it just did
+  and repeats itself. **Too high:** less room to cut before step 3 gives up;
+  on a small window, overflows come sooner.
+- **`safety_margin` too low:** an underestimate reaches the server, and a
+  silently cutting server drops the start of the prompt. **Too high:** wasted
+  context. Lower it only once the correction from reported usage has settled,
+  which takes a few calls per model.
+- **Deciding:** if milestones stop with "cannot fit", look first at
+  `tools.max_output_chars` and the model's window, not at these.
+
+### Declaring a local model's limits
+
+LiteLLM knows commercial models' limits. For a local model, declare them in
+LiteLLM's config as the model's `model_info`:
+
+```yaml
+model_list:
+  - model_name: local-fast
+    litellm_params: {model: openai/qwen3-8b-awq, api_base: http://inference-host:8080/v1}
+    model_info:
+      max_input_tokens: 32768   # the server's per-request context
+      max_output_tokens: 4096   # the most one response may use
+```
+
+`max_input_tokens` is the *whole* window, prompt plus response. The agent
+takes the output ceiling out of it. That's how the commercial numbers work
+too: Claude's prompt and response together must fit in its 200k.
+
+The number to use is what the **server** allocates per request, not the
+model's advertised maximum:
+
+- **The model's ceiling:** `max_position_embeddings` in `config.json`, or
+  `<arch>.context_length` in a GGUF file. Some models reach further with RoPE
+  scaling; the model card says whether that needs switching on.
+- **llama.cpp / llama-swap:** the `-c` / `--ctx-size` flag, shown as `n_ctx`
+  in the load log and in `GET /props`. With `--parallel N` each request gets
+  `n_ctx / N`, so run the agent's models with `--parallel 1`.
+- **vLLM:** `--max-model-len`, reported by `GET /v1/models`.
+- **Ollama:** 4096 unless `OLLAMA_CONTEXT_LENGTH` or a Modelfile's `num_ctx`
+  says otherwise. `GET /api/ps` shows the loaded model's `context_length`;
+  `ollama show` shows only the trained maximum.
+
+Then check the declaration against the server:
+
+    uv run python scripts/probe_context.py local-fast
+
+It sends one prompt half again as long as the declared window and reports
+whether the server refused it, cut it (and where), or took it all. A cut
+below the declaration is the case to fix: prompts between the two would be
+silently cut. Measured on the Jetson's `llama3.2`: cut at 4096, matching its
+declaration.
 
 ---
 
@@ -259,7 +345,9 @@ change one thing at a time.
 2. **Context length and KV-cache precision** on the server. Take the largest
    context that fits beside the weights with 1–2 GB of headroom. Try an 8-bit
    KV cache before giving up context.
-3. **`llm.max_output_tokens`**, so the prompt keeps most of that context.
+3. **Declare the window** in LiteLLM and check it with the probe, then set
+   the model's `max_output_tokens` there so the prompt keeps most of the
+   window.
 4. **Real milestones.** Run representative work and look at the evidence:
    turns used, escalations, delegation outcomes, and time taken.
 5. **The loop knobs**: `failure_limit`, `max_turns`, `consult_budget`, the
@@ -278,9 +366,9 @@ run enough times that the result would survive a coin flip.
 The next changes in the 4090-readiness work add knobs, and this guide gains a
 section for each:
 
-- **A context budget.** Rendering a transcript that fits the model's context
-  window, with what to drop or summarize when it doesn't, and per-model output
-  ceilings.
+- **Compaction.** A model-written summary replacing old turns, instead of
+  dropping them. The log already has an event for it; it needs a summarizing
+  model and a policy for what is safe to summarize.
 - **Enforced tool calls.** An explicit `finish` tool, so the server can be told
   a tool call is required and hold the model to the contract, where today it
   can fall back to prose.