Fit every model request to its context window #44

2026-09-24T00:26:31Z

dev-agent commented

2026-09-24 00:26:31 +00:00

The second 4090-readiness change: the context budget.

Why

Every turn re-sent the whole transcript, and nothing checked it against the model's window. On a local model that fails silently. Ollama drops the start of an over-long prompt, which is where the system prompt and the plan are. Measured on the Jetson's llama3.2: cut at 4,096 tokens, with no error.

What changed

Limits from LiteLLM. The model catalog reads max_input_tokens and max_output_tokens from model_info. Startup refuses a role on a model with no declared window, naming every such role at once.
BudgetedClient wraps both LLM routes and fits the canonical events before rendering. The log itself is never changed. Three steps, stopping at the first that fits:
1. Older tool results are cut to their head and tail.
2. The oldest turns are dropped, tool calls together with their results, and a note tells the model how many events were left out.
3. If it still doesn't fit, ContextOverflow: the request is never sent, and the milestone stops and asks the operator.
The output ceiling sent is the smaller of llm.max_output_tokens and the model's own limit.
Token estimates come from characters, pessimistically (3 per token). They're corrected from the prompt sizes the server reports, and the correction only ever raises them.
The executor sees only its milestone. The latest plan in the log is the approved milestone plan. It goes in the system prompt, which is never cut, and the executor sees only the events after it. That works for reruns and review responses with no orchestrator changes, and your reply after a stop is still included.
Tool output keeps the head and, mostly, the tail, so pytest's failure summary survives truncation. The split is fixed at a quarter and three quarters, not a knob; it didn't seem worth one.
New knobs: tuning.context.keep_recent_turns (6) and tuning.context.safety_margin (0.1).
scripts/probe_context.py ALIAS checks a declared window against what the server actually does: refuses, cuts (and where), or takes it all. Run against llama3.2: "the server cut the prompt at 4096 tokens, matching the declared 4096."
Guide: new sections on the budget, on declaring a local model's limits (llama.cpp, vLLM and Ollama specifics), and on the probe.

Tests

Head/tail truncation keeps pytest's summary.
Each budget step, in order. Surviving tool results always have their calls.
The ceiling is the smaller of the two limits. Usage correction only ever raises the estimate.
The catalog reads limits, and require_limits names every missing one.
The executor's scope, and an over-budget request stopping the milestone without being sent.
Context config validation. Probe verdicts, plus a fake Ollama that silently cuts.
Live: declared limits from the real LiteLLM reach a budgeted request through the proxy, and the live milestone test passes with the milestone-only scope.
scripts/check.sh passes: 366 unit tests, and the integration suite with 18 passed and 5 skipped.

Deferred

Model-written compaction, which summarizes old turns instead of dropping them.

🤖 Generated with Claude Code

The second 4090-readiness change: the context budget. ## Why Every turn re-sent the whole transcript, and nothing checked it against the model's window. On a local model that fails silently. Ollama drops the *start* of an over-long prompt, which is where the system prompt and the plan are. Measured on the Jetson's `llama3.2`: cut at 4,096 tokens, with no error. ## What changed - **Limits from LiteLLM.** The model catalog reads `max_input_tokens` and `max_output_tokens` from `model_info`. **Startup refuses a role on a model with no declared window**, naming every such role at once. - **`BudgetedClient`** wraps both LLM routes and fits the canonical events before rendering. The log itself is never changed. Three steps, stopping at the first that fits: 1. Older tool results are cut to their head and tail. 2. The oldest turns are dropped, tool calls together with their results, and a note tells the model how many events were left out. 3. If it still doesn't fit, `ContextOverflow`: the request is never sent, and the milestone stops and asks the operator. - **The output ceiling** sent is the smaller of `llm.max_output_tokens` and the model's own limit. - **Token estimates** come from characters, pessimistically (3 per token). They're corrected from the prompt sizes the server reports, and the correction only ever raises them. - **The executor sees only its milestone.** The latest plan in the log is the approved milestone plan. It goes in the system prompt, which is never cut, and the executor sees only the events after it. That works for reruns and review responses with no orchestrator changes, and your reply after a `stop` is still included. - **Tool output keeps the head and, mostly, the tail**, so pytest's failure summary survives truncation. The split is fixed at a quarter and three quarters, not a knob; it didn't seem worth one. - **New knobs:** `tuning.context.keep_recent_turns` (6) and `tuning.context.safety_margin` (0.1). - **`scripts/probe_context.py ALIAS`** checks a declared window against what the server actually does: refuses, cuts (and where), or takes it all. Run against `llama3.2`: *"the server cut the prompt at 4096 tokens, matching the declared 4096."* - **Guide:** new sections on the budget, on declaring a local model's limits (llama.cpp, vLLM and Ollama specifics), and on the probe. ## Tests - Head/tail truncation keeps pytest's summary. - Each budget step, in order. Surviving tool results always have their calls. - The ceiling is the smaller of the two limits. Usage correction only ever raises the estimate. - The catalog reads limits, and `require_limits` names every missing one. - The executor's scope, and an over-budget request stopping the milestone without being sent. - Context config validation. Probe verdicts, plus a fake Ollama that silently cuts. - **Live:** declared limits from the real LiteLLM reach a budgeted request through the proxy, and the live milestone test passes with the milestone-only scope. - `scripts/check.sh` passes: 366 unit tests, and the integration suite with 18 passed and 5 skipped. ## Deferred Model-written compaction, which summarizes old turns instead of dropping them. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dev-agent added 1 commit 2026-09-24 00:26:31 +00:00

Fit every model request to its context window

CI / checks (push) Skipped

CI / checks (pull_request) Successful in 55s

Details

494715efa9

Every turn re-sent the whole transcript with nothing checking it against
the model's window. On a local model that fails silently: Ollama drops
the start of an over-long prompt, which is where the system prompt and
the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with
no error.

- The model catalog reads max_input_tokens and max_output_tokens from
  LiteLLM's model_info, and startup refuses a role on a model with no
  declared window
- BudgetedClient wraps both routes. It fits the canonical events before
  rendering: older tool results cut to head and tail, then the oldest
  turns dropped with a note, keeping calls with their results; a request
  that still cannot fit fails with ContextOverflow and is never sent.
  The output ceiling is the smaller of config and the model's limit
- Tokens are estimated from characters, pessimistically, and the
  estimate is corrected from reported prompt sizes, only ever upward
- The executor takes its scope from the log: the latest plan, which is
  the approved milestone plan, goes in its system prompt, and it sees
  only events after it
- Tool output keeps its head and, mostly, its tail, so a test runner's
  failure summary survives truncation
- tuning.context.keep_recent_turns and safety_margin are new knobs
- scripts/probe_context.py checks a declared window against what the
  server does; it confirmed llama3.2 cuts at the declared 4096
- The tuning guide covers the budget, declaring a local model's limits,
  and the probe

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dongar commented

2026-09-24 13:37:33 +00:00

What is max_output_tokens used for? Is there any reason to actually require it to be set in LiteLLM?

dev-agent commented

2026-09-24 13:38:17 +00:00

It's used for one thing: the output ceiling sent with each request is the smaller of llm.max_output_tokens (our config, default 16384) and the model's max_output_tokens from LiteLLM. The budget then takes that ceiling out of the window before fitting the prompt. That's because the server has to hold room for the whole response, and vLLM, for one, rejects a request whose prompt plus max_tokens exceeds its context.

It isn't required. require_limits only checks max_input_tokens. Without a model limit, the ceiling is just our config value.

The catch is on small windows. On the Jetson's 4,096-token llama3.2, a 16,384-token ceiling leaves nothing for the prompt, so every request would fail as "cannot fit" and the milestone would stop. Today there are two ways out:

set the model's max_output_tokens in LiteLLM (what the guide recommends);
lower llm.max_output_tokens, which also shrinks the frontier planner's ceiling.

A better default would make it truly optional: when a model declares no output limit, cap the ceiling at a quarter of its window. llama3.2 would get 1,024 with nothing set in LiteLLM, and frontier models are unaffected (200k / 4 is well above 16k). Setting it in LiteLLM would then only be for tuning, not for making a model usable at all. I can add that to this PR. Want me to?

It's used for one thing: the output ceiling sent with each request is the smaller of `llm.max_output_tokens` (our config, default 16384) and the model's `max_output_tokens` from LiteLLM. The budget then takes that ceiling out of the window before fitting the prompt. That's because the server has to hold room for the whole response, and vLLM, for one, rejects a request whose prompt plus `max_tokens` exceeds its context. **It isn't required.** `require_limits` only checks `max_input_tokens`. Without a model limit, the ceiling is just our config value. The catch is on small windows. On the Jetson's 4,096-token `llama3.2`, a 16,384-token ceiling leaves nothing for the prompt, so every request would fail as "cannot fit" and the milestone would stop. Today there are two ways out: - set the model's `max_output_tokens` in LiteLLM (what the guide recommends); - lower `llm.max_output_tokens`, which also shrinks the frontier planner's ceiling. A better default would make it truly optional: when a model declares no output limit, cap the ceiling at a quarter of its window. `llama3.2` would get 1,024 with nothing set in LiteLLM, and frontier models are unaffected (200k / 4 is well above 16k). Setting it in LiteLLM would then only be for tuning, not for making a model usable at all. I can add that to this PR. Want me to?

dongar approved these changes

2026-09-24 13:41:11 +00:00

dongar merged commit 389b71a5d1 into main

2026-09-24 13:41:16 +00:00

dongar deleted branch context-budget

2026-09-24 13:41:16 +00:00

dev-agent referenced this pull request

2026-09-24 22:54:50 +00:00

Enforce the tool-call contract, asking servers only where it works #45

Sign in to join this conversation.