Fit every model request to its context window
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 55s

Every turn re-sent the whole transcript with nothing checking it against
the model's window. On a local model that fails silently: Ollama drops
the start of an over-long prompt, which is where the system prompt and
the plan are. Measured on the Jetson's llama3.2, cut at 4096 tokens with
no error.

- The model catalog reads max_input_tokens and max_output_tokens from
  LiteLLM's model_info, and startup refuses a role on a model with no
  declared window
- BudgetedClient wraps both routes. It fits the canonical events before
  rendering: older tool results cut to head and tail, then the oldest
  turns dropped with a note, keeping calls with their results; a request
  that still cannot fit fails with ContextOverflow and is never sent.
  The output ceiling is the smaller of config and the model's limit
- Tokens are estimated from characters, pessimistically, and the
  estimate is corrected from reported prompt sizes, only ever upward
- The executor takes its scope from the log: the latest plan, which is
  the approved milestone plan, goes in its system prompt, and it sees
  only events after it
- Tool output keeps its head and, mostly, its tail, so a test runner's
  failure summary survives truncation
- tuning.context.keep_recent_turns and safety_margin are new knobs
- scripts/probe_context.py checks a declared window against what the
  server does; it confirmed llama3.2 cuts at the declared 4096
- The tuning guide covers the budget, declaring a local model's limits,
  and the probe

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-09-24 00:26:30 +00:00
1 parent 31851b7396
commit 494715efa9
17 files changed
+1082 -45

No files matched your search

+6 -3
View File
@@ -15,8 +15,8 @@ project: demo
llm: # A5: LiteLLM is the only LLM destination
base_url: http://litellm.example:4000
token: ${NISSE_LLM_TOKEN}
# Ceiling on each response. Must fit inside a local model's context window
# alongside the prompt; see docs/Tuning-Guide.md.
# Ceiling on each response. The model's own max_output_tokens in LiteLLM
# caps it further, which is where to hold a local model to less.
max_output_tokens: 16384
roles: # A11: role -> LiteLLM model alias
@@ -54,4 +54,7 @@ tuning:
wall_clock_s: 900
tools:
command_timeout_s: 120 # one run_command before it is killed
max_output_chars: 4000 # tool output kept for the model; the rest is cut
max_output_chars: 4000 # tool output kept for the model, head and tail
context: # fitting each request into the model's window
keep_recent_turns: 6 # turns never cut or dropped
safety_margin: 0.1 # fraction of the window held back