Compaction: the model's notes replace the middle of a long run (#158)
CI / checks (push) Skipped
CI / checks (pull_request) Successful in 1m57s
CI / e2e (pull_request) Successful in 4m57s

Before each driver call, over compact_at (0.6) of the prompt budget, the
same model is asked for notes over the very request it was about to get,
plus one message, so the server's cache covers all but the notes. The notes,
with code's list of the calls they replace, become a compaction event that
renders in place of its range, keeping word for word what started the work
and the last few turns. Rare, large steps keep the prompt's start unchanged
between them. A later compaction folds in the earlier; one from an earlier
run's view is hidden.

Failed notes are logged and fitting takes over. Fitting now applies
compactions before it measures, records what it cuts (Trim) and logs each
cut as context.trimmed. Wired into both agents' drivers and the benchmark;
nisse-dev's helper loop is a follow-up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
This commit is contained in:
dongarandClaude Opus 5.5 committed 2026-10-17 22:51:36 +00:00
1 parent fa8d93cf41
commit cb638fe687
20 files changed
+952 -44

No files matched your search

+35 -2
View File
@@ -283,9 +283,39 @@ what failed last; the start keeps the exit code.
- **Too high:** the transcript outgrows the context window sooner. The
context budget copes (see below), but by cutting older output harder.
### `tuning.context` — fitting each request into the window
### `tuning.context` — compaction, and fitting each request into the window
Before any request is sent, the agent checks it against the model's context
**Compaction comes first.** A driver run's request grows with every tool
result. Once it passes `compact_at` of the prompt budget, the model is asked
for notes on its run so far: findings with the code they rest on, what each
file told it, what is left to do. The notes replace everything between what
started the work (the brief, or the conversation) and the last few turns,
which stay word for word. The model server has the request cached already,
so the notes cost only their own output. Because it happens rarely and drops
the request well below the limit, the start of the prompt stays the same for
many calls, and the server keeps reusing its cache.
Logged as `context.compacted` (tokens before and after, seconds). Notes that
fail, come back empty or leave the request still too big are logged as
`context.compaction_failed`; the run goes on and fitting (below) cuts
instead, and compaction isn't tried again for `compact_keep_turns` turns.
| Knob | Default | |
|---|---|---|
| `compact_at` | 0.6 | fraction of the prompt budget that sets compaction off; 0 turns it off |
| `compact_to` | 0.25 | what the kept recent turns may take, at most |
| `compact_keep_turns` | 3 | recent turns kept word for word, at most |
- **`compact_at` too high:** little room left for the notes call and a big
result that lands in one turn, so fitting cuts more often. **Too low:**
compactions come often, and each one loses detail.
- **`compact_to` / `compact_keep_turns` too low:** the model loses sight of
what it just did. **Too high:** the notes don't bring the request far
enough down, and compactions come sooner.
- **Deciding:** look for `context.trimmed` in the log. After compaction it
should be rare; each one means the model lost something it had read.
**Fitting is the last guard.** Before any request is sent, the agent checks it against the model's context
window and makes it fit. A server doesn't always refuse a prompt that's too
long. Ollama silently drops its start, which is where the system prompt and
the plan are, so an unfitted request can fail in a way nobody sees.
@@ -304,6 +334,9 @@ order, stopping at the first that fits:
3. **If the system prompt and the recent turns still don't fit,** the
request is not sent. The milestone stops and asks you.
Each cut is logged as `context.trimmed`, with how many results were
shortened and events dropped.
Tokens are estimated from characters (pessimistically, at 3 per token) and
the estimate is corrected from the prompt sizes the server reports. The
correction only ever makes the estimate larger.