Compaction: the model's notes replace the middle of a long run (#158)
Before each driver call, over compact_at (0.6) of the prompt budget, the same model is asked for notes over the very request it was about to get, plus one message, so the server's cache covers all but the notes. The notes, with code's list of the calls they replace, become a compaction event that renders in place of its range, keeping word for word what started the work and the last few turns. Rare, large steps keep the prompt's start unchanged between them. A later compaction folds in the earlier; one from an earlier run's view is hidden. Failed notes are logged and fitting takes over. Fitting now applies compactions before it measures, records what it cuts (Trim) and logs each cut as context.trimmed. Wired into both agents' drivers and the benchmark; nisse-dev's helper loop is a follow-up. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk
This commit is contained in:
1 parent
fa8d93cf41
commit
cb638fe687
20 files changed
+952
-44
No files matched your search
+35
-2
@@ -283,9 +283,39 @@ what failed last; the start keeps the exit code.
|
||||
- **Too high:** the transcript outgrows the context window sooner. The
|
||||
context budget copes (see below), but by cutting older output harder.
|
||||
|
||||
### `tuning.context` — fitting each request into the window
|
||||
### `tuning.context` — compaction, and fitting each request into the window
|
||||
|
||||
Before any request is sent, the agent checks it against the model's context
|
||||
**Compaction comes first.** A driver run's request grows with every tool
|
||||
result. Once it passes `compact_at` of the prompt budget, the model is asked
|
||||
for notes on its run so far: findings with the code they rest on, what each
|
||||
file told it, what is left to do. The notes replace everything between what
|
||||
started the work (the brief, or the conversation) and the last few turns,
|
||||
which stay word for word. The model server has the request cached already,
|
||||
so the notes cost only their own output. Because it happens rarely and drops
|
||||
the request well below the limit, the start of the prompt stays the same for
|
||||
many calls, and the server keeps reusing its cache.
|
||||
|
||||
Logged as `context.compacted` (tokens before and after, seconds). Notes that
|
||||
fail, come back empty or leave the request still too big are logged as
|
||||
`context.compaction_failed`; the run goes on and fitting (below) cuts
|
||||
instead, and compaction isn't tried again for `compact_keep_turns` turns.
|
||||
|
||||
| Knob | Default | |
|
||||
|---|---|---|
|
||||
| `compact_at` | 0.6 | fraction of the prompt budget that sets compaction off; 0 turns it off |
|
||||
| `compact_to` | 0.25 | what the kept recent turns may take, at most |
|
||||
| `compact_keep_turns` | 3 | recent turns kept word for word, at most |
|
||||
|
||||
- **`compact_at` too high:** little room left for the notes call and a big
|
||||
result that lands in one turn, so fitting cuts more often. **Too low:**
|
||||
compactions come often, and each one loses detail.
|
||||
- **`compact_to` / `compact_keep_turns` too low:** the model loses sight of
|
||||
what it just did. **Too high:** the notes don't bring the request far
|
||||
enough down, and compactions come sooner.
|
||||
- **Deciding:** look for `context.trimmed` in the log. After compaction it
|
||||
should be rare; each one means the model lost something it had read.
|
||||
|
||||
**Fitting is the last guard.** Before any request is sent, the agent checks it against the model's context
|
||||
window and makes it fit. A server doesn't always refuse a prompt that's too
|
||||
long. Ollama silently drops its start, which is where the system prompt and
|
||||
the plan are, so an unfitted request can fail in a way nobody sees.
|
||||
@@ -304,6 +334,9 @@ order, stopping at the first that fits:
|
||||
3. **If the system prompt and the recent turns still don't fit,** the
|
||||
request is not sent. The milestone stops and asks you.
|
||||
|
||||
Each cut is logged as `context.trimmed`, with how many results were
|
||||
shortened and events dropped.
|
||||
|
||||
Tokens are estimated from characters (pessimistically, at 3 per token) and
|
||||
the estimate is corrected from the prompt sizes the server reports. The
|
||||
correction only ever makes the estimate larger.
|
||||
|
||||
Reference in new issue
Block a user