Context compression: fit the window without losing what matters or the prompt cache #158

New Issue

2026-10-17T06:49:10Z

dev-agent commented

2026-10-17 06:49:10 +00:00

The operator asked for this after nisse-review failed on #156 (fixed on the reviewer's side in #157). nisse_core.models.context.fit is shared with nisse-dev, and it only knows how to cut by age.

What #156 showed

The transcript reached about 575 KB in two attempts: 97 tool results (411 KB), plus two briefs of about 60 KB each.
Results it still needed were cut. Step 1 cut tool results older than the last 6 turns down to 600 characters, head and tail. Files read earlier dropped out of view and were read again: review.py four times, gitea.py and scheduler.py three times each. Each re-read added another 20 KB.
One model call took longer than 600 seconds and the run died.

Problems with the current approach

Age isn't relevance. The oldest results are often the files the reviewer is still reasoning about. A file read three times is kept three times, while an earlier, unique read is the one that gets cut.
It works against the prompt cache. llama.cpp reuses the longest unchanged start of the prompt. Once trimming begins:
- step 1's cut point moves forward one turn per call, so every call reprocesses from there on (roughly the last 6 turns of results);
- step 2 drops events from the start, which changes the start of the prompt itself, so every call reprocesses the whole prompt.
On a single-slot 4090, that probably explains the slow calls. It's likely, not measured: it could be checked against llama-server's prompt-reuse log lines.
Step 2 can drop the brief. The PR, its diff and its description are the first event, so they're the first thing to go.

Ideas to weigh (needs a design pass and sign-off)

Repeated reads: keep the latest read of each file in full, and shrink the earlier reads of the same file to a pointer to that latest read.
Pinned events: never cut or drop the brief (or nisse-dev's plan); summarise around them.
Cut in steps, not every turn: when over budget, trim well below the limit in one go, so the prompt's start stays stable for many calls and the cache is reused.
Summarise instead of truncating: have the helper model summarise old turns (and say what it read and concluded), rather than keeping a head and tail.
Ask for less up front: smaller default diffs in the brief (diff_chars), with files read on demand.

Related: the 4090 runs llama-server with --parallel 1. --cache-ram is being raised after the RAM upgrade, which helps across conversations, but not with a prompt whose start changes on every call.

🤖 Generated with Claude Code

https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

The operator asked for this after nisse-review failed on #156 (fixed on the reviewer's side in #157). `nisse_core.models.context.fit` is shared with nisse-dev, and it only knows how to cut by age. ## What #156 showed - **The transcript reached about 575 KB** in two attempts: 97 tool results (411 KB), plus two briefs of about 60 KB each. - **Results it still needed were cut.** Step 1 cut tool results older than the last 6 turns down to 600 characters, head and tail. Files read earlier dropped out of view and were read again: `review.py` four times, `gitea.py` and `scheduler.py` three times each. Each re-read added another 20 KB. - **One model call took longer than 600 seconds** and the run died. ## Problems with the current approach 1. **Age isn't relevance.** The oldest results are often the files the reviewer is still reasoning about. A file read three times is kept three times, while an earlier, unique read is the one that gets cut. 2. **It works against the prompt cache.** llama.cpp reuses the longest unchanged start of the prompt. Once trimming begins: - step 1's cut point moves forward one turn per call, so every call reprocesses from there on (roughly the last 6 turns of results); - step 2 drops events from the start, which changes the start of the prompt itself, so every call reprocesses the whole prompt. On a single-slot 4090, that probably explains the slow calls. It's likely, not measured: it could be checked against llama-server's prompt-reuse log lines. 3. **Step 2 can drop the brief.** The PR, its diff and its description are the first event, so they're the first thing to go. ## Ideas to weigh (needs a design pass and sign-off) - **Repeated reads:** keep the latest read of each file in full, and shrink the earlier reads of the same file to a pointer to that latest read. - **Pinned events:** never cut or drop the brief (or nisse-dev's plan); summarise around them. - **Cut in steps, not every turn:** when over budget, trim well below the limit in one go, so the prompt's start stays stable for many calls and the cache is reused. - **Summarise instead of truncating:** have the helper model summarise old turns (and say what it read and concluded), rather than keeping a head and tail. - **Ask for less up front:** smaller default diffs in the brief (`diff_chars`), with files read on demand. Related: the 4090 runs llama-server with `--parallel 1`. `--cache-ram` is being raised after the RAM upgrade, which helps across conversations, but not with a prompt whose start changes on every call. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01J5Utyfzw7gF7UhjRDEtjTk

dev-agent referenced this issue from a commit

2026-10-17 22:51:36 +00:00

Compaction: the model's notes replace the middle of a long run (#158)

dev-agent referenced a pull request that will close this issue

2026-10-17 22:51:51 +00:00

Compaction: the model's notes replace the middle of a long run (#158) #163

dev-agent closed this issue

2026-10-17 23:36:11 +00:00

dev-agent referenced this issue from a commit

2026-10-17 23:36:11 +00:00

Merge pull request 'Compaction: the model's notes replace the middle of a long run (#158)' (#163) from context-compaction into main

Sign in to join this conversation.