Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • The advantage of running local stack is that you can do the compaction at the time of inference, i.e. some tool call runs out of context, you can just pause inference, purge/replace old tool calls with their summaries or just logs by operating directly over tokens on a GPU, rebuilding KV cache (one time prefill hit) and resuming the inference, easily being able to e.g. read 1000 markdowns, each 50k long, in a single LLM call. That's not possible with current agentic harnesses using LLM calls.
    by storus
  • I implemented a few additional strategies in https://github.com/rcarmo/piclaw/tree/main/runtime/src/exten... - including Codex-native server-side compaction. They all have slightly different trade-offs, but I run very long sessions quite successfully
    by rcarmo
  • Was expecting the article to go more in-depth.

    Say, what happens when chain of summaries grows so long, that it still overflows context window. Is summarization runned over the summaries in the context window?

    by zahrevsky
  • What more depth is there to go to? Compaction is a single LLM call (practically) which can have some deterministic diffing/extraction baked in, or multiple LLM calls (generally wasteful). There's only 1 summary in the context window at one time. Every prompt goes [CONVERSATION_HISTORY] + input -> model turn. As soon as total context exceeds that it compacts, so there's no summarisation overflow (you can enable an agent to access past summarisations from past compactions, but the snake starts to eat it's own tail).
    by alansaber
  • There is no chain, just a lossy summary that includes a summary of any previous summary along with the fresh messages.
    by skeledrew
  • I want to like Pi but compaction is why I had to go back to Open Code…

    My problem is that when in a loop and it’s calling tools, it won’t check how close it is to the compacting limit until the whole loop returns to you for the next prompt. And so if you have a run that could go for hours, it’s a gamble if you’ll OOM or an interrupting compaction breaks context and stops the loop without continuing.

    There’s a few extensions that all try to solve this problem, but I’ve found none actually work :(

    by alfiedotwtf
  • Great thread, I was just thinking about compaction. My current line of thought is that compaction/pruning/ctx management in general should be something ongoing and maybe recursive. For example:

    User:'How is auth implemented?' -> [thinking] [codebase exploration with [thinking] in between, 10 file reads, 3 of which were "wrong"] [thinking] -> agent_response

    This little exchange contains a WHAT (how auth actually is implemented) and a HOW (where that info is and how to retrieve it). Maybe this question was part of a larger task. I think that whole exchange could be summarised before it enters context, kind of like what happens with subagents. The main thread would then consist mostly of [summaries]. Eventually the context will fill up anyway and we would summarise those summaries again. Alternatively one could maintain a [master_summary], kind of like an internal state. So new [summaries] get integrated directly and the [master_summary] gets updated.

    by Imanari
  • In juggler (https://news.ycombinator.com/item?id=48883305) I spent a lot of effort on some compaction tricks that are elegant, but in reality find myself not really using them..

    Because juggler has a nested-thread architecture, I had a great ah-ha! moment when I realised that with sub-threads, compaction becomes almost free: you just take the list of items in the conversation, move them all into a new sub-thread, then allow that sub-thread to summarise itself (this summary is what the parent thread sees). Hey-presto, your parent thread is now compacted, and the sub-thread contains all the old messages, so you can browse them if you need to, or just undo the whole operation if you change you mind. This struck me as super-elegant, and I was chuffed that it works.

    However.. What I find myself actually doing now is just using the "new conversation" tool to hand over to a new conversation - I just tell the LLM "create a new conversation, and tell it everything it needs to continue this task". It gets you to the same place, but generally I'm not interested in the history, and having a fresh conversation with a new name usually feels neater.

    by julesrms
  • OMP changed the default compaction to images! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.
    by jakswa
  • What's helpful about that?
    by bakies
  • But what if the model you're using doesn't have image processing capabilities?
    by UncleOxidant
  • I don't like any of current solutions when it comes to compaction. I'd love to have a way to say what exactly should be summarized, because most of the time I just need to compact some noisy MCP tool calls, test runs and things like that. Just let me pick what should be summarized and keep the rest as is.
    by damsta
  • I mean, not to be flippant but can't you just prompt the agent to write a file as you're getting closer to the compaction limit? I tend to just go to roughly 50-70% context utilization and then tell the agent to summarize the conversation and save it to a file, manually /clear, then say let's continue that last conversation. You can inspect the summary first and make any changes.
    by boorang
  • I think /handoff on pi (or at least oh my pi) is what you are looking for
    by fermuch
  • Sounds like you might like subagents. Agent > subagent receives agent context (presumably cached)->tool call->compact/summarise->return to main agent
    by alansaber
  • You can do that in Pi!

    > Extensions can intercept and customize both compaction and branch summarization

    https://pi.dev/docs/latest/compaction

    Just make an extension (or ask Pi to write an extension for itself) that intercepts compaction and leaves only what you want, or rewrites it in any other way. Should be just a few lines.

    by flexagoon
  • I think the way prompt caching works really discourages more creative compaction techniques. Like perhaps some kind of heuristic progressive compaction that replaces tool results and thinking traces after use with pointers could potentially keep the model smart for much longer, but that'd mean breaking cache every turn, and possibly even within a turn, seriously driving up cost.
    by skeledrew
  • In my experience, the best approach to compaction is to never get to the point where you need compaction and to generally stay below about 30% context window utilization. Even for long agentic workflows this can be accomplished for quite a while, much longer than most people might think.

    Here's what I do for each of my sessions:

    1. For asides, off-topic work, or repetitive work that has already been done in the session, branch backwards (with /tree) and summarize.

    2. If I've exceeded 30% or the 'price-doubling' multi-tier pricing, prune (my custom extension).

    3. If I've already pruned and I'm still close to 30%, 'prune all' (more extensive prune).

    Definition:

    '/prune': Removes ~50% context on a fresh session (not previously pruned)

      - Keeps: User messages, normal assistant prose, commands/status markers, extension receipts, model settings, and a plain-text receipt for each tool call.
  - Removes: Thinking, signatures, actual tool calls/results, tool output, images, compaction summaries, and other extensions’ state.
    '/prune-extended': Removes ~80% context on a fresh session

      - Keeps: User messages, normal assistant prose and conclusions, commands/status markers, extension receipts, and model settings.
  - Removes: Thinking, signatures, all tool calls/results and output, images, compaction summaries, other extensions’ state, and any tool-activity receipts created by /prune.
    Both create a new session and delete the old one after a successful switch.

    Using these I can keep a session going for weeks (or longer), even with extensive use and almost all the important context is preserved while dumping the less important context. Neither command requires an LLM summarization so they execute quickly.

    by errantmind
  • This is terrible. Models have been RLed on looking at the previous tool call chain, and reasoning. No chance this does not reduce performance. The point of compaction is that it also includes useful signal from the tool outputs itself so agent does not repeat it afterwards
    by MikhailTal
  • People nowdays have no shame and completely forget the art of plug.
    by bvrmn
  • What is that 30% number based on? Surely that's a model specific limit, and is based more on the absolute token length, not percentage, right? I'm not sure it makes sense for e.g. Opus 0.2M and Opus 1M to both degrade at 30% of their respective context lengths.
    by hatthew
  • I am surprised at 'removes actual tool calls/results, tool output'. Your approach with /prune seems to be 'keep the WHAT, remove the HOW (we got here)'. I would have thought that the HOW contains some useful signal.
    by Imanari
  • Compaction is painful if you run just one local LLM, the best way to avoid it is to keep context as small as possible.

    One trick I find useful is to have one model with two KV caches running and while first cache has produced tokens, second cache immediately summarizes them during input tokens are being generated (tools time), then harness switches to the second KV cache which takes newly produced input tokens while KV in first cache is getting replaced with compacted summary tokens. This is a kind of ping pong, so we trade more space for less time. Still experimenting but it looks it works, and nice bonus it improves GPU utilization. Btw I have my own harness and model serving code, but it can be easily implemented in any other harness and model server.

    by novaRom
  • I tend to ask a line of questions to the LLM as first step to a point the context is enough for me. Then i ask for the work i need but then get back to previous context using /tree.

    It's like asking the questions needed to get the answer you need from the LLM, can be either an answer for a plan/todo or a task to code/change something.

    by kristianpaul
  • Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages.

    For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration.

    Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations lead to more frustrating future chats because the LLM misses intent and or context. (Or, the presence of paragraphs and paragraphs of LLM output makes the next token predictor dumber? Unsure.)

    by kierangill
  • Potentially: remove thinking blocks, and keep the rest. At least this would ensure that the entire context of the conversation is still there, and anything said isn't lost.

    Having a second model also iterate the resulting messages and remove low-value tool calls could also be interesting. Especially failed calls which add no value.

    by WhyNotHugo
  • I do this in my own harness, where a context only exports certain messages when you step out of it — eg, my request for an essay and its final output, while dropping everything from the first draft through various intermediary edits.

    That naturally trims a lot of context while removing mistakes from the context to prevent poisoning (ie, every draft but the final negatively contributes in some way — that’s why we edited them).

    by zmgsabst
  • OMP has this, it's called "/shake" where bloat from tool calls and such is turned into an artifact ref instead of remaining in the context.

    There are other pruning or compaction strategies you can configure too.

    You can also use "/btw" to have side tangent conversations with the current context but with the benefit that it will not bloat your current context after the tangent is done.

    I wish other harnesses were prioritising this sort of flexibility with context management, but they're likely more incentivised on optimising their cache rates

    https://github.com/can1357/oh-my-pi

    by behindsight
  • I have a few variations of pruning (trimming all long tool outputs, agent-directed trimming, etc) in my aichat collection of session tools:

    https://pchalasani.github.io/claude-code-tools/tools/aichat/...

    by d4rkp4ttern
  • I'm very interested in this too. I feel like when my Claude session compacts I immediately have to re-alert it to critical aspects of the task, but it should be possible even to have a secondary low-skill agent crawl over the whole context window semi-continuously and nominate pieces for removal or summarization. Or at a higher level, have something able to decompose, like hey I realised we're actually working on three distinct aspects of this problem, let's split the context into those three and I'll just retain a high level summary of what's in the other ones, enough to know if I should pass off control to one of the others for a particular ask.

    I feel like this is kind of subagents, but it doesn't have the clear task -> work -> return result -> die model, rather it's more closely aligned to how I myself multitask, maintaining several long-lived threads of execution, and having to intentionally "forget" about the other ones when it's time to lock in and do real work on the most important one.

    by mikepurvis