Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • I just spawn a subagent in the cheapest range (for example flash-lite) to summarise a tool use. It's the only way that has worked based on my benchmarks and generalises well.
    by elij
  • Is there an easy way to do this with eg codex? It seems like eg sol agents can’t spawn Luna subagents, so eg a “code research” subagent can save the main agent’s context, but can’t save tokens necessarily. (I suppose a tool to call codex CLI would work, but a bit unsatisfying.)
    by bustermellotron
  • Surprised this is so far down. A subagent with a cheap model like haiku or similar is the way to go instead of dumping tool output directly into the main agent context
    by chorizo
  • Some months ago I was evaluating command output compressors to integrate into Dirac[1] as that seemed like an easy win that would compliment and compound with Dirac's other mechanisms.

    I tested rtk among these and it was actually a net negative in both CPU time and accuracy, the latter would throw LLMs way off and make it hard to recover. If you are building a coding agent, I'd hard pass on rtk.

       ~ $ time grep Return * 2> /dev/null | wc -l
   966
   grep Return * 2> /dev/null 0.36s user 0.02s system 98% cpu 0.382 total
   wc -l 0.00s user 0.00s system 1% cpu 0.380 total


   ~ $ time rtk grep Return * 2> /dev/null | wc -l
   260
   rtk grep Return * 2> /dev/null 4.10s user 17.10s system 92% cpu 23.008 total
   wc -l 0.00s user 0.00s system 0% cpu 23.007 total
    Much worse CPU consumption, and more importantly, plain wrong result. These kind of results compromise the entire agent performance because the model trusts wrong output. Without the correct results, any hypothetical savings are penny wise pound foolish

    So yeah I am still on the lookout for a credible CLI wrapper, do let me know if you have any in mind.

    [1] https://dirac.run/

    by GodelNumbering
  • This makes sense. “Don’t try to penny-pinch your employees” is a lesson most managers learn eventually, and I guess agent-orchestrators will have to learn it too.
    by fwlr
  • The technique looked dubious from the start, because LLMs were trained to expect certain outputs from common bash tools. If the output is not what it expects, an LLM may issue more tool calls than before, because it will assume the tool is broken, the arguments passed to it were wrong, or it's a newer/older version of the tool etc => more tokens. Sounds like just adding to the prompt to use `grep` and `tail` extensively will do the trick without any special tooling.
    by kgeist
  • This was the problem I saw. I installed rtk when it came out and liked the idea of it. But over time with newer model generations I kept seeing the model get confused in the reasoning text and retry a command bypassing rtk. I didn't even need a benchmark to see it was regularly an impediment to the final outcome.
    by jghn
  • Patrick here, I build rtk.

    The `rtk find` loop was fixed in 0.46.0. That's the pattern: it gets reported, we fix it.

    We published our own reproduction after the JetBrains run: https://www.rtk-ai.app/blog/rtk-on-skillsbench/

    Bash output is a small share of the bill, so that's the ceiling, and turn-count variance is bigger than the ceiling. Your 7% terminal share on Fable is the same finding from the other direction.

    We're finalizing an integrated proxy inside rtk itself.

    by patrick_rtk
  • I wrote article about it couple months ago that I didn't believe it works. Nice to see numbers now

    https://mroczek.dev/articles/the-token-compression-illusion-...

    by lackoftactics
  • Yes, your article was noticed in my circle, but numbers was what was lacking
    by daliusd
  • Glad to see that more and more people realize these are just snake oils. Without objective metrics like benchmarks, none of the claims mean anything.

    That's also how I feel about skills/plugins. While some provide important context for specific projects/environments, I am very skeptical about (over)generalized skills like "writing JS tests" or "creating a spec". There are dozens of these skills internally at my company, but I haven't seen a single benchmark that shows any of those are better than just plain, single sentence prompts in a meaningful way (aka statistically significant).

    by fg137
  • I see the same thing and have effectively the same philosophy. If I'm using something like figma or glean or playwright/chrome dev tools, plugin/skill/mcp - likely very useful.

    But so many of the weird collections of skills that people on YouTube get viral followings for - I just don't get it.

    People excitedly ask me what skills I use and I feel bad just saying only things we've directly authored for some express purpose. None of the "hot" ones.

    I've written a large handful of skills, but they aren't like vim plugins. I don't just leave them "on".

    This has been my experience at least- curious if I'm just behind the times.

    I also effectively didn't leave the IDE+ChatGPT copy/paste workflow until the first release of Claude code. So maybe I'm slow to adopt.

    by jasonjmcghee
  • Yes I’m also suspicious of skills. I think the main issue is forcing everything down the skill’s path even if it’s not actually necessary.
    by joegibbs
  • Main takeaway:

      Average cost per attempt, without → with RTK:
  
  Claude/Fable: $1.72 → $1.64 (~5% cheaper)
  DeepSeek: $0.115 → $0.121 (~5% more expensive)

  Almost all Claude savings came from a single task.
  Excluding it, savings were under 1%.
    It took me a few rereads to parse out the top-line. This article really buries the lede.
    by gillesjacobs
  • It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves? My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark.

    Here are other cases demonstrating the exact same issues with these kinds of tools:

    https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-... https://brandonbarker.me/writing/headroom-fewer-tokens-bigge...

    by ProjectBarks
  • Even JetBrains is now AI blog-slop, how disappointing.
    by grim_io
  • > If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves?

    Not defending these tools, but one reason these might not be upstreamed is because it would negatively impact vendor margins, and they have no incentive to save their users money

    by ericyd
  • Naively, I think some optimizations would require access to the whole codebase and that would make people nervous (plus incur more cost).

    Eg absurd idea, but you could write something that minifies a codebase (by token, rather than byte) and then translates edits back into the expanded code. Probably an insane use of fuse lol. Partially minifying on each tool call sounds like a huge pain with a lot of state to track.

    There’s also a lot of common situations where humans prefer solutions that take more tokens because it’s easier for us to read (eg for loop vs map vs list comprehension), which may have some gains.

    I strongly suspect there is some form of token compression that works, but I don’t think it will be as simple as “pipe arbitrary text with no context into this tool”.

    Jetbrains feels like a place this might come from. “Take this code, parse it to an AST, find the fewest token representation of it” feels like something they’d do, or maybe Astral (specifically in Python land, type checkers feel sort of adjacent as well).

    by everforward
  • It's pretty damn obvious to anyone who ever bothered to look at rtk gain output, no benchmark needed at all. Agent runs

      rtk command-that-prints-100k-tokens | tail -5
    costs 5 lines, maybe 100 tokens without rtk, but rtk will report 100k savings. Of course it doesn't know about that tail -5.

    Worse, since rtk defaults to persisting that savings stat, it breaks sandboxing. Prefixing with rtk leads to random auto-mode denials from time to time too (this is independent of disabling savings stat persistence).

    Honestly have no idea why anyone who knows the first thing about CLIs would take rtk gain seriously. I guess clueless vibe coders who has hardly ever worked in a terminal before will look at the stat and feel good about it?

    That said, rtk is still mildly useful for compressing repeated test run outputs and stuff, but you should only ever use it on whitelisted commands; wrapping everything like they suggest you to do is just stupid.

    by oefrha
  • All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md.

    What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.

    Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://news.ycombinator.com/item?id=49654229). Semantic search looks for similarities so you have to do less brute forcing. Comes of course at the cost of indexing everything first.

    You can find the project here: https://github.com/ory/lumen

    by aeneas_ory
  • You use caveman to save on tokens, I use it to make the agent shut up. We are not the same =)

    When caveman was released, token efficiency was a lot more relevant. But in the time I've actually come to like its matter-of-fact and short way of responding. There are other similar ones, but caveman is familiar.

    by theshrike79
  • A bit different but in the same domain GitNexus has been really good for me. Symbol based search and impact analysis. Once I added git hooks to auto index and added some explicit instructions it saved quite a bit for me. I have no affiliation just wanted to add that as some thing to try out.
    by alemanek
  • How does lumen compare to semble?

    https://github.com/MinishLab/semble

    by gslepak
  • I've gotten good usage out of a structured search tool (not mine, by someone else here) called Tilth:

    https://github.com/jahala/tilth

    It combines search with tree sitter grammars so the results can annotate usage vs definitions, cite line number ranges, inline the actual definition if it's short, etc. Not as precise as LSP but simple (no daemon), human readable, and in many cases works without configuration.

    by hedgehog
  • Isn't that what everyone was doing in the beginning of harnesses(cursor) and learned not to do for various reasons? https://news.ycombinator.com/item?id=43164253
    by tracerbulletx
  • I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only tool that actually doesn't regress on reasoning while taking costs down. But I still have a lot more evals to write, so this isn't conclusive or anything. (Also I haven't tried Lumen yet)
    by lopatin
  • This sounds quite similar to dirac which made a stir a few months ago:

    https://github.com/dirac-run/dirac

    I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness.

    If any person other than the author has time to verify these Lumen benchmark results I'd be curious to hear it. I don't have the time to do it myself at the moment.

    by esperent
  • I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours?

        Save 30% token costs when using Claude Code, Codex, OpenCode for free - with open source, local semantic search. Works for small and large codebases and monorepos! Enterprise-ready and fully compliant via Ollama and SQLite-vec.
    Releases v0.0.42 Latest last month
    Why should I trust that what you're peddling isn't snakeoil?
    by Whitespace