Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- They should increase the cache to 10 minutes. 5 is just too low, you can even miss it by taking time to select a response from a question.by jaimehrubiks
- 10 mins would be amazingby gabigrin
- this is so cool. claude should actually opensource part of their harness so the community can improve it
context windows and sessions already get maxed out faster with fable and opus 5 and cost a ton. this should lead to significant savings. checking it out
*(and ingesting into mer personal ai builder /hamzaish. opensource)
- This is a good idea if it could manage it in an acceptable way.
For example, there might be something I intended to complete in one sitting, but took two sittings in the same day unexpectedly. Maybe it could just be a few cache delays per day or something, tagged in advance somehow.
by j45 - How will this not lead to tragedy of the commons?by cosmotic
- Well, it's not a free lunch, you're still paying for the cache-warming request. Most of that request will be cached by definition, but cached requests merely cost less, they aren't free.by Wowfunhappy
- Exactly. I really wish people wouldn't use this. If this becomes popular, anthropic will just modify their cache policy to be much less fair. It's not like they have infinite cache.by davesque
- I assumed (perhaps incorrectly, but it was a guess since I never dug into it) that less used "hot pockets" of previous inference gradually got more stale as time went on, and the conversation went elsewhere and didn't need those bits.
Hearing one byte refreshes the whole thing is huge! 5min is wayy too slow, because sometimes I want to spent more than 5 min looking at a diff before choosing where to go next.
Kind of outrageous, I hope this kind of feature gets built into claude code =D
by purpleidea - Is the 5-minute expiration correct? I thought it was more like ~1 hour.by jonas21
- You can set the retention globally (whole session) to 1 hour which will actually make it more expensive. The default is 5 minutes.
*UPD:* actually it appears the default is authentication-dependent. API key gets 5 minutes, subscriptions - 1 hour.
by s0ck_r4w - If you're running the subscription, by default you are paying 2x for cache writes and you're getting an hour for expiry. So refreshing based on 5 min is wasteful. You need to detect whether you are in 5 min or 1 hour mode.by leemoore
- Interested in how the critics of approaches like this defend an agentic session (with Fable, for example) that stops and runs a multi-hour ML training session. It's a script, so the actual LLM convo goes stale, but then when the results get returned to the main thread you get an expensive cache hit without doing anything.
You would have avoided that cache hit if the LLM session was kept "alive" for those few hours. Why not automate the part where you keep the large main thread alive until you're ready to analyze the results?
by boc - I think you mean miss?by PcChip
- Feels like will be shut down real quickby alukin
- Agreed, the Thermos company's lawyers will probably fire off a terse letter once they find out.by Cyberdog
- Why? Seems fine if people want to burn more cache reads.by l1n
- Yes, this wastes cycle. They dump your cache and deallocate the VMs so that others can use it. This will result in tokens wasted and downtime for others.by gogobio
- FYI, on Pro and Max plans caching lasts for one hour, not five minutes, unless you're currently using Extra Usage.
https://code.claude.com/docs/en/prompt-caching#on-a-claude-s...
(Thank you to EliasWatson for giving me this link just a few days ago, as I was previously confused too.)
by Wowfunhappy - I've directly inspected calls for pro/Max plans and as of today they have 1hr cache expiries. This has definitely degraded to 5 min in the past but that's the behavior today.
If you're paying API rates, you can choose 5m or 1hr yourself (and pay different rates).
Keeping a 1hr cache warm could still be useful, sure, but outside that, I don't see much use of this today.
by unholiness - When using API keys the default is still 5 minutes. Setting 1 hour for the entire session is actually a lot more wasteful considering the 2x rate that it comes withby s0ck_r4w
- This is just making it more expensive for everyone else, right?
How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can.
Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.
by SwellJoe - Eh, I assume Anthropic prices queries somewhere in line with their actual costs. If they don't, and they misalign incentives instead, that's kinda on them.by pkulak
- You’re already paying for the subagent, why should you pay twice? (Once for subagent, and again for waiting for the subagent). That’s double billing.by engineer_22
- How do a few extra requests with the same prefix make it more expensive for everyone else?by s0ck_r4w
- From the README:
> Detect the danger window. When the main lineage goes idle and a subagent is actively running, the main prefix is at risk of expiring.
So, this is not demanding to be at the front of the queue, it's just paying someone to take the place you already had in the queue, when you want to take a leak.
by devnonymous - I agree with you. And sadly, there's nothing we can do about it because it's an arms race. People will try to exploit loopholes to get more benefits and companies will try to prevent exploits--sometimes by making the service worse for normal users.
In this specific case, Anthropic can avoid keeping the cache if it detects this kind of prompt (i.e., if max tokens < some number).
by GMoromisato - You PAY for cache writes. Holding the prefill in VRAM for a certain period of time is part of what you're paying for.by dannyw
- No? You still pay for cache reads... Actually this is actually MORE expensive than just redoing prefill if your context is below 100k.by himata4113
- This has nothing to do with queueing.
Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer.
Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is a non sequitur.
by brookst