Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I absolutely believe it.
Codex has been pushing things to my main branch all week despite me repeatedly telling it not to and adding to my AGENTS.md very clear instructions for creating feature branches and putting up a PR. It keeps doing it in spite of all that.
I'm probably going to need to enable branch protection on my personal projects... What a pain.
by donatj - What really helped me was to run any .md/prompts through an LLM to find contradictions, duplicates or ambiguities, repeatedly. That led to agents much better following instructions.by storus
- I noticed this behaviour a few months back, I think I was using Sonnet 4.6 at the time... I have strict rules about comments in the codebase, this all for personal projects, and the reason I restrict comments is to keep the token count low.
At some point between the model i was using and the previous version of it, Claude started inserting massive comments with references to tickets and other tasks. All this while having specific directives on the CLAUDE.md
Since then I resorted to developing my crapware as if I was the floor manager of a vehicle assembly line, and I have a few highly-specialized sub-agents running errands around the main session, but only ever taking care of a single concern. The main session builds with the knowledge contained in things like CLAUDE.md but the sub agents make sure things like the no/low-comments directives are either enforced, or factored into the final product.
- This article to me also implies that there are some potential issues with large Spec based development flows, which I haven't been able to pin down lately.
Specifically, having agent implementation drift from the Spec.
by twosdai - Drift is huge between any large spec and agent implementations.
I've done a ton of testing and the model doesn't matter, fable or sol still miss a ton of detail and drift.
I'm building http://engine.build which closes the gap and makes sure the implementation matches the spec.
It's not the same as the satisfaction you get when solving complex problems with code yourself but writing clear specs and thinking through the problem is still very satisfying to me.
by alasano - I have run into the same problem, and started calling the phenomenon vision drift.
I noticed that the issue tracker I built was well suited to address this problem, as it handles board time-travel. So a simple command like ":replay 4h" gives you a clear overview of how the workflow has evolved over time, and you can checkout previous states at will.
Wrote more about it here:
https://dev.to/ljtn/vision-drift-addressing-the-next-problem...
by jolaflow - For Claude, I used a UserPromptSubmit hook running inject_rules.py which reads RULES.md from the disk and prepends the whole thing to every prompt. That helps the rules to stop fading as context fills because it is reinforced every prompt.
Sure, it uses tokens slightly faster in the prompt, but I find it reduces overall token use, you can use it with pro, but of course, nothing works 100% of the time, but it's better. Emptying the memory helps too to avoid Claude making up stuff that messes with how I want it to act.
The general gist of inject_rules.py is :
RULES_PATH points at RULES.md
reads it with encoding='utf-8-sig' so the BOM is stripped
wraps it in a JSON object — hookSpecificOutput.hookEventName = "UserPromptSubmit", additionalContext = a preamble plus the full rules text
the preamble is the line you see above the rules: rules are in force for this turn, run rule 33's five tests before raising anything unasked
prints that JSON to stdout, which is how Claude Code takes it in
on OSError it returns 0 silently — if RULES.md is missing or unreadable, nothing is injected and the turn proceeds with no rules
by My_Name - why this vs having some response hook check the next outputs against the bunch of rules, and injects only when it goes off track?by 8note
- This is real. I was very angry that AI kept breaking the rules I wrote, so I asked Claude to crawl its own history logs. And then I found: every time after it breaks a rule, the chance of breaking rules goes up.
I feel this is like few-shot in reverse. Few-shot is supposed to be good examples AI should follow. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.
I wanted to know if there is any difference between writing rules in the prompt, writing them in CLAUDE.md, or not writing at all. So I did some short tests before. I asked Claude to open brand new sessions, test different topics with the rules I want to apply. The result turns out to be: in a fresh session, no matter the rule is in a prompt or in CLAUDE.md, models (Opus 4.8, 5 or Fable) all follow it fine, across models. Even Opus 4.8, the one always violates rules in our conversations, does it well.
I suspect it's the long context that breaks rules. But simulating a long conversation experiment is kind of hard, I still haven't found a good way to test it. So seeing this paper now, it completely answers the question I was stuck on these few days.
Besides, something caught my eyes in this paper: sometimes the model does run the check by the rules, and it really finds the violation, but its narrative still insists on its original wrong output.
My current approach is same as everyone here: use a separate hook or post-check to fix things. Because if you let the model fix it during generation, its narrative or main generation part sometimes just rejects the rule error it found.
by missmoss - >. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.
In Pre-LLM days the 'nearest unblocked neighborhood' problem, where patching out one issue just immediately runs into another issue, or a different path back to the same issue. Since the models can learn new long time behaviors it's difficult to change the behavior without changing the context quite a bit.
by pixl97 - Long policy documents are also a problem for human agents. Without special training no one will retain 180 pages HR employee handbook, fire codes, OSHA safety rules, FCC regulations, the US legal code.
If the stakes are high, e.g, proceeding in ignorance could lead to prison time, people will favor inaction, even if the policy technically permits a corner case. If the stakes are low, people will completely override policy for the path of least resistance.
by elevation - So what is tre answer then?
I feel like “discretion” parties missing. Do we need some kind of special discretion model?
by DenisM - There was an article a few years ago called "Lost in the Middle: How Language Models Use Long Contexts" https://arxiv.org/abs/2307.03172
From my experience this holds true to this day. It was one of my core observations for similarity to the limitations of human working memory on "Engineering for Bounded Cognition"
by supermatt - Richard Hendricks solved this decisively with middle-out compressionby JSR_FDED
- Most people don't understand that 'agentic AI' is a completely synthetic, force fed capability by extensive Reinforcement Learning on synthetic domain specific 'agentic' datasets on post training.
If the LLM wasn't post-trained to adhere to specific handbook, it just won't work. If the LLM wasn't trained on an use case the lab decided was worth making a synthetic agentic dataset, it won't work as well as you want.
There's a reason the main agentic task LLMs excel at are coding tasks, it's the way of working of the creators, and they understand intimately the flow and can train for it.
I believe the true way will be able to easily fine tune models on your agentic use cases, but it would require a big company to compile a huge dataset on it's way of working and I don't think anyone wants to be the first.
In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
If people spent more time studying about AI and how it works, they would realize that the default should be to one shot prompt your task with a big, cached system prmopt, with an user prompt that is just dynamic data, specified to the cheapest model that can do the job.
Unless you really can't do this given your problem, you should try to make a graph of well defined, step by step oneshot prompts, and THEN if your problem still can't be solved with that, then you start leveraging agents.
Despite this giving better results, and being more cost efficient, is evidently too much work then just letting the AI do all the work.
by msejas - can downvoters explain? this has been by experience with these tools too.
i thought it was well known that claude code got good at coding because anthropic bought tons of coding data from companies like mercor.
by dominotw - What do you mean by a graph of one shot prompts?by drob518
- > In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
"Every gambler knows the secret to survivin' is knowing what to throw away, and knowing what to keep." - Kenny Rogers
Humans have limited context, just like AI. The difference is that humans - at least some of the time - can figure out which pieces are more likely to be important, and therefore prioritize keeping those in the context.
by AnimalMuppet - Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before.
I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things in a prompt during the task, it performs way better.
Result is I'm trying to resist adding more and more things to CLAUDE.md files which in some scenarios it does well but in other scenarios totally ignores and messes up.
by ratmeadow - I’ve had a lot of success using the root Claude.md for a handful of high level application wide rules and directions (I keep it pretty small), module specific claude.md in subfolders alongside the code with more specific rules and direction, and a custom rules backed /code-review skill that enforces it all and catches anything that was missed during implementation.by agotterer
- I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions.
The coding harness is what's getting it to continually align to your current instructions.
This is very obvious with local models.
by cyanydeez - This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.by nonethewiser
- Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it
And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; reasoning depth is limited; and many real-world policies are not actually meant to be implemented in the same way they are written and have insufficient specification of edge cases
With humans, we usually do the equivalent of RLHF, both via "training" with simulated cases, and via feedback while on the job. You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month
by wongarsu - That's a great comparison, human vs ai on a wall of text.
The problem is that it doesn't fit the sales pitch of LLMs and agents - humanlike or better, repeatably, 24/7, for a fraction of the price, you just need to make sure that you give it all the rules.
Unfortunately we can't really have a meaningful conversation until the money vampires have left so we will need to reschedule this until after the bubble.
by ActionHank - isn't it because there are too many contradictions and ambiguity? the reason it works for humans is because we don't apply everything at once either.by loremium
- Exactly: The best way to deal with this for humans is to use procedural scripts for different tasks, referring to the relevant bits of the (declarative) documentation.
I would imagine that doing something similar (using agent skills for insurance) would work much better for AI.
by dinfinity - > Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it
Hmmm, if AI has to grow in the workplace, it has to follow processes to the letter, yet claude code forgets by the second turn my "don't commit" prompt.
Of course claude code is a generic and crap harness over terrific models that can't fit bureaucratic processes, and is increasingly worse at doing so anyway since Opus 4.6 peak.
by epolanski - "You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month"
So the behavior policy should reside in the weights, not in some ever expanding KV cache context, like sticking it in the wrong orifice if the goal is to be reproductive.
Instead of putting the policy document in some tightly crammed RAM, shove it in the weights that already exist, online / post training.
It does make one wonder if there is some way to compute the change to the weights from the computed context (so that we can free it up) without essentially continued "pre" training RMAD after each communication round.
by DoctorOetker - The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly governed by the shared weights and harness policies found to be successful for the average case.
- This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn't mean its true and you should do that.
Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely continue.
Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
by DiabloD3