Does code cleanliness affect coding agents? A controlled minimal-pair study

Join the discussion

• Hacker News
• In my experience this has an obvious answer (yes). LLM’s yes-and whatever patterns are present in the code base no matter how insane. If you don’t tend to your code garden it will be overcome with weeds in an instant.

  At work we had largely moved off of some old hardware but kept partial support for it in the code because normally that would be zero cost. But the constant overhead of bullying, bribing, and threatening the LLMs to leave that code alone was way more work than just excising it and reimplementing it if needed.

  by aomix
• A lot of those tokens are traversal - either searching for code or following call-sites. Basically building enough context to be able to work on the task.

  You can reduce a lot of the token use for traversal by giving your agent access to some form of LSP in addition to hierarchical direction with your AGENTS.md (or equivalent) for monorepos - but a spread-out codebase is always going to end up requiring some form of traversal to solve each task.

  And that traversal isn't just token use - its repeated round trip latency (LLM (queue time -> prefill -> decode -> output) -> Agent (parsing -> tool call -> tool response) -> back to LLM) for EACH step (well, some can be done in parallel, but in practice its mostly sequential) - slowing down the task considerably.

  Locality and structure are key when it comes to efficient use of agents. The context window is always bounded and attention across it is inconsistent.

  by supermatt
• In my experience everything that affects engineers affects an agent. Good abstractions, reasonably sized methods, good names, principled (intra & inter) service architecture, unit tests, etc.

  All of these things have historically been the job of engineers, because it helps other people contribute to the code.

  Now it helps other people and other agents contribute to the code.

  by ford
• It has taken LLMs to encourage companies to prioritise a clean codebase, tickets with unambiguous context and examples of what is right and wrong when onboarding new team members.

  If only we cared so much for junior developers.

  by hexasquid
• imo yes, based on my experience with 4+ repos in production built using AI.

  1. clean structure means the AI can predictably work because i want deterministic output.

  2. basic things like following SRP leads naturally to layered code which makes testing new features and fixing bugs super simple.

  3. reviewing code is simpler, because i see ai agents as "servants" i command to do something and thus an overseer is still required

  by moomoo11
• Interesting to see this quantified. Clean structure seems to lower the cognitive load for both humans and agents, which probably explains why naming and modularization matter more than we think.
  by jkwang
• The way we should define code quality is arguably how easy it is to affect correct changes to the code, that's hard to quantify, but ultimately the thing any code quality metric is trying to capture.

  Based on that, it should be a pretty unsurprising conclusion as long as the code quality metrics you are using are reasonable; as long as the quality metric is good (within the context of coding agents), then this is the result we'd expect to see.

  by marginalia_nu
• Even if agents can learn to navigate all the stubs and WET crap they leave behind do we really want a code base that no human can follow what's actually going on?
  by ngsevers
• If anything I've seen them go too far with DRY. Like two small functions have logic that you could separate into a shared helper, but no human programmer would do that because it's an unclean abstraction and breaks next time you want to alter either one at all.
  by frollogaston
• Even if the agent does everything, English is an imprecise description of what the code does.

  So I personally at a minimum will want to talk “in code” about what code does.

  by softwaredoug
• In similar work we're conducting at NJIT, we've found similar results. (We call it "Contextual Quality Contagion") Some of the most interesting comments here are about some of the real-world scenarios that are commonplace in industry: "mixed" quality codebases, as well as codebases with a mix of legacy code patterns with newer "better" patterns, causing the agents to confuse the conventions.

  The minimal-pair design is honestly one of the strengths, as it tries to isolate the cleanliness from other factors (e.g. architecture, dependencies, tests) which is more rigorous than comparing repos. However, using LLM-generated "sloppified" code (rather than some kind of mechanical or human-guided) is a bit questionable.

  I'd say the biggest critique which others have correctly mentioned here is the authors' choice to not check the full test suite. The claim of "behavioral equivalence" is only as good as the tests and coverage.

  This hypothesis is compelling for two reasons: 1) it makes sense (garbage in -> garbage out as it were) since LLMs will mimic what they see in the codebase; 2) it matches what many engineers feel somewhat intuitively over the last year or two of using these models. Greenfield is almost always easier than joining a busy codebase. The mess comes in complex integration, and maintaining a system for legacy purposes, etc.

  by tom-villani
• I can't imagine how it wouldn't. None of them can fit a real codebase in context and have to browse the code the same way a person would. Doing searches and reading files. If the files are in the places they would be expected to be and things are called what the model or a person would first guess to search then it gets found in the first attempt rather than requiring a deep search and multiple attempts.
  by Gigachad
• An LLM doesnt have to hold the whole codebase in context. Every path, then shift to next paths because you can ignore sections covered. Much like a developer would.
  by Supermancho
• "agent pipelines that [...] clean a messy [repository]"

  This feels like a terrible approach, sufficient to condemn the entire study.

  Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.

  by wgd
• Would you trust clean repos that are messed up by AI?
  by ramraj07
• First author here. Please let me offer a clarification. Our notion of "clean" isn't to just ask the agent to write better code. Rather, we give it a list of 50-100s static analyzer rule violations (and code LOC), and ask to remove them. We then check if the rule violations are resolved.

  Using LLMs to rewrite code to remove these violations is a rather accepted practice. Sonar's existing one-shot LLM based approach [1] (in production since 1+ year), and a recent agentic approach [2] to do the same work rather well to do this.

  [1] https://www.sonarsource.com/solutions/ai/ai-codefix/

  [2] https://www.sonarsource.com/products/sonarqube/remediation-a...

  by geraltofrivia
• One trick I've found that works well is to tell it to refactor, e.g for Python:

      Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions.
  

  A variant I've used for Rust code:

      The Rust codebase in `/src` has become bloated with several files >1k LoC. Refactor the Rust codebase to fit code organization standards expected of popular open-source Rust code without causing any benchmark performance regressions.
  

  Those types of prompts appear to a) reorganize the code logically and b) do seem to get better performance from the agents because the file names now provide semantic hints to where relevant code resides. For bloated 5k LoC files, the agent has to Read several chunks to find relevant code which is inefficient.

  In terms of benchmark performance it generally improves after the refactor which I suspect is coincidental (especially in Rust where it shouldn't matter due to compiling) but I'm not complaining.

  by minimaxir
• Have you tried telling it:

  “Write perfect code, make no mistakes”

  I use this one in my Ralph Harness all the time, it’s a classic!

  It’s not that it can’t do that, it’s just that you haven’t told it to!

  by hatefulheart
• yeah, me too. I usually ask it to do a code review using SOLID standards and it usually does a good job, if not a little overkill sometimes.
  by cpill
• the word your looking for is idiomatic
  by loremium
• I just say "refactor the codebase" and that also works pretty good!

  I have my code styling rules in my CLAUDE.md already anyway

  by gitaarik
• I can totally see doing this incrementally, but this seems extremely risky to do for the entire codebase in one shot on anything in production. Especially if you don’t have really thoughtful e2e tests of the whole system.
  by softwaredoug
• Asking it to apply the YAGNI principle also sems to work well for trimming codebases down. Generally ask it to review, generate a list of review points, then we go through each one together and I make a decision yes/no on each one (or suggest further modifications).
  by woggy
• Exactly. Simply asking agentic coding tools to clean up code bases, to do some targeted refactorings, to enforce things like SOLID principles, and other good practices can result in a lot of easy improvements.

  I've noticed a thing where by default, agentic coding tools are reluctant to remove code. Even when you tell them to. It will bend over backwards to keep old code around, to add complexity for allowing that code to still be called, etc. Super annoying if you are basically just prototyping. You basically end up with a lot of dead code, which than confuses things when you try to add to it. But once you know this, you can just ask it to get rid of the legacy stuff.

  Keeping the code base clean, actually stimulates AIs to do the right thing. If there are lots of tests, it will add more when creating new functionality. If there's documentation, it will update that without needing to be prompted as well.

  As code harnesses improve, a lot of this is probably being built in as well. Which means even less experienced prompters can get decent results.

  by jillesvangurp
• In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs just getting it right at the 1st/2nd attempt.
  by i_have_an_idea
• i mean this is feeling too but im too paranoid and frequently do refactoring and code organization passes and never don't do it, so i cant say i know for sure there is a delta.

  though people who complain that llms aren't that great strike me as the type to have messy code bases

  by dnautics
• I was reading your comment, agreeing with it but still feeling why this is a bad comment. It just occurred to me that an anecdotal statement like this is the antithesis of scientific discourse. We have a paper here, trying to answer a question, and anecdotal testimonials can only harm the discussion by biasing readers without adding anything of value to let anyone objectively conclude anything on the problem.

  The most useful discussion would be if we all read the paper and critique its methodology or results.

  by ramraj07
• Feel the same way myself when working in messy codebases… At some point, the horrible patterns start to rub off…
  by BobbyTables2
• I’ve been working with these things for quite some time now and every time I simply “treat it like I would a human” it seems to perform better. I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one. Just like it performs better when it has well formed specs and access to documentation.
  by yoyohello13
• Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.

  You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most mileage with agentic coding.

  I wrote down a more detailed post of the various linters I use here:

  https://www.balajeerc.info/Use-Deterministic-Guardrails-for-...

  by hannofcart