Birbla

Discussion summary

A study with 660 trials found that code cleanliness does not affect the pass rate of coding agents but influences their operational footprint. Several users discussed the impact of code quality on agent performance and maintenance.

What the discussion says

  • Clean code doesn't improve pass rates but affects operational footprint.
  • Agents can clean up and refactor code effectively.
  • Code quality tools like Fallow help maintain standards.
  • Messy code can hinder agent performance, but agents can also improve code quality.
  • The definition of a 'messy' codebase varies depending on context.
“Across 660 trials with Claude Code, code cleanliness does not change the agent's pass rate.”
— jakubmazanec
“Agents are pretty good at cleaning up a codebase, fixing bad abstractions.”
— jaxn

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Interesting to see this quantified. Clean structure seems to lower the cognitive load for both humans and agents, which probably explains why naming and modularization matter more than we think.
    by jkwang
  • Even if agents can learn to navigate all the stubs and WET crap they leave behind do we really want a code base that no human can follow what's actually going on?
    by ngsevers
  • In similar work we're conducting at NJIT, we've found similar results. (We call it "Contextual Quality Contagion") Some of the most interesting comments here are about some of the real-world scenarios that are commonplace in industry: "mixed" quality codebases, as well as codebases with a mix of legacy code patterns with newer "better" patterns, causing the agents to confuse the conventions.

    The minimal-pair design is honestly one of the strengths, as it tries to isolate the cleanliness from other factors (e.g. architecture, dependencies, tests) which is more rigorous than comparing repos. However, using LLM-generated "sloppified" code (rather than some kind of mechanical or human-guided) is a bit questionable.

    I'd say the biggest critique which others have correctly mentioned here is the authors' choice to not check the full test suite. The claim of "behavioral equivalence" is only as good as the tests and coverage.

    This hypothesis is compelling for two reasons: 1) it makes sense (garbage in -> garbage out as it were) since LLMs will mimic what they see in the codebase; 2) it matches what many engineers feel somewhat intuitively over the last year or two of using these models. Greenfield is almost always easier than joining a busy codebase. The mess comes in complex integration, and maintaining a system for legacy purposes, etc.

    by tom-villani
  • I can't imagine how it wouldn't. None of them can fit a real codebase in context and have to browse the code the same way a person would. Doing searches and reading files. If the files are in the places they would be expected to be and things are called what the model or a person would first guess to search then it gets found in the first attempt rather than requiring a deep search and multiple attempts.
    by Gigachad
  • "agent pipelines that [...] clean a messy [repository]"

    This feels like a terrible approach, sufficient to condemn the entire study.

    Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.

    by wgd
  • One trick I've found that works well is to tell it to refactor, e.g for Python:

        Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions.
    A variant I've used for Rust code:

        The Rust codebase in `/src` has become bloated with several files >1k LoC. Refactor the Rust codebase to fit code organization standards expected of popular open-source Rust code without causing any benchmark performance regressions.
    Those types of prompts appear to a) reorganize the code logically and b) do seem to get better performance from the agents because the file names now provide semantic hints to where relevant code resides. For bloated 5k LoC files, the agent has to Read several chunks to find relevant code which is inefficient.

    In terms of benchmark performance it generally improves after the refactor which I suspect is coincidental (especially in Rust where it shouldn't matter due to compiling) but I'm not complaining.

    by minimaxir
  • In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs just getting it right at the 1st/2nd attempt.
    by i_have_an_idea
  • Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment.

    Worse, they don't control for breaking the application's tests.

    > Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests already present in the repository, and a cleaner-side and messier-side solution that both pass the hidden test may still differ on tests they were not graded on.

    Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output.

    by root_axis

Explore Birbla archives