Birbla

Discussion summary

A CLI tool leveraging embedding models detects non-exact code duplication, addressing challenges in cross-module deduplication. Discussions highlight its use cases, benefits, and comparison with traditional AST-based methods.

What the discussion says

  • Embedding-based detection helps find duplicates across distant modules.
  • Traditional AST tools are easier to debug but may miss distant similarities.
  • Some argue duplication can be preferable to over-abstraction.
“Code deduplication across distant modules is hard for traditional tools.”
— NYCHMPAI
“Embedding models fill a gap in code duplication detection.”
— rkochanowski

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • How does it compare to jscpd? https://github.com/kucherenko/jscpd
    by msephton
  • self plug (for similar tool): https://github.com/forhadahmed/refactor
    by forhadahmed
  • For false positives, how about generating the test units and run it in isolation? All major languages have the interpreters or embedded languages to execute function only.
    by ahmetson
  • Have you compared this to https://github.com/MinishLab/semhash (or considered using that for the deduplication backend)?
    by Bibabomas
  • Nice idea. I can see this being useful before refactors, especially when the duplication is semantic rather than copy paste.
    by murats
  • I implemented this for a large monorepo last year, it runs as an analysis during code review and it shows what are possible similar snippets wrt the code under review. It was a very nice project. It also allows to see across the repo what are the most common constructs for the different languages. This could also be helpful to see if some code has been copied e.g. from open source projects.
    by vander_elst
  • Cool project, I've been meaning to do this myself at work for a codebase, and it's nice to see that this exists now.

    Does the project you simply compute embeddings for every function unit and cluster them, or do we also mean-pool significant dependencies of a function? In other words, given the function

        def a():
      b()
      c()
      d()
    Do we also embed b, c, and d as well and combine them somehow in the embedding of a?
    by supriyo-biswas
  • I built Slopo to solve one specific problem: finding similar code that is hardest to detect by other tools, coding AI agents, and humans.

    It finds similar-looking code with embeddings. This detects more than just copy-paste clones or even clones with minor changes. Similar code is often not a clone to refactor, and this is a trade-off. Initial results need to be verified, but coding agents can do this quickly. Example prompts are available on https://slopo.dev

    Additionally, similar code distant in the codebase is ranked higher to focus on less obvious duplication.

    The results differ a lot depending on the codebase. I noticed that sometimes most of the detected duplicates are false positives, but the remaining ones are strong candidates to refactor or even bugs. Sometimes it reveals much more real duplication.

    by rkochanowski

Explore Birbla archives