CLI tool for detecting non-exact code duplication with embedding models

Join the discussion

• Hacker News
• I think that this is pretty cool, but is there any reason why we would want to remove similar/possible duplicate code?
  by SpyCoder77
• Have you written software before?
  by Zopieux
• (without sarcasm) Is this a serious question?

  If so - maintainability, testability. This is old software engineering best practice at this point.

  You shouldn’t hyper optimize for deduplication, but it’s usually worth considering. Fewer places to fix issues or improve as well.

  by rufius
• Recently there was a popular article on HN saying that sometimes code duplication is better than abstraction, so I assume that this question is not a joke.

  While testing this tool, one detected duplication was interesting for a use case. Permission check logic was duplicated and placed in different distant places in the codebase. The code was similar, but not identical, the logic was not the same. One version had stricter checks. I analyzed this with the coding agent, and we found out that both versions are used for the same thing, which means that in some cases validation is insufficient. Having only a single validation place, this bug could be prevented or easily detected.

  by rkochanowski
• This is a great use case for embeddings. Code deduplication across distant modules is notoriously hard for traditional AST-based tools.

  How do you handle chunking and parsing for different languages to make sure the embeddings capture semantic meaning effectively? For instance, do you chunk by functions/classes, or use a fixed token window? If a function is too long or too short, it can drastically skew the embedding similarity.

  by NYCHMPAI
• Generally, I chunk by function/method (not by whole class), but different languages have specific concepts and features. Nested code units, anonymous functions, lambdas, closures are extracted as separate chunks.

  The chunk size has allowed range and those outside are simply ignored.

  - Upper limit is hardcoded with a body size of 10k chars

  - Lower limit is configurable with a default of 10 AST nodes inside the body

  The chunking strategy is something that can be improved in future versions.

  by rkochanowski
• looks super useful- thanks for sharing!
  by noashavit
• have you considered a deterministic tier before the embedding pass? I feel that approach can be more efficient.
  by rohanat
• There are good mature tools for deterministic duplication detection and I intentionally focused on embedding-based to fill this gap (I didn't find other tools using this approach).

  If by "more efficient" you mean to avoid embedding of the same code multiple times, this optimization is already implemented internally.

  by rkochanowski
• We did this by using the ASTs you can go quite far without embeddings and the result is easier to debug and follow what's going on.
  by vander_elst
• Did you benchmark it against simpler methods like BM25?
  by janalsncm
• I just focused on embeddings without comparing them to deterministic solutions.

  But I plan to do my own analysis of different embedding models in the context of code similarity detection. Including BM25 in the comparison is a very good idea.

  by rkochanowski
• Nice, what's the chunking level? I would want sub function, logical blocks, etc
  by mempko
• Function level only. I will add more granular chunking in next versions.
  by rkochanowski
• Very nice. I can imagine putting this into a pre push hook to keep things clean after an initial sweep.
  by hdz
• What a simple and smart idea. Wonderful
  by BrandiATMuhkuh
• This is neat. Have you noticed any difference in duplicate detection between strongly typed and loosely typed languages / code bases?
  by philajan
• No. It depends the most on general code quality and architecture. Some implementations require more code similarity by design. Some languages, like Java, may tend to have more duplication, but it's only a theoretical guess. It also depends on what kind of software is developed with what language.

  If you are interested in data, you can check my article. Analysis was done with this tool, but a previous version where exact-copy duplicates were excluded from analysis. https://rkochanowski.com/article/analysis-code-duplication/

  by rkochanowski
• Looks very cool! I'd be very interested in applying this to my Elixir projects. What does it take to add proper support for a new language?
  by romanoonhn
• Elixir was added in the latest release.
  by rkochanowski
• It can be added with https://pypi.org/project/tree-sitter-elixir/ similar to other languages. I will add this and plan to release a new version today.
  by rkochanowski
• self plug (for similar tool): https://github.com/forhadahmed/refactor
  by forhadahmed
• How does it compare to jscpd? https://github.com/kucherenko/jscpd
  by msephton
• It's the opposite.

  jscpd is advertised as "Copy/paste detector", Slopo is advertised as "non-exact code duplication".

  Slopo also detects copy/pasted code, but this is not the main goal and the report focuses more on similar code units.

  by rkochanowski
• If your using the embedding models to detect the code duplication then how you will be over coming the Agents hallucination???
  by yr_animesh
• For false positives, how about generating the test units and run it in isolation? All major languages have the interpreters or embedded languages to execute function only.
  by ahmetson
• Do you mean to verify if similar code units produce the same result?

  The goal of the tool is to also detect code that is similar and behaves differently. There are not ideal duplicates, but still a code that can be refactored, abstracted, or fixed (because the variance may be the result of bug).

  by rkochanowski
• Have you compared this to https://github.com/MinishLab/semhash (or considered using that for the deduplication backend)?
  by Bibabomas
• Finding similar code is something different than deduplication, even when the final goal looks similar.

  Deduplication backend is the easiest part of the tool and it doesn't need any additional libraries. Just calculate embeddings and find close pairs. The complexity is everything around.

  Using local models is worth considering and the tool already uses the LiteLLM wrapper, allowing it to configure different models, including local. I left this part for the user.

  by rkochanowski
• Nice idea. I can see this being useful before refactors, especially when the duplication is semantic rather than copy paste.
  by murats