Birbla

Discussion summary

A CLI tool leveraging embedding models detects non-exact code duplication, addressing challenges in cross-module deduplication. Discussions highlight its use cases, benefits, and comparison with traditional AST-based methods.

What the discussion says

  • Embedding-based detection helps find duplicates across distant modules.
  • Traditional AST tools are easier to debug but may miss distant similarities.
  • Some argue duplication can be preferable to over-abstraction.
“Code deduplication across distant modules is hard for traditional tools.”
— NYCHMPAI
“Embedding models fill a gap in code duplication detection.”
— rkochanowski

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • What a simple and smart idea. Wonderful
    by BrandiATMuhkuh
  • This is neat. Have you noticed any difference in duplicate detection between strongly typed and loosely typed languages / code bases?
    by philajan
  • No. It depends the most on general code quality and architecture. Some implementations require more code similarity by design. Some languages, like Java, may tend to have more duplication, but it's only a theoretical guess. It also depends on what kind of software is developed with what language.

    If you are interested in data, you can check my article. Analysis was done with this tool, but a previous version where exact-copy duplicates were excluded from analysis. https://rkochanowski.com/article/analysis-code-duplication/

    by rkochanowski
  • Looks very cool! I'd be very interested in applying this to my Elixir projects. What does it take to add proper support for a new language?
    by romanoonhn
  • Elixir was added in the latest release.
    by rkochanowski
  • It can be added with https://pypi.org/project/tree-sitter-elixir/ similar to other languages. I will add this and plan to release a new version today.
    by rkochanowski
  • If your using the embedding models to detect the code duplication then how you will be over coming the Agents hallucination???
    by yr_animesh
  • How does it compare to jscpd? https://github.com/kucherenko/jscpd
    by msephton
  • It's the opposite.

    jscpd is advertised as "Copy/paste detector", Slopo is advertised as "non-exact code duplication".

    Slopo also detects copy/pasted code, but this is not the main goal and the report focuses more on similar code units.

    by rkochanowski
  • self plug (for similar tool): https://github.com/forhadahmed/refactor
    by forhadahmed
  • For false positives, how about generating the test units and run it in isolation? All major languages have the interpreters or embedded languages to execute function only.
    by ahmetson
  • Do you mean to verify if similar code units produce the same result?

    The goal of the tool is to also detect code that is similar and behaves differently. There are not ideal duplicates, but still a code that can be refactored, abstracted, or fixed (because the variance may be the result of bug).

    by rkochanowski
  • Have you compared this to https://github.com/MinishLab/semhash (or considered using that for the deduplication backend)?
    by Bibabomas
  • Finding similar code is something different than deduplication, even when the final goal looks similar.

    Deduplication backend is the easiest part of the tool and it doesn't need any additional libraries. Just calculate embeddings and find close pairs. The complexity is everything around.

    Using local models is worth considering and the tool already uses the LiteLLM wrapper, allowing it to configure different models, including local. I left this part for the user.

    by rkochanowski
  • Nice idea. I can see this being useful before refactors, especially when the duplication is semantic rather than copy paste.
    by murats
  • I implemented this for a large monorepo last year, it runs as an analysis during code review and it shows what are possible similar snippets wrt the code under review. It was a very nice project. It also allows to see across the repo what are the most common constructs for the different languages. This could also be helpful to see if some code has been copied e.g. from open source projects.
    by vander_elst
  • Cool project, I've been meaning to do this myself at work for a codebase, and it's nice to see that this exists now.

    Does the project you simply compute embeddings for every function unit and cluster them, or do we also mean-pool significant dependencies of a function? In other words, given the function

        def a():
      b()
      c()
      d()
    Do we also embed b, c, and d as well and combine them somehow in the embedding of a?
    by supriyo-biswas
  • Based on your example there is only a single function a() which is embedded. The rest is just a code and dependencies are not resolved. Did you think about adding this feature in your tool?
    by rkochanowski
  • It looks like it works only on function bodies[1]. I'm not sure I understand why you would want to look at invoked callables code, though. Calling the same set of helper functions is already flagged; repeated code in helpers is flagged as well when those helpers are analyzed. Do you have a specific example where you'd like a function flagged as a duplicate based on the code it calls out to?

    [1] https://github.com/rafal-qa/slopo/blob/main/src/slopo/indexi...

    by klibertp
  • I built Slopo to solve one specific problem: finding similar code that is hardest to detect by other tools, coding AI agents, and humans.

    It finds similar-looking code with embeddings. This detects more than just copy-paste clones or even clones with minor changes. Similar code is often not a clone to refactor, and this is a trade-off. Initial results need to be verified, but coding agents can do this quickly. Example prompts are available on https://slopo.dev

    Additionally, similar code distant in the codebase is ranked higher to focus on less obvious duplication.

    The results differ a lot depending on the codebase. I noticed that sometimes most of the detected duplicates are false positives, but the remaining ones are strong candidates to refactor or even bugs. Sometimes it reveals much more real duplication.

    by rkochanowski
  • If it did PHP I would love to run it over WordPress. What would it take to add that?
    by realxrobau
  • What a clever little tool. This is exactly the kind of pragmatic AI tools I want to see more of: linux-y single purpose tools!
    by jadbox
  • Correct me if I'm wrong, but looking at [1] it seems to be specifically using function definitions (I'm guessing this works with functions, methods, and lambdas (the "<unknown>" part)?) as units of repetition. If yes, that's fine, but I would seriously consider adding some settings to allow the user to control that granularity. Sometimes, the repeated code is a conditional branch within larger functions (i.e., "every else:" or "every except Ex:" looks the same). If the functions are large enough, the dissimilarity of the rest of the body would (probably?) cause such things to be missed.

    I would also consider - perhaps as a separate pass, with scoring set differently - to analyze comments (especially docstrings in Python). If I read the code correctly, you're currently just stripping them, which is the right thing to do when looking for code duplication, but duplicated docstrings are also often a signal that something is wrong in the codebase. The "different scoring" is because we expect docstring to be structured similarly (at least more than normal code), so some tweaking would be needed.

    Finally: very nice project, congrats! :)

    [1] https://github.com/rafal-qa/slopo/blob/main/src/slopo/indexi...

    by klibertp