Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Reflink conversion would be another nice feature to have. It allows userspace apps to create filesystem-native copy-on-write clones of files, so to the next app they work exactly like another copy of the same data. But the data is on disk only once.
    by Maakuth
  • Interested to know what algorithm you use for --fuzzy. Presumably not any of the worst-case-quadratic diff algorithms... MinHash on n-grams?
    by akoboldfrying
  • I have a thing like this - one thing I have found useful is also going up the directory tree to find entire duplicated directories. (So dir hash is hash of the concatenation of the subdirs and the files).

    I also used go, it excels at just this sort of thing. Although I saturate the disk to memory bandwidth (for the SHA) way sooner than CPU on my laptop, so it still takes a while to run.

    Nice.

    by lanstin
  • Here's a tip, if you are making regular compressed backups of the configuration of your network devices, if you use gzip, make sure to set the mtime to 0 when writing the file as this can throw off deduplication. This also applies to the name field.

    https://docs.python.org/3/library/gzip.html#gzip.GzipFile.mt...

    by chaz6
  • Wow. Wasn’t expecting to see this here… I am the author…
    by jdefr89
  • imo fclones is the one to beat, https://github.com/pkolaczk/fclones#benchmarks

    nice that there are actually some ok interfaces here. with fclones, i tend to generate a file of candidates then do a little review to make super certain everything is square / as expected. it's very built for intermediary files a core pattern, which is very unix, very convenient. but it did take me a little while to settle on this, and it felt like i wasn't being offered a ton of options for management out of box. looks like some real attempts to be more user friendly here.

    by jauntywundrkind
  • Why do you need a cryptographic hash function for detecting duplicates? xxHash, rapidHash or anything that’s fast will do the job. There’s no need to be secure against inverses.
    by divyekapoor
  • I'm not sure exactly how it compares, but a similar tool you can use that also does phashes of images/videos to find potential duplicate media files is czkawka [0].

    [0]: https://github.com/qarmin/czkawka

    by psYchotic

Explore Birbla archives