Byte to byte comparison too slow

The best solution for finding and removing duplicate files.
Post Reply
GibbsBR
Posts: 2
Joined: Wed Jun 23, 2021 1:00 pm

Byte to byte comparison too slow

Post by GibbsBR »

What could I do to make byte-to-byte comparison faster?

I had a big pool (151000 files, 3.87TB) and everything went fine until it got to 45%. When it got there, it started to take 20-40 minutes each single file. Suggestions?
User avatar
DigitalVolcano
Site Admin
Posts: 1717
Joined: Thu Jun 09, 2011 10:04 am

Re: Byte to byte comparison too slow

Post by DigitalVolcano »

With a large amount of files it's much faster to use hashes (SHA-1 should be fine)
GibbsBR
Posts: 2
Joined: Wed Jun 23, 2021 1:00 pm

Re: Byte to byte comparison too slow

Post by GibbsBR »

Found the problem: about 40000 identical files. Maybe it was doing several unnecessary comparisons?

(I used byte-to-byte due to the high chance of corrupted files. it was a file recovery)
User avatar
DigitalVolcano
Site Admin
Posts: 1717
Joined: Thu Jun 09, 2011 10:04 am

Re: Byte to byte comparison too slow

Post by DigitalVolcano »

If you have a lot of identical files then using hashing will be an order of magnitude faster!
Post Reply