Byte to byte comparison too slow

[GibbsBR]

What could I do to make byte-to-byte comparison faster?

I had a big pool (151000 files, 3.87TB) and everything went fine until it got to 45%. When it got there, it started to take 20-40 minutes each single file. Suggestions?

[DigitalVolcano]

With a large amount of files it's much faster to use hashes (SHA-1 should be fine)

[GibbsBR]

Found the problem: about 40000 identical files. Maybe it was doing several unnecessary comparisons?

(I used byte-to-byte due to the high chance of corrupted files. it was a file recovery)

[DigitalVolcano]

If you have a lot of identical files then using hashing will be an order of magnitude faster!