Am i Missing something ? (wrong duplicates although same content)

The best solution for finding and removing duplicate files.
maniacbone
Posts: 22
Joined: Fri Apr 05, 2019 3:17 pm

Am i Missing something ? (wrong duplicates although same content)

Post by maniacbone »

Dear Community,

i did not find a suitable thread to this so i hope its ok i write this thread:

i use duplicate cleaner A LOT... and recently i let it scan for 3 days through my whole data... MD5 same content search only limited to files from 0 to 4 gb.....

After it found literally millions of duplicates (yes i am a data freak i know) i found a group of duplicates consisting of 20 files marked as duplicate which are definately no duplicates ! all of them have the exact same size because they are split camcorder ripps from my fathers camcorder cassettes meaning all of them are pretty damn same SIZE... but the CONTENT must be different since the are all distinct different videos...

now i thought md5 makes sure that if 2 files are marked as duplicates this means they are REALLY the same....like "both are the same video"....and not "both are the same size"...hence the "same content" search and not by same size....

is there something i am missing here ? is md5 only capable to find duplicates until a certain amount of megabytes/gigabytes etc ?

most of the duplicates are of course RIGHT...but this wrong group of files really messes upo my trust in this and i must say i have abandoned a lot of programms after such errors...duplicate cleaner ist by far so far my favourite after all these years.... it would really be sad if i found some sort of bug cause i delete a lot of stuff on the daily basis because of DC
maniacbone
Posts: 22
Joined: Fri Apr 05, 2019 3:17 pm

Re: Am i Missing something ? (wrong duplicates although same content)

Post by maniacbone »

Aditional Info:

looking further intgo my results i have THOUSANDS of "dupes" that are no way identical....not anywhere the same size nor anything else comparable...i dont know what Duplicate Cleaner Pro has done here.....
User avatar
DigitalVolcano
Site Admin
Posts: 1717
Joined: Thu Jun 09, 2011 10:04 am

Re: Am i Missing something ? (wrong duplicates although same content)

Post by DigitalVolcano »

Are the hashes the same in the hash column for each group where you think the files should be different? Do the hashes show up as different using an external hash tool?

-Post the scan settings (it should be in your log file)
maniacbone
Posts: 22
Joined: Fri Apr 05, 2019 3:17 pm

Re: Am i Missing something ? (wrong duplicates although same content)

Post by maniacbone »

the Settings are:

13.07.2021 00:13:26
Scanning: Regular mode
- Same content (MD5)
- Maximum file size : 4GB

funny thing is...it did not show the hash column in my results (i could not select it to be shown aswell)

somehow i get the idea that the search did not search the way it should have.... i scanned dozens of terabytes (3 day scan) could that be the problem ?
User avatar
DigitalVolcano
Site Admin
Posts: 1717
Joined: Thu Jun 09, 2011 10:04 am

Re: Am i Missing something ? (wrong duplicates although same content)

Post by DigitalVolcano »

Strange - are you sure you aren't on the Unique files tab? (I have to check!). Database file corruption is also a possibility.
maniacbone
Posts: 22
Joined: Fri Apr 05, 2019 3:17 pm

Re: Am i Missing something ? (wrong duplicates although same content)

Post by maniacbone »

ok how can i check for corruption in the database file ?
User avatar
DigitalVolcano
Site Admin
Posts: 1717
Joined: Thu Jun 09, 2011 10:04 am

Re: Am i Missing something ? (wrong duplicates although same content)

Post by DigitalVolcano »

You can't - but you can delete or rename it to factory-reset the program.

See
https://digvolsoft.freshdesk.com/suppor ... 6000016720
maniacbone
Posts: 22
Joined: Fri Apr 05, 2019 3:17 pm

Re: Am i Missing something ? (wrong duplicates although same content)

Post by maniacbone »

thanks...i will try
User avatar
therube
Posts: 614
Joined: Tue Jun 28, 2011 4:38 pm

Re: Am i Missing something ? (wrong duplicates although same content)

Post by therube »

it did not show the hash column in my results
Columns may be displayed irrespective of other settings.

Right-click a column & enable, 'Hash'.

i found a group of duplicates consisting of 20 files marked as duplicate which are definitely not duplicates
Take two of those files, copy them into a separate directory.
Scan that directory only.
Do they show as duplicated?

(And as mentioned), "Do the hashes show up as different using an external hash tool?"
(Nirsoft, HashMyFiles.)
maniacbone
Posts: 22
Joined: Fri Apr 05, 2019 3:17 pm

Re: Am i Missing something ? (wrong duplicates although same content)

Post by maniacbone »

therube wrote: Mon Jul 19, 2021 6:37 pm
it did not show the hash column in my results
Columns may be displayed irrespective of other settings.

Right-click a column & enable, 'Hash'.

i found a group of duplicates consisting of 20 files marked as duplicate which are definitely not duplicates
Take two of those files, copy them into a separate directory.
Scan that directory only.
Do they show as duplicated?

(And as mentioned), "Do the hashes show up as different using an external hash tool?"
(Nirsoft, HashMyFiles.)
i will check , right now another 3 day scan is still scaning...maybe there everything will be better :) if not i will check the steps you thankfully wrote here
Post Reply