To find duplicates in batch sub-folders

The best solution for finding and removing duplicate files.
Post Reply
xiaocc
Posts: 2
Joined: Mon Sep 13, 2021 9:03 am

To find duplicates in batch sub-folders

Post by xiaocc »

Hi there, I am facing a problem of find duplicates in batch files. I tested 2 types of method. 1st, I created 1 folder, and moved 5000 files in the folder, after that I run the cleaner to check duplicates in the folder, it takes X mins to find out all the duplicates and divided them into groups; 2nd, I created 100 sub-folders under a parent folder, and moved 50 files in each sub-folders, then I setting 'same folder name match depth from top 1' and run the cleaner to check the parent folder, it takes Y mins to find out the duplicates in each sub-folders and divided them into groups. However, Y roughly equal to X. I guess 1st method and 2nd method all used same algorithm which integrate all the files, so the complexity is 5000*5000; what I want is to make the complexity to (50*100)*100 by separating files into different folders. Could anyone help? Thanks a lot!
User avatar
therube
Posts: 614
Joined: Tue Jun 28, 2011 4:38 pm

Re: To find duplicates in batch sub-folders

Post by therube »

What you would want would be something like, "compare only files within the same parent directory" - but I'm not seeing how to do that?

If that were there, you would drag (or otherwise select) your 50 directories into DC & then each directory would be scanned only against itself.

Image

With that, 1d67 would only find dups within the 1d67 directory, & 5548 would only find dups within 5548...

but I'm not seeing how to do that?
Heh. Oh, there it is.

Scan location (tab), Find duplicates (column), you'd need to set each (of the 50) directories to 'Internal only'.
xiaocc
Posts: 2
Joined: Mon Sep 13, 2021 9:03 am

Re: To find duplicates in batch sub-folders

Post by xiaocc »

Thanks for ur reply! I tried this way before when I test. however, in my case, there are too many folders(hundreds of thousand above) need to check, therefore I have to set 'Internal' for each folder. That is why I am looking for help~
therube wrote: Mon Sep 13, 2021 4:36 pm What you would want would be something like, "compare only files within the same parent directory" - but I'm not seeing how to do that?

If that were there, you would drag (or otherwise select) your 50 directories into DC & then each directory would be scanned only against itself.

Image

With that, 1d67 would only find dups within the 1d67 directory, & 5548 would only find dups within 5548...

but I'm not seeing how to do that?
Heh. Oh, there it is.

Scan location (tab), Find duplicates (column), you'd need to set each (of the 50) directories to 'Internal only'.
Post Reply