Scanning two separate spindles at the same time

The best solution for finding and removing duplicate files.
Post Reply
Callistemon
Posts: 85
Joined: Fri Jun 25, 2021 5:15 am

Scanning two separate spindles at the same time

Post by Callistemon »

When there are multiple folders listed to scan and they are all on the same platters, then it performs exactly as it should, and only focuses on one folder at a time in order to not distract the actuator arm. However, even when scanning between two separate spindles, it still only scans one directory at a time, so it takes twice as long as necessary and leaves the other HDD waiting for half of the time. I have a disk that needs to be decommissioned and I want to delete everything from it that is also found on another disk, and then delete or transfer the remaining unique files. It will have to wait 20 extra hours if the two HDDs cannot be scanned at the same time.
Callistemon
Posts: 85
Joined: Fri Jun 25, 2021 5:15 am

Re: Scanning two separate spindles at the same time

Post by Callistemon »

Thank you for fixing this in version 5. Are there any plans to compile a 64-bit version of Duplicate Cleaner Pro? The 32-bit program freezes up frequently with long lists (over 100,000 items) even with plenty of memory available.
User avatar
DigitalVolcano
Site Admin
Posts: 1717
Joined: Thu Jun 09, 2011 10:04 am

Re: Scanning two separate spindles at the same time

Post by DigitalVolcano »

It runs in 64 or 32 bit automatically depending on your Windows Environment.

Still tweaking the list to try and make it faster. A few things help-

Don't turn on the Preview column with a long list.
Don't turn on grouped view
Callistemon
Posts: 85
Joined: Fri Jun 25, 2021 5:15 am

Re: Scanning two separate spindles at the same time

Post by Callistemon »

Apparently it is running as 64-bit, as Task Manager does not show (32-bit). I thought it was 32-bit since it is in the x86 folder.
Callistemon
Posts: 85
Joined: Fri Jun 25, 2021 5:15 am

Re: Scanning two separate spindles at the same time

Post by Callistemon »

Although it now scans both spindles at once within the partial or full stages, it does not move a disk to the next level until all other disks finish the process the first disk already completed. If one disk finishes much sooner for partial hashes (because it contains less files total), but then takes longer for full hashes (because there are more duplicates), there is unnecessary time spent with each disk waiting on the other when the first disk could have proceeded, and had the second disk outpace it later and finish at the same time.

If a disk has I/O Device Errors after the actuator arm parks during inactivity, as with older Seagate 3.5" HDDs that have exceeded 55C, it is essential to avoid idle to the maximum extent possible. If an I/O Device Error occurs from inactivity, the disk needs to be removed to clear the error, which interrupts the scan before it has a chance to complete. If each disk moved to the next level immediately, even if there were I/O Device Errors caused by the final idle while the entire scan completes, at least the disk with I/O Errors can be removed and inserted without interrupting the scan.

Current: burnt disk finishes Partial Hashes first, enters idle until other disk finishes Partial Hashes. Other disk finishes Partial Hashes, so burnt disk starts up with Full Hashes along with the other disk. However, since it was idle, it cannot proceed without an I/O Device Error, which can only be cleared by power cycling the burnt disk. Power cycling the disk requires removing it, which interrupts the scan. The scan progress cannot be saved until done, so all progress is reverted.

Proposed: burnt disk finishes Partial Hashes and continues through to Full Hashes without interruption. Since it gets a head start with Full Hashes, it does finish before the other disk, and starts idling in the end. Since it's inactive, it will have I/O Device Errors once resumed. However, that does not matter for now, and the burnt disk can wait to be power cycled until the other disk finishes. Once the other disk finishes, the scan progress is saved, so the burnt disk can now be removed and inserted without affecting the progress and results.
Post Reply