Story: Hardware monitoring is critical! Unique files will save me! (I hope)

[wwcanoer]

My story: I am just starting to scan my mess of old USB drives. I first scanned 4 and then added three more and everything seemed fine. When the computer went to sleep and wouldn't resume, I shut it down, disconnected the drives and restarted.

Hard Disk Sentinel always pops up when I start my computer. I usually close it. This time, it was open while I reconnected the USB drive.

SHOCK: One 5TB drive turned the screen red and announced 5% health!

Panic: OMG! I don't have 5TB of free space! What am I going to do?! Will this drive last long enough for me to dedupe 5TB of space?! Hopefully there's a second copy of everything on it because I try to have two backups (and my problem is that I have too many copies). But do I? This is my newest drive, so I could have some unique files on there.

Relief: Oh ya, Duplicate Cleaner has a unique files tab.

Panic: Where's the unique files tab? It's gone in version 5?! (I just upgraded to it.)

Read the manual: Oh, I have to select "Find Remaining" for that drive on the Locations tab. Ok, calm, let's try this.

Rescan disks: I need to connect all 7 disks again and re-scan. This shouldn't take too long because thankfully version 5 remembers the previous scans. (Good to save them too!) Hopefully any unique files fit on my empty 1TB drive.

Current status: Scanning....

KEY MESSAGE: Always have a hard disk monitor running and watch it! Especially when connecting drives that haven't been connected for a long time.

[wwcanoer]

Good news. Took three hours to re-scan 7 USB drives (10 million files in 21.8 TB) and found 415 GB of unique files on the failing 5TB drive. Small enough that I can copy them to the nvme drive on that computer. Now copying the the files.

Hard Disk Sentinel Pro reports 158 uncorrectable errors. Still 5% health. Hopefully no errors in the unique folders.

After it copies then I'll look at the created folders to see if they appear complete.

Last year (6/25/2022), I cataloged all of my drives in WinCatalog, so I can manually compare the copied folders to that record to see if I'm missing any important files that I need to try to recover. (It's possible that the originals are still on the computers that were backed up to this drive, unless I was simply freeing up space on them and only have the one copy. I can search WinCatalog to see if there's duplicates of those folders and where.)

This 5TB Seagate Backup Plus 2.5" USB drive was always a poor drive. So very slow. My two older 3GB WD MyPassport Ultra 2.5" drives always performed far far better that the newer one. [Well, right now it's reading at a consistent 80 MB/s second, so not too bad. 100% utilization on this USB HDD is only 5% utilization of the nvme drive that it's copying to. Advances. ]

[wwcanoer]

UGH. Copying froze at 209GB.
Tried pausing but that didn't complete.
Disconnected and reconnected. Now speed 3 MB/s. Now zero again. Health 3%
I'll stop and try to copy directly without the windows shell...

Now saving and exporting the file lists.

UGH! I just noticed that the default selection is "Don't create subfolders" which would make it impossible for me to review for missing files. So, will have to recopy the 200 GB...

Now 2% health... fingers crossed that it works this time. Copying at 140 MB/s...

[wwcanoer]

362 GB copied. 9,186 files could not be processed. (Check Log)

Problem: How do I find and try the failed files again?

They're listed in the text file log but not noted in the actual program, so I need to manually go through the text file or write my own program to retry the copying the files. I first need a clean list of files. So paste into excel and strip the excess text from each line. Maybe TeraCopy can read a text file?

I could manually pick out the top folders and use a sync program to try to copy the full folders, but then may waste time copying unnecessary files.

My log file is 1.3 million lines long, going back to 2022, and includes logs from several computers because it's stored in My Documents, which is a OneDrive folder, so every installation of DC writes to that same file. I'll rename it to start fresh.

Are all of the errors listed at the end or interspersed with the successful copies? Oh my. I guess that I need to copy the last 58,000 lines from the log file into excel and then filter on the error text... OK. Saved only the last operation to a text file. 57,000 lines is too long to paste into excel online. Installing excel locally... imported. UGH. There's breaks in the data every 1000 lines with a blank line and then a line with only date & time. So, need to delete those lines. Ok, took some work but I got a clean file list.

Copy the file list using:
TeraCopy.exe copy *C:\Temp\filelist.txt D:\Target

First MP4 file started at 81 MB/s then access denied error. Unfortunately, TeraCopy did not skip to the next file and pressing pause/skip/stop didn't do anything.

So, searched that file in WinCatalog. I have 27 copies of this 800MB video file! There are 3 different MD5 hashes for this file but this one has a blank MD5 hash, so it must have been corrupted back in 2022. What a mess to clean up.

TeraCopy also stopped on the second file. This isn't going to work for a 9,700 long file list. Shutting down. Reboot. Update to the latest version of TeraCopy. Two decades ago I would use excel VBA to copy the list of files but what are the chance I can remember how to do that within a reasonable time?

Maybe I scan the drive with WinCatalog and then search for files with blank MD5. If there's the same number, then corrupted. If less, then I need to try to copy or search if there's other copies.

I wish that Duplicate Cleaner could have given me a tab with the failed files to retry...

If I retry to the same directory, will it skip or replace existing files or give me a choice? (I don't want a good file to be overwritten with a now corrupted one.)

[wwcanoer]

Re-copying the files with Duplicate cleaner lead to 582 GB of files instead of 415 GB of marked remaining files, so I don't know what happened there.

When I ran TeraCopy again with the first problem files removed (that I have other copies of. DC presumably shows as unique files because corrupt file has no MD5 or different MD5), then it ran fine and copied & verified all but 29 files. Those remaining ones aren't important, so I'm done with that.

[DigitalVolcano]

Thanks for posting the story

A few notes-
-Duplicate Cleaner 5 now has an option (on the scan tab) to prevent the computer from sleeping during a long scan.
-V5.21 will now alert you to check the log when files/folders have been skipped during the scan (Either due to errors or user settings)
-Adding a column in the list which flags files which have failed deleting/copying/etc is now on the development list