I have several external hard drives that I will de-duplicate on a dedicated computer.
I could use a Lenovo ThinkPad T430s i5-3320M with SSD, using it's USB 3.0 ports.
Could I dedupe significantly faster if I got a faster desktop?
What would I need without getting unneeded capacity? (buying a used one)
I see posts saying that Duplicate Cleaner never uses more than 10 to 20% CPU even on a fast computer with fast SSD. Is this true? What's the bottleneck?
(1) What CPU is optimum?
- Will an i7 be significantly faster than an i5?
- Will an i5 gen 6 or 7 be significantly faster than an i5 gen 3?
(Obviously bench tests would be faster but does it matter for this application?)
(2) Is it worth removing the bare drives from my external drives?
- Will 3.5" USB 3.0 MyBooks be faster if I connect it directly to SATA in a desktop?
- Will 2.5" USB 3.0/3.1 WD MyPassport/Seagate BackupPlus Slim external drives be faster with SATA?
(assuming they have removable controllers and aren't encrypted.)
(3) Anything else to increase the speed?
Thanks in advance.
System for max speed?
Re: System for max speed?
I just tried DV version 4.1.4 on my SSD drive on T530 i5-3320M 2.6GHz
Running windows on msata drive.
Scanning windows partition on different 2.5" SSD drive that's not in use, 350,000 files, 43GB
DV ran at 30 to 36% CPU utilization.
Both SSDs have low utilization and speed during duplicate checks. Was higher when listing files. (Scanning a windows drive that I'm not using, 350,000 files, 43GB)
Says DV power usage very high red.
These measures say that my 10 year old computer isn't a bottleneck, but are there other restrictions in this old computer that slow it down? Would a newer 5 year old computer deduplicate significantly faster? (I can't afford a recent one but a gen 6 or 7 desktop or T570 are possible IF they would really speed things up.)
What do you think?
Running windows on msata drive.
Scanning windows partition on different 2.5" SSD drive that's not in use, 350,000 files, 43GB
DV ran at 30 to 36% CPU utilization.
Both SSDs have low utilization and speed during duplicate checks. Was higher when listing files. (Scanning a windows drive that I'm not using, 350,000 files, 43GB)
Says DV power usage very high red.
These measures say that my 10 year old computer isn't a bottleneck, but are there other restrictions in this old computer that slow it down? Would a newer 5 year old computer deduplicate significantly faster? (I can't afford a recent one but a gen 6 or 7 desktop or T570 are possible IF they would really speed things up.)
What do you think?
Re: System for max speed?
Running with the database, or the files to be compared, on an NVME drive will significantly speed things up vs SATA or USB 3? Or not?
-
DigitalVolcano
- Site Admin
- Posts: 1735
- Joined: Thu Jun 09, 2011 10:04 am
Re: System for max speed?
The limiting factor is usually the drive speed. If you have things cached (by the program) or as a virtual folder things should move very fast.
The scan settings can affect speed too. 'Similar' searches are very slow, and byte-to-byte mode can be slower too.
The scan settings can affect speed too. 'Similar' searches are very slow, and byte-to-byte mode can be slower too.
Re: System for max speed?
Thank you. I went ahead and bought an i7 gen 8 machine to bring me 5 years into the future from my i5 gen 3. I will try running a dedup analysis on the gen 3 and then repeat it on the gen 8, both from the drive and a virtual folder, to learn how much of a difference it will make.