Hardware Acceleration options for DC5 Video comparison function.

The best solution for finding and removing duplicate files.
Post Reply
timerrington
Posts: 1
Joined: Sat Sep 04, 2021 5:52 am

Hardware Acceleration options for DC5 Video comparison function.

Post by timerrington »

Dear Developers.

I am a happy long term user of version Duplicate Cleaner Pro 4 and was a beta tester/user for V5.

First of all - Version 5 seems great - and I love the inclusion of the ability to compare video files... but it is very slow if there are a lot of video files.

Just wondering if there is any way to enable users to access FFmpeg's hardware acceleration functions to make the comparisons faster??

eg for NVIDIA: https://docs.nvidia.com/video-technolog ... index.html

I am not an expert by any means, but FFmpeg does not seem to be able to be compiled in a way to make using Hardware Acceleration the default action.

I would be quite happy trying my hand at compiling my own versions of FFmpeg, FFplay and FFprobe to include the required functionality, but there does not (currently) seem to be a way to get Duplicate Cleaner to use/substitute the required command options to FFmpeg when the decoding is taking place.

Any ideas, suggestions?

Thanks for Listening.
User avatar
DigitalVolcano
Site Admin
Posts: 1717
Joined: Thu Jun 09, 2011 10:04 am

Re: Hardware Acceleration options for DC5 Video comparison function.

Post by DigitalVolcano »

You can of course swap out the FFmpeg exe build in the program folder, but the FFmpeg command line paramters are hardcoded currently.

I plan to have user-editable commands, which will help with adding HW acceleration and should also open the doors to all sorts of customization and improvements over the original settings
schmosef
Posts: 1
Joined: Fri Jun 24, 2022 8:29 am

Re: Hardware Acceleration options for DC5 Video comparison function.

Post by schmosef »

Any progress on adding settings to enable hardware acceleration?

If not, do you have a link to a version of FFMPEG with NVENC enabled?

The video comparison feature is amazing but it's painful to use with a lot of files due to the slow speed.
Last edited by schmosef on Wed Jun 29, 2022 2:17 am, edited 2 times in total.
aeciolemos
Posts: 8
Joined: Thu Nov 24, 2022 7:53 am

Re: Hardware Acceleration options for DC5 Video comparison function.

Post by aeciolemos »

I see no furher replies for this topic. Is there any progress in using GPU?

For image comparisson, it should help as well. I am currently testing DCPro5 and comparing to the current software I use, Duplicate & Similar Photo Cleaner and I see a big difference in usage of resources.
Image
The other software uses GPU much more AND it compares photos and videos in the same scan. I also noted how slow it DCP5 is in comparisson
Image
As you can see, in the first 18 minutes, DCP5 found 628 files while DSPC found 38,607.
The reason I want to use DCP5 is that DSPC does not have options when it comes to selecting what you want to keep or delete. It randomly selects images/videos and does not allow you to protect a drive/folder or really anything else. The result is that I have to go through the duplicates list manually and delete one by one, which could take months in a huge collection.

I currently have about 150K photos and videos on my mapped GDrive and I want to eliminate duplicates that are scattered over other drives as well as find images that are NOT on the GDrive, which should be the single source of truth.

Lastly, as per another topic, DSPC caches the results during the scan, so if the app locks up or the computer locks up, I can resume from the last point it stopped. I can close the app and come back and the results are there, cached automatically.

I look forward to seeing DCP5 use more of the computer's resources.
By the way, my internet connection is a Gigabit connection, so pulling from GDrive is by no means slow. The one time I let DCP5 scan the entire collection, it took almost 2 days while DSPC takes about 4 hours.
pnachtwey
Posts: 15
Joined: Thu Apr 15, 2021 9:44 pm

Re: Hardware Acceleration options for DC5 Video comparison function.

Post by pnachtwey »

The OP is asking for a lot. The problem is that there are many types of GPU although there are two main camps. AMD and NVIDIA. Writing code for these is a challenge because they are yet another instruction set and development system. The GPUs won't help much if the files are all on disk. The disk is the limiting factor. When I do video processing I always copy my files to a NVNE USB stick with a PCI-E interface for speed. When I look for duplicates, the files are on remove drives like a NAS were GPU will not help if limited by Ethernet speed.
Post Reply