Multi-threading bottleneck while processing image files
Posted: Sun Nov 12, 2017 2:45 pm
Hello,
I am currently testing DC Pro compared to various other such applications and like what it has to offer. Before I used Alldup, but it lacked an option to only find files inside the same sub-folders of a base folder structure, which is possible in DC.
Now I run a Image mode scan of 15280 files to find photo similarities of "Very close match (97%)". During the "Calculating image metrics" phase CPU load varies between 6.25% (1 logical core fully utilized) and 50% (16 logical cores partially utilized by VCOMP120.DLL!vcomp_atomic_div_r8). It seems that the 6% (1 core) bottleneck keeps happening mostly when new files are read. Or rather, one specific thread keeps maxing out a single core nearly all the time and the presence/absence of other threads likely depend on the current status of said thread.
2116 6.24 3.649.458.312 clr.dll!GetMetaDataPublicInterfaceFromInternal+0x5330
Furthermore I noticed that DC's average CPU load for processing ORF (Olympus raw) and JPG files drops down to only about 18%, while it increases well over 30% when NEF (Nikon raw) files are processed. I don't know yet if the bottleneck thread drags down the former's average or if the higher resolution NEF files just need much more processing per file.
Overall I wonder: Would be possible to spread the work of that single thread to multiple threads to make more use of multi-core CPUs during that state of reading/decoding or whatever the thread is doing?
Thanks and best regards!
I am currently testing DC Pro compared to various other such applications and like what it has to offer. Before I used Alldup, but it lacked an option to only find files inside the same sub-folders of a base folder structure, which is possible in DC.
Now I run a Image mode scan of 15280 files to find photo similarities of "Very close match (97%)". During the "Calculating image metrics" phase CPU load varies between 6.25% (1 logical core fully utilized) and 50% (16 logical cores partially utilized by VCOMP120.DLL!vcomp_atomic_div_r8). It seems that the 6% (1 core) bottleneck keeps happening mostly when new files are read. Or rather, one specific thread keeps maxing out a single core nearly all the time and the presence/absence of other threads likely depend on the current status of said thread.
2116 6.24 3.649.458.312 clr.dll!GetMetaDataPublicInterfaceFromInternal+0x5330
Furthermore I noticed that DC's average CPU load for processing ORF (Olympus raw) and JPG files drops down to only about 18%, while it increases well over 30% when NEF (Nikon raw) files are processed. I don't know yet if the bottleneck thread drags down the former's average or if the higher resolution NEF files just need much more processing per file.
Overall I wonder: Would be possible to spread the work of that single thread to multiple threads to make more use of multi-core CPUs during that state of reading/decoding or whatever the thread is doing?
Thanks and best regards!