Selection assistant: very slow

[olafmar]

Hello,
the selection assistant is very slow with long lists of duplicates. I would expect it to be quite fast, since (e.g.) I'm going only to mark files according to the position and no pre processing should be required for that. "Slow" means I'm waiting minutes on a 2x1.6GHz C2D with a list of 80k+ files.

Minor feature request: if I don't check files for content no hash is calculated. A contextual menu item to calculate it manually on selected files would be useful.

[therube]

Quick test here ... Intel E4300, 2GB RAM

I could only come up with 13,980 dups in 5,597 groups, so not even close to your 80K, but Selecting All but one, or oldest, or largest, or ... Select by location; takes only one second for me.

How much RAM do you have?
Task Manager shows no other process taking high amounts of CPU?

Wonder if for some odd reason your A/V could be coming into play?

[DV]

No hash is currently calculated unless you are scanning the content (and not byte-by-byte mode).

The assistant can be slow with huge amounts of rows - it's a limitation of the listview grid. 80K shouldn't be too bad though - a few seconds maybe. How's the memory looking? Could be struggling if lots of apps running.

[olafmar]

Correction: Pentium D805 (dual 2.66GHz).
Memory: 2 GB RAM, no swapping at all, no other processes running, system just restarted.

With 35k files in 10k groups it takes between 1 and 5 seconds. So, almost nothing for small lists (but definitely noticeable, and I don't expect it to be), some seconds for 10k-30k groups or files, at least a couple of minutes for 80k files => the process is non linear. For the techie, the algorithm executed after I click "selection assistant" button (or context menu entry) scales as O^n, with n>1. You may want to check it. Whick kind of calculations are you executing while loading the dialog window?

therube> What do you mean by A/V? audio/video? Why should it? I have a gt9800
card, I don't think it would slow down a duplicate file finder

[DV]

The sel assistant window was calculating the selected filesizes from the list (slow).
In version 3.0 the file sizes and selection counts are totalled on the fly, so avoiding slow traversals of the list. Also the selection assistant window has been removed, replaced with a pop up menu, and live selection totals displayed next to the list

http://www.digitalvolcano.co.uk/duplica ... _files.png


A/V = Anti Virus. Good idea, but the sel assistant doesn't access the filesystem, only it's own internal list.

[olafmar]

In the screnshot I see the new method to select duplicates. It may be better for some users, but I think it misses something I was heavily relying on: manual correction of the path. I often select a file, go to the selection assistant, press the arrow to input the file path, then cut its last 1-3 levels and then mark the duplicates (subdirs is of course checked). This speeds up a lot the process if I notice that some paths are replicated interely. Of course I may look for files located at the level I care, but then I have to search for them. And sometimes there may not be any files at all there.

Another question: how do you replicate in v.3 the case where "include subdirs" is unchecked? It happened to me that some files are replicated in a directory and in some of its subdirs. In this case I always unchecked the checkbox and I marked the files in the subdirs. The options I see in the screenshot may not be able to offer this kind of flexibility.

The old assistant was VERY good and powerful (at least the "per position" tab), are you sure you want to change/remove it?

[DV]

All the functions of the old assistant are there, but pop up from the magic wand button in the screenshot . Mark by location and text are accessible from this, and you can still modify the path and subdirs, etc as usual.

[LuiFigo]

IF the selection assistant is really slow then go for a latest version of duplicate cleaner or you can also try the pro version.