I run into these errors all the time when I try to scan 100GB+. Usually trying to delete or move the files I selected is when I run into the errors. I have 16GB physical RAM and a 32GB swap file.
The upcoming v3.1 should address all memory issues - it has a new database engine under the hood which should keep memory use low and stop fragmentation.
Hi. I've recently downloaded the free edition. I know this is the "Pro" section, but as this also happens in the Pro and I want to buy it, I guess this is the right place.
I'm getting the out of memory error. I have a not so small file system in my backup. Can't remember the exact number, but it is in the tens of thousands of files and around 1TB.
One thing I notice is that when the comparison ends, your software already has 1GB in memory. This may be due to the large amount of meta-data stored in memory, but it seems excessive.
Then, when the deletion processes starts, the memory occupied by the process starts to grow, a lot.
I can select <500 files and I won't get the error, but sometimes, 500 is too much.
I'm willing to make some tests with the free version and send you some debug info. I really like your software (selection assistant is great - just need a user defined hot-key) and prefer to spend some time helping ironing out some bugs, than try to find a better duplicate cleaner.
Regular Mode:
- Same Content - (in the progress window it refers MD5)
- Same File Name
- Don't Scan System Files/Folder
- Don't Follow NFTS Mountpoints and Junctions
File Filter - *.*
- Any Size
- Any Date
Scan Details:
122874/122874 Files Scanned (731 GB)
17691 Groups of Duplicates
32618 Files have Duplicates (74,8 GB)
Memory after scan: 153.824 KB
Then I marked 16903 files for deletion and initiated the File Removal process. The memory started rising... rising... with 900 files deleted I reach the 1GB memory. At 1.2GB memory I canceled the deletion process. The log was filled with messages like this:
Delete Error: Z:\path\0005.jpg : Exception of type 'System.OutOfMemoryException' was thrown.
I'm now looking at the list with still 15700 files marked for deletion and the process memory is still at 1.131.960 KB. The memory was not freed. CPU is at 50%.
I press the "more options" button and memory starts to fall. CPU rises to 90% and after a while the window shows, the memory stabilizes at 188.988 KB and an exception is thrown:
Unhandled exception.. bla bla bla ... Not enough storage is available to process this command.
Details:
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.ComponentModel.Win32Exception: Not enough storage is available to process this command
at System.Drawing.BufferedGraphicsContext.CreateCompatibleDIB(IntPtr hdc, IntPtr hpal, Int32 ulWidth, Int32 ulHeight, IntPtr& ppvBits)
at System.Drawing.BufferedGraphicsContext.CreateBuffer(IntPtr src, Int32 offsetX, Int32 offsetY, Int32 width, Int32 height)
at System.Drawing.BufferedGraphicsContext.AllocBuffer(Graphics targetGraphics, IntPtr targetDC, Rectangle targetRectangle)
at System.Drawing.BufferedGraphicsContext.AllocBufferInTempManager(Graphics targetGraphics, IntPtr targetDC, Rectangle targetRectangle)
at System.Drawing.BufferedGraphicsContext.Allocate(IntPtr targetDC, Rectangle targetRectangle)
at System.Windows.Forms.Control.WmPaint(Message& m)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
at System.Windows.Forms.ContainerControl.WndProc(Message& m)
at System.Windows.Forms.Form.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
-------------------------------------
I exported the duplicate list and the marked duplicates list. I can send it if you wish.
I started the removal process again and canceled it after it passed 800 MB. This time the log window did not appear, but memory was not freed and is at 1.1GB. CPU is at 60-75%. I now press the "File removal" button and memory start to get freed, with CPU at 90%. At 195MB it stops, CPU goes to 0% and the windows does not appear. I press it again and it appears. This time the exception is not thrown.
It seems that when memory reaches some point, which I believe to be around 1300-1500MB, files are no longer deleted, the log window will register all these failed deletions with the 'System.OutOfMemoryException' exception. Then, when I press one of the buttons, they'll spawn a window. This action will trigger the memory cleaning, but in the end the exception is thrown.
The memory issues in 3.0.x are due to
-The 'list' taking up a lot of memory when filled
-DC additionally keeping the details of files scanned in memory.
Version 3.1 (about to go in beta) has a re-written database-centered engine, and uses some different controls. The memory usage is better, so hopefully this won't be a problem going forward!
Please PM me if you are interested in trying the beta. Thanks!
