Are there any scan size limits?
Posted: Tue May 28, 2024 1:20 pm
I'm trying to run a simple scan (video same length) across shares on two NAS servers.
The file count of the "reading metadata" part of the scan stops incrementing at around 10% of the files. I've left it running for a few days and it doesn't change or complete. If I filter to less files, it completes quite quickly in a few hours.
The details are:
DC Pro 5.22.0 running on:
Windows 11 Pro Workstation with 22 core / 44 thread Xeon E5-2696v4 cpu, 256GB ECC ram, NVME drives, NVIDIA Quadro M2000 workstation GPU, 5Gbps network.
Scan location is 4 parent folders on 2 shares on 2 NAS servers. Total size of all shares is 338TiB of which 230TiB is in use with 30.6million files. The filtered scan correctly discovers 230K files (196TiB), but the "reading metadata" doesn't get beyond around 18,500 files (8%).
The DC5 process still shows activity i.e. <1% CPU (around 20 threads), ~180MB mem and up to 25% GPU usage (similar to when it completes properly).
If I filter by selecting fewer files e.g. 61K files over 130TiB, the scan completes in a few hours.
The file count of the "reading metadata" part of the scan stops incrementing at around 10% of the files. I've left it running for a few days and it doesn't change or complete. If I filter to less files, it completes quite quickly in a few hours.
The details are:
DC Pro 5.22.0 running on:
Windows 11 Pro Workstation with 22 core / 44 thread Xeon E5-2696v4 cpu, 256GB ECC ram, NVME drives, NVIDIA Quadro M2000 workstation GPU, 5Gbps network.
Scan location is 4 parent folders on 2 shares on 2 NAS servers. Total size of all shares is 338TiB of which 230TiB is in use with 30.6million files. The filtered scan correctly discovers 230K files (196TiB), but the "reading metadata" doesn't get beyond around 18,500 files (8%).
The DC5 process still shows activity i.e. <1% CPU (around 20 threads), ~180MB mem and up to 25% GPU usage (similar to when it completes properly).
If I filter by selecting fewer files e.g. 61K files over 130TiB, the scan completes in a few hours.