Problem with Large Amounts of Files

[anthonyaykut]

Hi,

I have a directory with 750.000 files in it, which I would like to transfer part by part to my FTP server (ie files with a*.*, b*.*, etc). However, once I click on the directory - after a lot of read activity on the disk - the directory contents are not shown (shows up as an empty directory). Is there any way around this - this is proving to be a big problem for us.

Regards,
Anthony

[bigstar]

WOW! I'm not entirely sure where to begin. Many problems can arise from a single directory containing that many files.

First thing we need to consider is can windows own file browser display the files shown in this folder?

I'm not entirely sure where I read it but I recall seeing something that Microsoft recommends no more than 9000 files per folder otherwise it will encounter performance issues while opening and managing the folder.

The problem is that each file needs to be enumerated and displayed, and doing it for so many files requires a lot of memory and time.

Since FlashFXP uses the same functions as windows it has the same performance as windows, though a bit slower because we do a few more things that windows normally does.

Up until now the most files I've ever personally tested FlashFXP with is about 50k files.

I'm going to attempt to enter a folder containing 750,000 files and see what happens.

I'll post a reply once I conclude my test.

[bigstar]

I created a folder with 750k files, Attempted to enter this folder in FlashFXP after 30min and 500mb of memory usage I decided it was time to throw in the towel.

I decided it was time to try and open this folder in windows explorer, 10min and 177mb later windows explorer was able to display the folder content.

I ended up deleting 250k files with a simple app I wrote to re-test. I ended up leaving windows explorer open to this location without thinking and for the past 20min it's cpu usage has been stuck at 100%.

After restarting windows I decided to try this again, however I decided to test with 100k files as starting point. 100k files will display though the performance isn't as good as I'd like it, In the next few months I may attempt a rewrite of the shell enumeration objects and management.

Sadly you're not going to be able to use FlashFXP with a folder that contains 750k files, or even 150k, the performance and usability becomes extremely poor around 100k files though 100k files is usable.

I would have to recommend no more than 50k files per folder for sanity.

I will attempt to improve performance a few months down the road, however it would be very unlikely that we ever reach the point where handling 750k files in a single folder is a viable option.

[anthonyaykut]

Thanks for that - well I am working around the problem by working with batches of 50K files, and that works OK for now.