Go Back   FlashFXP Forums > >

General Discussion Need help? Have a problem? Let us help you. Bug reports and feature requests should be made using the Bug Tracker or Feature Tracker

Closed Thread
 
Thread Tools Rate Thread Display Modes
Old 01-10-2008, 10:18 AM   #1
anthonyaykut
Junior Member
FlashFXP Registered User
 
Join Date: Jun 2007
Posts: 2
Default Problem with Large Amounts of Files

Hi,

I have a directory with 750.000 files in it, which I would like to transfer part by part to my FTP server (ie files with a*.*, b*.*, etc). However, once I click on the directory - after a lot of read activity on the disk - the directory contents are not shown (shows up as an empty directory). Is there any way around this - this is proving to be a big problem for us.

Regards,
Anthony
anthonyaykut is offline  
Old 01-10-2008, 02:17 PM   #2
bigstar
FlashFXP Developer
FlashFXP Administrator
ioFTPD Beta Tester
 
bigstar's Avatar
 
Join Date: Oct 2001
Posts: 8,012
Default

WOW! I'm not entirely sure where to begin. Many problems can arise from a single directory containing that many files.

First thing we need to consider is can windows own file browser display the files shown in this folder?

I'm not entirely sure where I read it but I recall seeing something that Microsoft recommends no more than 9000 files per folder otherwise it will encounter performance issues while opening and managing the folder.

The problem is that each file needs to be enumerated and displayed, and doing it for so many files requires a lot of memory and time.

Since FlashFXP uses the same functions as windows it has the same performance as windows, though a bit slower because we do a few more things that windows normally does.

Up until now the most files I've ever personally tested FlashFXP with is about 50k files.

I'm going to attempt to enter a folder containing 750,000 files and see what happens.

I'll post a reply once I conclude my test.
bigstar is offline  
Old 01-10-2008, 04:16 PM   #3
bigstar
FlashFXP Developer
FlashFXP Administrator
ioFTPD Beta Tester
 
bigstar's Avatar
 
Join Date: Oct 2001
Posts: 8,012
Default

I created a folder with 750k files, Attempted to enter this folder in FlashFXP after 30min and 500mb of memory usage I decided it was time to throw in the towel.

I decided it was time to try and open this folder in windows explorer, 10min and 177mb later windows explorer was able to display the folder content.

I ended up deleting 250k files with a simple app I wrote to re-test. I ended up leaving windows explorer open to this location without thinking and for the past 20min it's cpu usage has been stuck at 100%.

After restarting windows I decided to try this again, however I decided to test with 100k files as starting point. 100k files will display though the performance isn't as good as I'd like it, In the next few months I may attempt a rewrite of the shell enumeration objects and management.

Sadly you're not going to be able to use FlashFXP with a folder that contains 750k files, or even 150k, the performance and usability becomes extremely poor around 100k files though 100k files is usable.

I would have to recommend no more than 50k files per folder for sanity.

I will attempt to improve performance a few months down the road, however it would be very unlikely that we ever reach the point where handling 750k files in a single folder is a viable option.
bigstar is offline  
Old 01-16-2008, 06:10 AM   #4
anthonyaykut
Junior Member
FlashFXP Registered User
 
Join Date: Jun 2007
Posts: 2
Default

Thanks for that - well I am working around the problem by working with batches of 50K files, and that works OK for now.
anthonyaykut is offline  
Closed Thread

Tags
directory, files, ftp, part, problem

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 09:05 PM.

Parts of this site powered by vBulletin Mods & Addons from DragonByte Technologies Ltd. (Details)