Directory listing

[tmgomez]

I have a 8TB ftp and some of my drives are 1TB drives I have many file folders in some of the drives. It used to take like a minute to list the directorys. Now that some of the drives are getting full it will time out on the directory list is there a way to make it cache the directory so it will list quicker?? its not a speed issue or a hardware issue it is the directory size. How can i resolve this listing problem?



Thanx, TM

[EwarWoo]

Have u tried using stat -al?
Dunno if it will make enough of a difference but worth a try at least.

[Yil]

I just totally re-wrote the listing code so I think I can give you a little insight :) This turned out a little long, but I think you'll appreciate the coverage of the issues involved...

ioFTPD when printing any directory listing shows you the size of FILES in immediate subdirectories of the current directory. To do this it opens up each subdir and caches it. Then when printing the list it just looks at the directory size field in the cache which reflects the sum of all the files in that directory. This of course won't count files in subdirectories of that directory though.

Thus if you have a very large directory and nothing in cache ioFTPD opens and caches all of those directories before it can print anything... That can be time consuming especially if the disk is heavily used since all the disk seeks required to retrieve the directory information from the filesystem are unavoidable and beyond ioFTPD control.

Unfortunately there isn't any listing option that will solve your problem. "LIST -1" just shows you the names of files/directories without any other information and in theory wouldn't need to know the size of subdirectories but both in the previous and my current version it computes them anyway as a side-effect...

I AM playing with the directory caching logic, but have you tried adjusting the DirectoryCache_Size option in ioFTPD.ini? 8 buckets (currently fixed) of 1000 (current value of option) entries are kept ordered and binary searched with the oldest entry (perhaps should be changed to least recently used?) dropped when space is needed in the bucket. This results in a max of 8k dirs cache. Just for kicks try 10000 and run around your filesystem and see how it responds, restarting the server between tries. There are arguments for and against a low bucket size, but my TODO list includes making this a ioFTPD.ini option so we can try it ourselves... I'm also considering turning this into a skiplist later on instead of an array to limit shuffling the array up/down all the time.

I have attempted to optimize a few things involved in generating a listing, but I haven't benchmarked it yet. Assuming it's still too slow I'll offer you some solutions and see what you and others think. These may or may not be possible as I haven't looked at the repercussions of skipping opening/caching subdirs, but I don't see any problems.

1) A new ioFTPD.ini option to turn OFF subdirectory size information totally. All directories will show up as 0 size in directory listings. For many other FTP servers this is the only option.

2) Only show directory size information for directories in cache, but display 0 (or something like 00 to indicate unknowns) for uncached directories.

3) Provide a new ioFTPD.ini option to turn off subdirectory size information for specific glob paths. I.e. /Dir1/really-large-dir/*/ would disable caching of subdirectories of really-large-dir ONLY but would cache subdirs of that directory again. Or /Dir1/* would disable all caching in that tree. Etc...

4) I could change the listing algorithm to manually recurse through subdirs and use some sort of timeout parameter to decide when to stop asking for directory size info and then revert to option #2 of returning 00 for the remaining unknowns.

I suppose it's possible to implement all 4 options since they aren't exclusive to each other. Without testing what returning 00 does to FTP clients (I don't think it should do anything wierd) options 3 and 4 would seem to confuse people if we were forced to return just 0.


Oh, and one more thing... Assuming your using FlashFXP have you adjusted the "Stop and resume if no data transfered for (minutes)" option under the Transfer tab of Preferences? I don't know if the applies to directory listings since they can't be resumed but worth a try...

FYI: FlashFXP will re-issue a plain LIST command after a timeout and in theory this would have a greater chance of success since a lot of the directories needed would already be cached this time... This isn't a good solution for end users who wouldn't know why they had to wait minutes to get a listing, but at least it might be a temp solution.