FlashFXP Forums - View Single Post

Yil · 04-25-2010, 02:13 PM

pion: I think you can ignore the debug.log messages. At the moment it looks like I goofed the test for when to output the AsyncSelectCancel line so I'm printing the un-interesting case. Doh!

The SSL routines complaining about SERVER_HELLO errors are almost certainly caused by transfers where someone didn't enable SSL so it can't perform the handshake right after connect. This is just the type of information I was looking for, but you can safely ignore anything in the GET_SERVER_HELLO category right now.

Regarding the slow start problem. I found 2 bugs.

I said it would process the VFS= file, if defined, and then do the Default.vfs file. It does that, but instead of processing the individually specified directories (i.e. 1 = /dir) only once, it does it for both. The second time should be fast since it's likely in cache, but it's extra work and if stuff gets pushed out of the cache it's a lot of extra work.

Remember when I mentioned that enabling DEBUG=True would cause the server to wait until finished, but it would use lots of threads to work in parallel so it would be faster? That is true only for the first time it's executed. That is correct behavior, BUT with the above bug the 2nd time is for Default.vfs if VFS= was defined...

For the moment try Delay=TRUE, don't define VFS=, and don't define any individual dirs with a depth of 1 already defined in the default VFS file as they should already be done. My guess is the only thing you really want is Delay=TRUE set and loading times should go down.

When the server comes up, login and use "site dircache". What you want to look for is how many buckets have the same size as the max size which I think defaults to 100. If you see 10% or so them at 100 then too much stuff is being forced out of the cache during preloading and we can either up the size of each bucket, or the number of buckets to distribute dirs over. I added this command for just this type of tuning. You can post the output of this command here and I'll examine it for you if you want. I'd be interested in it anyway.

Long term I think I might have an idea to help everyone. I think it's possible to write the current directory cache to disk during shutdown and read it back in on the next startup. Thus whatever was in cache and/or popular before will be cached now. Preloading would then work against an already loaded cache and just fill in the blanks. This isn't a simple change, but would be really useful for very large sites. The next change would be to use that same code, but to limit it to just write out a single dir whenever some threshold is past. So a directory like a /XVID dir would have a .iofTPD.cache file or something in it which could be loaded and provided the timestamps match it wouldn't have to examine all the subdirs themselves.

Both of those changes would actually create more work for the server, but it would be spread out and the impact wouldn't be felt by the user so much...

04-25-2010, 02:13 PM
Yil Too much time... Join Date: May 2005 Posts: 1,194	pion: I think you can ignore the debug.log messages. At the moment it looks like I goofed the test for when to output the AsyncSelectCancel line so I'm printing the un-interesting case. Doh! The SSL routines complaining about SERVER_HELLO errors are almost certainly caused by transfers where someone didn't enable SSL so it can't perform the handshake right after connect. This is just the type of information I was looking for, but you can safely ignore anything in the GET_SERVER_HELLO category right now. Regarding the slow start problem. I found 2 bugs. I said it would process the VFS= file, if defined, and then do the Default.vfs file. It does that, but instead of processing the individually specified directories (i.e. 1 = /dir) only once, it does it for both. The second time should be fast since it's likely in cache, but it's extra work and if stuff gets pushed out of the cache it's a lot of extra work. Remember when I mentioned that enabling DEBUG=True would cause the server to wait until finished, but it would use lots of threads to work in parallel so it would be faster? That is true only for the first time it's executed. That is correct behavior, BUT with the above bug the 2nd time is for Default.vfs if VFS= was defined... For the moment try Delay=TRUE, don't define VFS=, and don't define any individual dirs with a depth of 1 already defined in the default VFS file as they should already be done. My guess is the only thing you really want is Delay=TRUE set and loading times should go down. When the server comes up, login and use "site dircache". What you want to look for is how many buckets have the same size as the max size which I think defaults to 100. If you see 10% or so them at 100 then too much stuff is being forced out of the cache during preloading and we can either up the size of each bucket, or the number of buckets to distribute dirs over. I added this command for just this type of tuning. You can post the output of this command here and I'll examine it for you if you want. I'd be interested in it anyway. Long term I think I might have an idea to help everyone. I think it's possible to write the current directory cache to disk during shutdown and read it back in on the next startup. Thus whatever was in cache and/or popular before will be cached now. Preloading would then work against an already loaded cache and just fill in the blanks. This isn't a simple change, but would be really useful for very large sites. The next change would be to use that same code, but to limit it to just write out a single dir whenever some threshold is past. So a directory like a /XVID dir would have a .iofTPD.cache file or something in it which could be loaded and provided the timestamps match it wouldn't have to examine all the subdirs themselves. Both of those changes would actually create more work for the server, but it would be spread out and the impact wouldn't be felt by the user so much...