Go Back   FlashFXP Forums > > > >

ioFTPD General New releases, comments, questions regarding the latest version of ioFTPD.

Reply
 
Thread Tools Rate Thread Display Modes
Old 04-10-2010, 01:26 PM   #61
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Let me take the posts one at a time, starting from before my last post.

pion: I'm guessing the old version is at fault in the preloading taking a while problem. Look carefully at the logfile entries you showed. If preloading must be performed before the server is allowed to accept connections (DELAY=True mode) then the PRELOAD: lines will appear before the START: entry in the logfile. This is clearly true of your example so you need to set the DELAY option in VFS_Preload to FALSE. V7.0 defaulted this to TRUE if undefined and v7.1+ changed this to default to FALSE if undefined. So set it to FALSE explicitly to get the same behavior across those 2 versions. You can examine the logfile to make sure you get the correct behavior.

pion: The color warning error you are seeing probably is because you are running an older version. Perhaps to test with? V7.2+ changed all the theme color codes into letters and got put into Theme.ini instead of ioFTPD.ini so the line you are showing isn't possible. Also, the theme format changed between v6.9 and v7.0 to support sharing subthemes and the "+" sign was added to make sure an error just like the one you showed was displayed to indicate the definitions weren't updated. I'm guessing you tested v6.9 or something just to check behavior and thus you can ignore this error as it will only disable the theme and isn't a big deal.

Directory caching of entries in the mountfile do not decend into the directory tree, however they do have to process all immediate subdirs in order to get their sizes. Thus a large XVID folder will take a little bit of time because it has a lot of entries but a folder with just months like 01xx will be fast. The optional preloading section in the .ini file where you can specify where and to what depth to decend can be used to make it go deeper.

pion (and perhaps o_dog from earlier?): Take a look at that exchange. The FTP Client is at fault not the server!

(21:17:06) [Site2] PORT 1,1,1,1,165,166
(21:17:06) [Site2] 200 PORT command successful.
(21:17:06) [Site2] STOR my.sfv

No ABOR command was EVER sent, the client just kept trying to send new PORT commands and they should be rejected, in fact it even says so in the error reply! After like 20 seconds ioFTPD will timeout the attempt and it would probably continue working from that point. I forget which version of FlashFXP had this bug, but I know many earlier versions of the current beta did and it's possible other clients do/did as well. Newer, or older, versions of FlashFXP work fine and send the ABOR command before trying again...

pion: "Error converting string" errors were fixed in v7.1 when the TCL temporary internal name for the waitobject was increased in size to handle paths instead of just short names. Just use a newer version.

pion: That v7.0.3 lockup isn't that useful to me, but your next post with the minidump/crashlog might be. However, v7.1+ included a number of bugfixes that might have solved the reason for the problem, but I'll take a look at the crashlogs and get back to you if it's something new.
Yil is offline   Reply With Quote
Old 04-10-2010, 01:32 PM   #62
Zer0Racer
Senior Member
ioFTPD Scripter
 
Join Date: Oct 2002
Posts: 703
Default

Please use pastebin.com
Zer0Racer is offline   Reply With Quote
Old 04-10-2010, 03:00 PM   #63
pion
Senior Member
 
Join Date: Feb 2006
Posts: 138
Default

The color warning was not the issue, it was the timestamp on the log for how long it took before login was possible.

Running through 10000 or even 100000 folders should really not take 5 minutes in any case afaik? Especially not with idle IO?

Ideas on where to put the blame for disappearing groups?

My first post describe a COMPLETE lock. I could log out and in, and not get a single transfer through, neither using PASV or PORT. After a log in/out, what the client did in a previous session should be completely irrelevant for the new session? This behaviour lasted for many minutes, through multiple sessions until I restarted the process manually.

As for the second post with similar behaviour, the interesting part is the preceeding commands before the process is refusing all connections. (Or that's what I figured anyway) due to 7.3.3 locking up the very same way

Last edited by pion; 04-10-2010 at 03:40 PM.
pion is offline   Reply With Quote
Old 04-10-2010, 04:01 PM   #64
pion
Senior Member
 
Join Date: Feb 2006
Posts: 138
Default

I was able to reproduce the blocking of all transfers using io 7.3.3 now. I also issued 'site crashnow' but that didn't do anything other than blocking the current session and writing 04-10-2010 22:48:55 CRASHNOW: "pion" to log, forcing a reconnect. Just minutes before transfers where progressing just fine. Nothing at all has been done to the ftpd except transfers. It appeared that all transfers (4 different sessions) all stopped at once. There are entries in log about LOGIN, LOGOUT, NEWDIR, after the crashnow, but no file transfers. Debug.log also got written after site crashed: 04-10-2010 22:49:40 AsyncSelectCancel flags: 0

In fact, I tried 'site crashnow' three times, in three different sessions without any noticeable effect.

After about 5 minutes of this - the process locked up completely, not accepting any connections at all.

ioftpd-watch.exe also just sits there without stopping the daemon. Manual restart needed.
pion is offline   Reply With Quote
Old 04-10-2010, 05:54 PM   #65
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

pion: Hmm, that's weird. Crashnow should cause the process to crash. It's a fatal error! If it doesn't something is seriously wrong, and usually it has been the loader lock getting stuck which is now tested for. Perhaps there is another reason, some sort of 3rd party module catching crashes, etc. I may write a simple connectivity tester in the next release similar to the loader lock tester just to keep an eye on things so it can restart if it gets stuck...

I'm about 50% of the way done writing the openSSL conversion and noticed something important. The Overlapped I/O structure is being re-used without being cleared. I failed to do this when I re-wrote the EXEC module and it caused problems and is now fixed, but this is original unchanged by me code deep inside the transfer logic. The important thing here is it's possible not clearing this causes the winsock library to freak out once in a rare while and is causing the lockup bugs. I think I'll put out a 7.3.4 release in a bit just to see if it makes a difference.

Important tidbit. ioFTPD pre-creates 10 sockets so it's ready to accept connections quickly. This means that even when ioFTPD gets stuck it's sometimes possible to connect to the server but not be able to login, or transfer files/listings over new data ports. However, since new sockets cannot be created to replace those 10 sockets it eventually stops accepting connections and you'll get timeouts instead of connections (which may or may not print anything). The only reason I mention this is because often times you'll see the server accept the connection and then timeout versus the server not accepting the connection at all and the client timing out the attempt. If you see the first start happening you can always try to login 10 times and see if the behavior changes which will tell us something about what is going on...
Yil is offline   Reply With Quote
Old 04-11-2010, 06:36 AM   #66
pion
Senior Member
 
Join Date: Feb 2006
Posts: 138
Default

Looking at the log, there's only 4 logins within the timeframe of the errors. 3 of those where after first 'site crashnow' event i log
pion is offline   Reply With Quote
Old 04-14-2010, 01:28 AM   #67
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Update: I got the openSSL version of ioFTPD with a heavily modified transfer engine to compile and do a key exchange. It seems I haven't figured out some of the trickier aspects of using the undocumented (but faster) non-copying interface though... I learned a bunch of things so I'll have to re-write part of it again, but I did restore full functionality to it without encryption which is actually a big step given all the changes. The new model handles the encryption in just 2 places instead of over lots of code so it should be easier once I figure out how it works...

I've been working really hard the last 7 days or so on this and assuming I can keep the pace for a few more there might be something to play with soon.
Yil is offline   Reply With Quote
Old 04-15-2010, 04:08 PM   #68
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

I found a really good reason why the server sometimes won't accept login connections on startup and it's a trivial mistake. I'm not sure what version introduced it but it was v6+ and therefore my bad. It obviously had serious consequences and because it's a race condition it's more likely on multi-core / faster machines. It's also possible this could explain some cases of the server not accepting logins after startup BUT still allowing logged in users to transfer files (i.e. not the bad lockup bug).

I'm not positive yet, but I think I found a bug in FlashFXP and FTPRush transferring files when uploading via the loopback interface. It's not too big of a deal when using SSL transfers, but it truncates the file when using unencrypted transfers. I couldn't believe BOTH of them wouldn't handle this correctly and so it's still possible it's my fault but ioFTPD to another copy of ioFTPD on the same machine via loopback is perfect... Needless to say a lot of hours were lost trying to track this down and then convincing myself they must be at fault and trying to prove that, but let's face it uploading via the loopback interface which means the same machine just isn't that common of a thing unless testing... It's possible this is a win7 x64 new "feature" but my research says this is allowed behavior by the windows library and a somewhat common programming bug.

Now I can get back trying to figure out the openSSL issues
Yil is offline   Reply With Quote
Old 04-16-2010, 09:37 AM   #69
pion
Senior Member
 
Join Date: Feb 2006
Posts: 138
Default

How soon do you think you'll have an openssl version or bugfix version ready? Crashing/lockup has gotten so bad that I'm actively checking in here multiple times a day for updates :P
pion is offline   Reply With Quote
Old 04-16-2010, 09:42 AM   #70
Silly
Junior Member
ioFTPD Registered User
 
Join Date: Oct 2003
Posts: 27
Default

lol..same here.....but since there is watch...i don't need reboot it manually that offen..
Silly is offline   Reply With Quote
Old 04-16-2010, 02:09 PM   #71
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default

Dont stress the cook man. You´ll get bad food
Flow is offline   Reply With Quote
Old 04-16-2010, 02:23 PM   #72
pion
Senior Member
 
Join Date: Feb 2006
Posts: 138
Default

If you're hungry enough, bad food is better than no food!
pion is offline   Reply With Quote
Old 04-16-2010, 04:57 PM   #73
monk-
Member
 
Join Date: Oct 2007
Posts: 32
Default

Quote:
Originally Posted by Yil View Post
I found a really good reason why the server sometimes won't accept login connections on startup and it's a trivial mistake. I'm not sure what version introduced it but it was v6+ and therefore my bad.
to be precise, version 693 has no hang on startup, where 701+ will
and i dont think multi-core is the problem, since i also have this problem on single-core's
and its also not an os dependend bug, cos its on xp too, with 2003
i still think the problem is in the dir caching
fyi

Last edited by monk-; 04-16-2010 at 05:51 PM.
monk- is offline   Reply With Quote
Old 04-17-2010, 04:08 AM   #74
pion
Senior Member
 
Join Date: Feb 2006
Posts: 138
Default

# IDNT command restricted to use by these hosts. You may list up to 10 IP
# addresses or hostnames (i.e. BNC_HOST_10) without skipping numbers.
BNC_HOST_1 = 192.168.0.* <-- doesn't work, sais "doesnt resolve" in log
BNC_HOST_2 = my.dns.name <-- doesn't work, tested using ip instead of dns here, which worked fine

In dynamic ip scenarios, both cases would be a welcomed addition to this feature.

As a temp fix, how to disable the ip restriction alltogether? I tried just removing the BNC_HOST entries without much success.

Using io 7.3.3
pion is offline   Reply With Quote
Old 04-17-2010, 12:46 PM   #75
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

pion: My assumption is that BNCs use static IPs. They are basically either local or on a shell account somewhere and in both cases are usually static. I could support using a DNS name and refresh the address after a "site rehash" if you want, but it certainly isn't going to look it up on demand for each connection because of the performance issue. Disabling the restriction is a really bad idea though as it opens up a security hole.
Yil is offline   Reply With Quote
Reply

Tags
application, bug, download, fix, nxmydb

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 03:53 AM.

Parts of this site powered by vBulletin Mods & Addons from DragonByte Technologies Ltd. (Details)