ioFTPD v7.3.3 Released (Beta) - Page 5

Yil · 04-10-2010, 01:26 PM

Let me take the posts one at a time, starting from before my last post.

pion: I'm guessing the old version is at fault in the preloading taking a while problem. Look carefully at the logfile entries you showed. If preloading must be performed before the server is allowed to accept connections (DELAY=True mode) then the PRELOAD: lines will appear before the START: entry in the logfile. This is clearly true of your example so you need to set the DELAY option in VFS_Preload to FALSE. V7.0 defaulted this to TRUE if undefined and v7.1+ changed this to default to FALSE if undefined. So set it to FALSE explicitly to get the same behavior across those 2 versions. You can examine the logfile to make sure you get the correct behavior.

pion: The color warning error you are seeing probably is because you are running an older version. Perhaps to test with? V7.2+ changed all the theme color codes into letters and got put into Theme.ini instead of ioFTPD.ini so the line you are showing isn't possible. Also, the theme format changed between v6.9 and v7.0 to support sharing subthemes and the "+" sign was added to make sure an error just like the one you showed was displayed to indicate the definitions weren't updated. I'm guessing you tested v6.9 or something just to check behavior and thus you can ignore this error as it will only disable the theme and isn't a big deal.

Directory caching of entries in the mountfile do not decend into the directory tree, however they do have to process all immediate subdirs in order to get their sizes. Thus a large XVID folder will take a little bit of time because it has a lot of entries but a folder with just months like 01xx will be fast. The optional preloading section in the .ini file where you can specify where and to what depth to decend can be used to make it go deeper.

pion (and perhaps o_dog from earlier?): Take a look at that exchange. The FTP Client is at fault not the server!

(21:17:06) [Site2] PORT 1,1,1,1,165,166
(21:17:06) [Site2] 200 PORT command successful.
(21:17:06) [Site2] STOR my.sfv

No ABOR command was EVER sent, the client just kept trying to send new PORT commands and they should be rejected, in fact it even says so in the error reply! After like 20 seconds ioFTPD will timeout the attempt and it would probably continue working from that point. I forget which version of FlashFXP had this bug, but I know many earlier versions of the current beta did and it's possible other clients do/did as well. Newer, or older, versions of FlashFXP work fine and send the ABOR command before trying again...

pion: "Error converting string" errors were fixed in v7.1 when the TCL temporary internal name for the waitobject was increased in size to handle paths instead of just short names. Just use a newer version.

pion: That v7.0.3 lockup isn't that useful to me, but your next post with the minidump/crashlog might be. However, v7.1+ included a number of bugfixes that might have solved the reason for the problem, but I'll take a look at the crashlogs and get back to you if it's something new.

Zer0Racer · 04-10-2010, 01:32 PM

Please use pastebin.com

pion · 04-10-2010, 03:00 PM

The color warning was not the issue, it was the timestamp on the log for how long it took before login was possible.

Running through 10000 or even 100000 folders should really not take 5 minutes in any case afaik? Especially not with idle IO?

Ideas on where to put the blame for disappearing groups?

My first post describe a COMPLETE lock. I could log out and in, and not get a single transfer through, neither using PASV or PORT. After a log in/out, what the client did in a previous session should be completely irrelevant for the new session? This behaviour lasted for many minutes, through multiple sessions until I restarted the process manually.

As for the second post with similar behaviour, the interesting part is the preceeding commands before the process is refusing all connections. (Or that's what I figured anyway) due to 7.3.3 locking up the very same way

pion · 04-10-2010, 04:01 PM

I was able to reproduce the blocking of all transfers using io 7.3.3 now. I also issued 'site crashnow' but that didn't do anything other than blocking the current session and writing 04-10-2010 22:48:55 CRASHNOW: "pion" to log, forcing a reconnect. Just minutes before transfers where progressing just fine. Nothing at all has been done to the ftpd except transfers. It appeared that all transfers (4 different sessions) all stopped at once. There are entries in log about LOGIN, LOGOUT, NEWDIR, after the crashnow, but no file transfers. Debug.log also got written after site crashed: 04-10-2010 22:49:40 AsyncSelectCancel flags: 0

In fact, I tried 'site crashnow' three times, in three different sessions without any noticeable effect.

After about 5 minutes of this - the process locked up completely, not accepting any connections at all.

ioftpd-watch.exe also just sits there without stopping the daemon. Manual restart needed.

Yil · 04-10-2010, 05:54 PM

pion: Hmm, that's weird. Crashnow should cause the process to crash. It's a fatal error! If it doesn't something is seriously wrong, and usually it has been the loader lock getting stuck which is now tested for. Perhaps there is another reason, some sort of 3rd party module catching crashes, etc. I may write a simple connectivity tester in the next release similar to the loader lock tester just to keep an eye on things so it can restart if it gets stuck...

I'm about 50% of the way done writing the openSSL conversion and noticed something important. The Overlapped I/O structure is being re-used without being cleared. I failed to do this when I re-wrote the EXEC module and it caused problems and is now fixed, but this is original unchanged by me code deep inside the transfer logic. The important thing here is it's possible not clearing this causes the winsock library to freak out once in a rare while and is causing the lockup bugs. I think I'll put out a 7.3.4 release in a bit just to see if it makes a difference.

Important tidbit. ioFTPD pre-creates 10 sockets so it's ready to accept connections quickly. This means that even when ioFTPD gets stuck it's sometimes possible to connect to the server but not be able to login, or transfer files/listings over new data ports. However, since new sockets cannot be created to replace those 10 sockets it eventually stops accepting connections and you'll get timeouts instead of connections (which may or may not print anything). The only reason I mention this is because often times you'll see the server accept the connection and then timeout versus the server not accepting the connection at all and the client timing out the attempt. If you see the first start happening you can always try to login 10 times and see if the behavior changes which will tell us something about what is going on...

pion · 04-11-2010, 06:36 AM

Looking at the log, there's only 4 logins within the timeframe of the errors. 3 of those where after first 'site crashnow' event i log

Yil · 04-14-2010, 01:28 AM

Update: I got the openSSL version of ioFTPD with a heavily modified transfer engine to compile and do a key exchange. It seems I haven't figured out some of the trickier aspects of using the undocumented (but faster) non-copying interface though... I learned a bunch of things so I'll have to re-write part of it again, but I did restore full functionality to it without encryption which is actually a big step given all the changes. The new model handles the encryption in just 2 places instead of over lots of code so it should be easier once I figure out how it works...

I've been working really hard the last 7 days or so on this and assuming I can keep the pace for a few more there might be something to play with soon.

Yil · 04-15-2010, 04:08 PM

I found a really good reason why the server sometimes won't accept login connections on startup and it's a trivial mistake. I'm not sure what version introduced it but it was v6+ and therefore my bad. It obviously had serious consequences and because it's a race condition it's more likely on multi-core / faster machines. It's also possible this could explain some cases of the server not accepting logins after startup BUT still allowing logged in users to transfer files (i.e. not the bad lockup bug).

I'm not positive yet, but I think I found a bug in FlashFXP and FTPRush transferring files when uploading via the loopback interface. It's not too big of a deal when using SSL transfers, but it truncates the file when using unencrypted transfers. I couldn't believe BOTH of them wouldn't handle this correctly and so it's still possible it's my fault but ioFTPD to another copy of ioFTPD on the same machine via loopback is perfect... Needless to say a lot of hours were lost trying to track this down and then convincing myself they must be at fault and trying to prove that, but let's face it uploading via the loopback interface which means the same machine just isn't that common of a thing unless testing... It's possible this is a win7 x64 new "feature" but my research says this is allowed behavior by the windows library and a somewhat common programming bug.

Now I can get back trying to figure out the openSSL issues

pion · 04-16-2010, 09:37 AM

How soon do you think you'll have an openssl version or bugfix version ready? Crashing/lockup has gotten so bad that I'm actively checking in here multiple times a day for updates :P

Silly · 04-16-2010, 09:42 AM

lol..same here.....but since there is watch...i don't need reboot it manually that offen..

Flow · 04-16-2010, 02:09 PM

Dont stress the cook man. YouÂ´ll get bad food

pion · 04-16-2010, 02:23 PM

If you're hungry enough, bad food is better than no food!

monk- · 04-16-2010, 04:57 PM

Quote:

Originally Posted by Yil

I found a really good reason why the server sometimes won't accept login connections on startup and it's a trivial mistake. I'm not sure what version introduced it but it was v6+ and therefore my bad.

to be precise, version 693 has no hang on startup, where 701+ will
and i dont think multi-core is the problem, since i also have this problem on single-core's
and its also not an os dependend bug, cos its on xp too, with 2003
i still think the problem is in the dir caching
fyi

pion · 04-17-2010, 04:08 AM

# IDNT command restricted to use by these hosts. You may list up to 10 IP
# addresses or hostnames (i.e. BNC_HOST_10) without skipping numbers.
BNC_HOST_1 = 192.168.0.* <-- doesn't work, sais "doesnt resolve" in log
BNC_HOST_2 = my.dns.name <-- doesn't work, tested using ip instead of dns here, which worked fine

In dynamic ip scenarios, both cases would be a welcomed addition to this feature.

As a temp fix, how to disable the ip restriction alltogether? I tried just removing the BNC_HOST entries without much success.

Using io 7.3.3

Yil · 04-17-2010, 12:46 PM

pion: My assumption is that BNCs use static IPs. They are basically either local or on a shell account somewhere and in both cases are usually static. I could support using a DNS name and refresh the address after a "site rehash" if you want, but it certainly isn't going to look it up on demand for each connection because of the performance issue. Disabling the restriction is a really bad idea though as it opens up a security hole.