Go Back   FlashFXP Forums > > > >

ioFTPD General New releases, comments, questions regarding the latest version of ioFTPD.

Reply
 
Thread Tools Rating: Thread Rating: 3 votes, 3.67 average. Display Modes
Old 08-06-2010, 11:55 AM   #106
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Check the [Virtual_Dirs] section of your ioFTPD.ini file. The /Search script allows you to enter dynamic queries to the nxTools dupe DB and get results you can click on back. It was a proof of concept script I wrote to test an early version of pure virtual directories but it should still work and is pretty useful. To enable it just follow the directions in the .ini file to copy the .itcl script file into your /scripts dir to activate it.
Yil is offline   Reply With Quote
Old 08-08-2010, 04:42 PM   #107
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default

Hi Yil!, whats juicy stuff is on the upcoming 7.6.0 ?. Feel like a milestone to me
Flow is offline   Reply With Quote
Old 08-08-2010, 05:39 PM   #108
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

I haven't had a lot of time to play with the server between work and StarCraft 2

Post #80 in this thread has a preview of the features for the next release. The re-write of the async event stuff is tricky to get right and took a bit of work. I seem to remember I had a working prototype but hadn't tested it much.
Yil is offline   Reply With Quote
Old 08-09-2010, 01:55 AM   #109
opcode
Junior Member
 
Join Date: Aug 2009
Posts: 21
Default

I can report that ioFTPD is randomly shutting itself down here aswell (7.5.9.8). Used to run 7.5.8 before and had no such problems. All i can find in the log is

Quote:
08-09-2010 01:44:37 No services appear to be online! Defined=1, Active=1, Online=0, Failed=0!
08-09-2010 01:45:37 Services_Test: Failed to connect to service 'FTP_Service' (IP=127.0.0.1) 3 times in a row!
08-09-2010 01:45:37 All services report failed status! Defined=1, Active=1, Online=0, Failed=1. Exiting!
08-09-2010 01:46:42 8 clients failed to finish cleanly during shutdown grace period.
08-09-2010 01:47:42 8 clients must be zombies - terminating process.
I don't run ioFTPD as a service (just starting it manual by executing ioftpd.exe) and tried with restart_on_deadlock = true and with the setting commented out. Both resulted in the same behaviour aka the ioftpd.exe process terminating itself every 1-2 hours.

Edit: More info! When doing "site who" i see the following:
Quote:
[2] 200-|?CID: 2 User: [Logging In] Idle: 22:40s|
[2] 200-| Action: |
[2] 200-|-------------------------------------------------------------------------|
[2] 200-|?CID: 3 User: [Logging In] Idle: 20:39s|
[2] 200-| Action: |
[2] 200-|-------------------------------------------------------------------------|
[2] 200-|?CID: 4 User: [Logging In] Idle: 16:39s|
[2] 200-| Action: |
[2] 200-|-------------------------------------------------------------------------|
[2] 200-|?CID: 5 User: [Logging In] Idle: 12:39s|
[2] 200-| Action: |
Could this be ioFTPD trying to verify it's own existence by logging in? There is nothing in Error.log about failed logins, so it can't be some other (real) user i guess.

Last edited by opcode; 08-09-2010 at 03:54 AM.
opcode is offline   Reply With Quote
Old 08-09-2010, 01:23 PM   #110
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

opcode: That's sort of interesting. The "classic" lockup bug where the loader lock gets compromised should be easily detected now and had that been the problem the server would report that and try to exit. Unless Restart_On_Deadlock was true it couldn't actually exit/terminate because it's so badly locked up though. So if you didn't enable the Restart_On_Deadlock feature and the process exited after deciding things were broken it isn't the "classic" lockup bug - which the logfiles would seem to support.

In that case, you have an example of the connection lockup bug which sorta looks the same but is far less deadly. I haven't see too many of these because there wasn't a good way to tell them apart before but for some reason winsock isn't happy and things stop working correctly but the "process" isn't corrupted so it can exit if it wants to.

Regarding "site who" (I think you meant "site swho"). You are correct that the server now connects to itself fairly often. Roughly every 30 seconds or so since that reduces the failure detection window to under 2 minutes since it must fail 3 times in a row. However, these connections should be immediately recognized and are thus never considered clients so they don't show up anywhere. It's possible under some serious lag/swap situation (more then 30 seconds without letting ioFTPD any CPU time) that one could slip through, but 3 in a row seems hard to do unless you had a runaway process on a single processor machine or something...

More than likely you just had a regular user trying to connect and because the server was hosed they couldn't actually log in or get disconnected... Since they got stuck no login or failure error messages can be generated so it's not surprising you didn't see anything in the logfile, however the login timer is set to like 15 seconds by default so it's really surprising to me to see client login entries with 20+ minute idle times. That must mean the server couldn't close the connection which isn't surprising, but I hadn't had confirmation of before...

Can you confirm the 2 releases where you did and did not see this problem? The 7.5.8 to 7.5.9 didn't change anything except the auto-ban logic... The 7.5.8 release relaxed the requirement for shutting down from 1 "offline" (a failed local connection attempt) service to all services marked as "failed" (3-in-a-row failed local connection attempts). This should mean that 7.5.8 is far more forgiving than 7.5.0-7.5.7 where this feature was first introduced.

Prior to upgrading had you been having lockup issues and just restarting it manually? Any trouble logging in at all? If there are problems does file transfer still work fine or does it have issues as well? Any scripts/extensions in use?

Looking forward, we can try defining a 2nd local only service and see if that makes a difference since the server may take longer to decide that service is locked up giving a greater window to look at what is wrong...

The v7.6 async event update may fix this issue entirely. Or at least, that's the hope...

Update: You can try running this as a service since it is exiting and that way you get auto-restart. Should be far less impact if you really are having it exit every couple of hours. How many users on average?
Yil is offline   Reply With Quote
Old 08-10-2010, 06:22 AM   #111
opcode
Junior Member
 
Join Date: Aug 2009
Posts: 21
Default

You are right about "site swho". i forgot that i set "site who" to be an alias of "site swho", because it gives me more information.
I have a site command "site ver" that just shows a a textfile with the current version (so i always know which version i have atm) and the .bak of that file says 7.5.8 so i am fairly confident that i updated from 7.5.8. to 7.5.9.

I only use ioFTPD for personal downloads (auto sfv check, mp3/imdb sorting etc is just soo much more convenient) and to be able to easily send stuff to friends. So usually there are about 3 logins with different uploads from me and maybe 1 other user downloading something. My friends usually ask me if they can download smth, so i am rather sure noone else was using the ftpd at that moment. Before upgrading to 7.5.9. i never had a single crash like that before, and i've been using io for years now.
Also the server is usually not under heavy load or any load at all, because my transfers often fail then (when the load is too high it often can't setup the connection in time making my transfers either timeout on that file and hang or even worse fail that file, then try to transfer the others which all fail with an "active adress is currently in use" error. which leaves me with a whole mess of failed files and stuff). Long story short, i try to not have any load on the box while transferring. Also i've seen the ftpd die while browsing on the box, and it did not have any significant load/slowdown at that moment.

The only problems i have when logging in is usually after i start ioFTPD. It often fails the ident check for some reason. If i leave the pc for like 5 mins and then try again it works. Usually connecting via localhost works first, then connecting via the dyndns hostname works aswell. It can't be my ftp client, because all other ftps that require ident work just fine.

I use a rather lean setup, the only scripts i use are ioNinja (v0.8.9.6b Released 2009-06-29), nxTools (1.2.1) and esmNewdir (version 0.70 (2005.11.02); i used it years ago as releasedb. nowadays nxTools is the primary db but i still keep esmNewdir around because it contains all these 10yr old directories incase i ever need to look something up).

I hope that information helps a bit.
opcode is offline   Reply With Quote
Old 08-10-2010, 06:15 PM   #112
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

opcode: You can always use "site ioversion" to get the current FTP server version. Also, be careful with your "site who" alias. The command info displayed by site who is designed to be viewed by normal users and thus doesn't display any command arguments so they'll just see things like "PORT", "site stats", etc. This prevents normal users from seeing the entire "site adduser" line which could be a bad thing... "Site swho" (and "site swho <cid>") are SiteOp functions and don't scrub anything but the PASS command so best keep that for your usage...

It sounds like you might have a couple of configuration problems though... I don't remember seeing the exact error you mentioned with failed transfers (could you double check the error message?) but one common problem is having a very small PASV port range and/or using a single fixed outgoing port. This is particularly apparent when 2 such servers try to FXP to each other.

This is from the .ini file:
Code:
# This option allows you to control which ports the server uses for outgoing
# connections.  If Out_Ports is undefined that means use the old default of
# Port-1 for the service (defined below) initiating the connection.  However
# to avoid "Connection closed: Only one usage of each socket address
# (protocol/network address/port) is normally permitted" errors caused by the
# receiving server or FTP client not having a large enough port range you can
# specify additional local ports to use.  An Out_Ports of 0 means use any
# port which for almost all cases eliminates the problem and is the new
# preferred setting unless you have a router/gateway that needs you to limit
# the outgoing ports.
# NOTE: Only the first single or range of ports is used.
Out_Ports = 0
Using any local port prevents PORT commands from having a destIP/srcIP/DestPort/SrcPort tuple collision too often which is not allowed by TCP. To solve the reverse problem where the other server is configured to use only 1 outgoing port you'll need a PASV port range that is large enough that you can allocate 1 port per file/listing used in a 30 second window. 30 seconds is the default/min TCP TIME_WAIT setting for windows. Try increasing the "Ports" range setting to see if that makes a difference. Remember to forward/allow the extra ports as well!

Assuming I understand the problem you were describing with quickly failing transfers after a while that should solve it provided it wasn't just an example of the server going having issues...

Regarding the ident lookup issue. Try adjusting the 'Ident_Timeout' setting in the .ini file. If you have a slow/lagged line the round trip query might be taking longer than you think and because the failure is cached for a while it will fail if you retry immediately. You can also try setting the cache time lower. Also, try enabling the 'Show_HostMask_Error' feature in the .ini file as well. As of v7.0 it will display 'Your user ident response did not match' instead of 'Your IP/hostname is not authorized' if just the ident is incorrect. That change resulted in a lot fewer hostmask update requests as people could see what was going on...


NOTE: There are a number of 3rd party DLL's which are poorly written, or have issues with the way ioFTPD uses winsock. The most annoying one is nVidia's Firewall that was distributed with lots of BIOS/install disks by OEMs. Older versions of NetLimiter also had issues. Comodo's defense+ component breaks some debugging stuff and possibly could be responsible for runtime errors as well. NVidia and NetLimiter exhibit behavior similar to what you reported: reasonably light load but would die every few hours... You can see what dll's are in use by downloading the free Microsoft Process Explorer which is much nicer than Task Manager. Check to see what 3rd party modules are in use by ioFTPD as this sounds like a really likely source of your problem.
Yil is offline   Reply With Quote
Old 08-11-2010, 08:57 AM   #113
opcode
Junior Member
 
Join Date: Aug 2009
Posts: 21
Default

Quote:
opcode: You can always use "site ioversion" to get the current FTP server version. Also, be careful with your "site who" alias. The command info displayed by site who is designed to be viewed by normal users and thus doesn't display any command arguments so they'll just see things like "PORT", "site stats", etc. This prevents normal users from seeing the entire "site adduser" line which could be a bad thing... "Site swho" (and "site swho <cid>") are SiteOp functions and don't scrub anything but the PASS command so best keep that for your usage...
Yeah i have "swho" set to 1A and "who" actually commented out in the [FTP_SITE_Permissions] section, so i guess noone should be able to use them besides me.

Quote:
It sounds like you might have a couple of configuration problems though... I don't remember seeing the exact error you mentioned with failed transfers (could you double check the error message?) but one common problem is having a very small PASV port range and/or using a single fixed outgoing port. This is particularly apparent when 2 such servers try to FXP to each other.
This is actually true, i only have "Ports = 23,24,443,80" in the .ini This used to be the old BBB.se trick (e.g. BBB.se had a cap on all outbound connections besides those on these ports, so it was the only way to dl something from these boxes with more then 5k/s. Not sure if it's still true these days though). But still, if i only have two transfers running, shouldn't it be able to use one of the other two ports if one of the transfers fails and needs to reset (i guess it can't immediately use the same port due to some technical circumstances)? Is the port for an upcomming connection selected randomly or does the ftpd know which ports are currently in use/free ?

Quote:
Using any local port prevents PORT commands from having a destIP/srcIP/DestPort/SrcPort tuple collision too often which is not allowed by TCP. To solve the reverse problem where the other server is configured to use only 1 outgoing port you'll need a PASV port range that is large enough that you can allocate 1 port per file/listing used in a 30 second window. 30 seconds is the default/min TCP TIME_WAIT setting for windows. Try increasing the "Ports" range setting to see if that makes a difference. Remember to forward/allow the extra ports as well!
I have "Out_Ports = 0", i don't think it's the other servers, because i assume these are rather default glftpd installations which usually don't severely limit the number of ports being used.

Quote:
Regarding the ident lookup issue. Try adjusting the 'Ident_Timeout' setting in the .ini file. If you have a slow/lagged line the round trip query might be taking longer than you think and because the failure is cached for a while it will fail if you retry immediately. You can also try setting the cache time lower. Also, try enabling the 'Show_HostMask_Error' feature in the .ini file as well. As of v7.0 it will display 'Your user ident response did not match' instead of 'Your IP/hostname is not authorized' if just the ident is incorrect. That change resulted in a lot fewer hostmask update requests as people could see what was going on...
I will try that. Weird that it only fails right after i start ioFTPD and never again once it works for the first time. I know that it's indeed just the ident failing because i see *@added.ip in the logfiles instead of ident@added.ip

Quote:
NOTE: There are a number of 3rd party DLL's which are poorly written, or have issues with the way ioFTPD uses winsock. The most annoying one is nVidia's Firewall that was distributed with lots of BIOS/install disks by OEMs. Older versions of NetLimiter also had issues. Comodo's defense+ component breaks some debugging stuff and possibly could be responsible for runtime errors as well. NVidia and NetLimiter exhibit behavior similar to what you reported: reasonably light load but would die every few hours... You can see what dll's are in use by downloading the free Microsoft Process Explorer which is much nicer than Task Manager. Check to see what 3rd party modules are in use by ioFTPD as this sounds like a really likely source of your problem.
Will do that aswell :-)
opcode is offline   Reply With Quote
Old 08-11-2010, 10:23 AM   #114
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

opcode: Interesting PASV port range issue there... First off, I think you should be able to use the same port multiple times BUT you can only be listening (waiting for the other server to connect) once per port. Thus you could have 20 active PASV transfers without issue, but only 4 unconnected at a time as of v7.5. Previous to the v7.5 release there was a rather serious problem with re-using the ports so that limit didn't exist, BUT it also meant that it was possible to transfer the wrong file because it could give out the same port to 2 different connections. Usually this didn't make a difference since things connected pretty fast and usually connected in the order given out but it was a rather large bug I stumbled across when testing mass transfers with just 2 PASV ports...

If the remote server uses the same port to initiate all outbound traffic (like ioFTPD did before v6.4) with just 4 ports you really couldn't transfer more than 4 files in under 30 seconds because of the timeout. In that case switch the PORT/PASV roles for FXP in your client since yours is configured to use random outgoing ports. TcpView is another free Microsoft tool (or from command line: netstat) which should show you the port numbers on both sides if you're interested.

The other issue would be port numbers themselves. Obviously you had a really good reason for choosing them, but they are among the worst possible ones for the FTP. Proxy scans, worms, etc often try to connect to those ports and the FTP server would think they were for the transfer it just gave that port out to. That may mean things would fail from time to time. Most likely this would be seen as very small files failing crc checks for larger ones.
Yil is offline   Reply With Quote
Old 08-11-2010, 09:43 PM   #115
o_dog
Senior Member
 
Join Date: May 2007
Posts: 692
Default

[resolve list "$pwd"]
there is something i would like you todo in this command, ioFTPD only write fileinfo to that list after the scripts has released the file. So uptime is not written before it's released,
Could you write the info to .ioftpd file before the scripts are called?
I know this will involve wiping the info from the file if script returns an error, but in order for that command to be of full use the script needs all info.

UPDATE: Might not be needed since i worked around it, still would be nice though =)

working on ioNiNJA v1.0, slowly and surely moving forward.
__________________
ioNiNJA

Last edited by o_dog; 08-11-2010 at 11:22 PM.
o_dog is offline   Reply With Quote
Old 08-12-2010, 08:21 AM   #116
ArtCore7
Junior Member
 
Join Date: Aug 2010
Posts: 14
Default

ioNiNJA 1.0 .... nice
ArtCore7 is offline   Reply With Quote
Old 08-12-2010, 11:32 PM   #117
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

o_dog: The fileinfo is usually updated twice per upload. An Upload or Overwrite will set the uid/gid to the user before starting the transfer, but a Resume will just update the uid. After the transfer is completed it will update the upload time by adding how long the transfer took to the current value after the script finishes. This allows the script the option of zeroing out a previous value or leaving it.

With the addition of the [ioTransferStats] command it probably makes more sense to adjust the value before the script runs much like you want since that's more flexible, but if you're just trying to get the upload time with higher resolution you can use that command...
Yil is offline   Reply With Quote
Old 08-13-2010, 01:55 PM   #118
o_dog
Senior Member
 
Join Date: May 2007
Posts: 692
Default

it passes the speed variable so no real need for either, but would be nice not to need have all the damn safty checks =)
__________________
ioNiNJA
o_dog is offline   Reply With Quote
Old 08-13-2010, 02:44 PM   #119
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Actually the speed variable and the upload time/filesize aren't exactly the same thing. The TCL speed variable is the speed for the last transfer which in the case of a resumed upload only covers the new part of the file. Hence the new TCL command to get the actual size of the transfer. The time it returns could be computed knowing the actual transfer size, but figured it was easy enough to return as well.
Yil is offline   Reply With Quote
Old 08-13-2010, 02:51 PM   #120
o_dog
Senior Member
 
Join Date: May 2007
Posts: 692
Default

ioninja does not support resumed uploads, atleast i don't think it does =)
I never saw any reason to include that in a zipscript, especially not today witth the bandwdth being what it is. So for ioninja it should be the same or close enough.
__________________
ioNiNJA
o_dog is offline   Reply With Quote
Reply

Tags
command, fixed, link, openssl, server

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 08:22 AM.

Parts of this site powered by vBulletin Mods & Addons from DragonByte Technologies Ltd. (Details)