ioNinja Issue

[HD1923]

Hey everyone,

I'm using the ioFTPD 7.0.3 ver and I like to use the ioNinja script, however, when installed unwanted behavior occurs eventually while uploading to the site. I'm not sure when it happens or the pattern of occurrence but uploading while the script is installed will crash the site and by crash I mean transfers to everyone stops and the site hangs. Looking at the logs the only errors I see related to the script are string conversion errors. Without the script, the site runs fine, but the script is mighty useful and I would like to keep using it.

can anyone provide any assistance?

Appreciate it.

sorry if I posted in the wrong place

[Yil]

I'm guessing that it's ioFTPD's fault and not the scripts. Check out the main forum postings for how to determine if you have the lockup bug. I'm working on the v7.1 release due in a little over a week or so and it will have a new attempt at fixing the problem. The newer version of TCL with ioFTPD v7+ and externally run scripts either via EXEC or through TCL (like unzip, etc) and TCL socket connections all seem to make things worse. If you can't wait a week you can try using 6.9.3 and ioNinja v7 something as I think that will show the problem less but that's a lot of changes. I'd probably give me a week or so and see if I get lucky...

The conversion error message can be safely ignored, it's fixed in the upcoming v7.1.

[Yil]

I just started some testing with the latest Ninja to see how it likes the new directory caching logic. Seems OK so far, but I started playing with some zip releases and noticed I wasn't getting any incomplete dirs being created. Am I missing something? Works fine for sfv/rar stuff though so switch to that.

I also didn't see a way to switch between ioFTPD style and windows style symlinks for the incomplete dirs. It looks like you can configure how you wanted the sorted stuff to show up, but not the incomplete tags. No biggie but I'm pretty sure I set that up with a v7 release. I manually created some junctions to test with though. Broken links show up now and are pretty easy to spot.

[o_dog]

junctions are very unsuitable for incomplete links, not really any upside to it, alot of downsides though. The logging would have to be rewritten and it has to check the link everytime it wites to the logs. The links are more to show that something is incomplete than for actually use. Also using them for uploading purpose would mean they get deleted on complete and then they would be created by user as regular dirs instead, and you would end up with alot of other problems. The only upside of junctions is that they work as regular dirs in windows and can be shared as such over smb. Using them for incomplete dirs is just a very very bad idea.

[HD1923]

Hey thanks for the reply,

Just to clear up, is the lockup bug associated with that race condition? Do I test it by crashing it? I can see how this would stop users from connecting, but I have no problem getting more people on there, they just cant get any data from their transfers, though I will try it next time I get the chance to redo the server and let you know of the results, also, where can I get ioNinja v7?

[Dahlia]

Hey thanks for the reply,

Just to clear up, is the lockup bug associated with that race condition? Do I test it by crashing it? I can see how this would stop users from connecting, but I have no problem getting more people on there, they just cant get any data from their transfers, though I will try it next time I get the chance to redo the server and let you know of the results, also, where can I get ioNinja v7?

ioninja.beta.v0.7.1-2008-09-21 (http://musicbyds.host.sk/utilz/ioninja.beta.v0.7.1-2008-09-21.rar)
io v7 and ioninja v8 works great. i never had such issue with 'site hangs' and i have like 2000 files and 50gb/per day up/down traffic. must be something with hardware or system issue.

[HD1923]

ioninja.beta.v0.7.1-2008-09-21 (http://musicbyds.host.sk/utilz/ioninja.beta.v0.7.1-2008-09-21.rar)
io v7 and ioninja v8 works great. i never had such issue with 'site hangs' and i have like 2000 files and 50gb/per day up/down traffic. must be something with hardware or system issue.

Cool, that's totally helpful. I don't blame ioNinja and its not a system issue, it's worked before just fine and the whole site works without the script enabled.

Sorry, didn't see that you were giving me the link

But the link is dead, thanks though.

[Silly]

i have the same issue here....

while i am using io 7.1 with the latest build of ioninjia....

i don't know if there is a way to fix it...or i have to change into some other zipscript...

[o_dog]

talk to YiL, since the script doesn't crash it's not a ioninja issue but an ioftpd one. IF you get some kind of crash message then i have somethign to work with.

[Silly]

have 0 error messege..the entire site just hang there....and shows time out...

[o_dog]

check if zip.exe is running or unzip.exe, if so kill them and see if it works again

[Silly]

check if zip.exe is running or unzip.exe, if so kill them and see if it works again

just checked...neither zip.exe nor unzip.exe runinng at background......

when crash happens list just can't show any files...

when i am trying to stat -l, everything seems fine....


can't really tell it's the bug of ioftpd itself or ioninja...

:confused:

[Silly]

……ok...stat -l could list now...but can't download...

following is my error log..

[1] CWD /TV/The.Pacific.Pt.II.720p.HDTV.x264-IMMERSE
[1] 250 CWD command successful.
[1] PWD
[1] 257 "/TV/The.Pacific.Pt.II.720p.HDTV.x264-IMMERSE" is current directory.
[1] STAT -l
[1] List Complete: 251 bytes in 0.02 seconds (15.69KB/s)
[1] TYPE I
[1] 200 Type set to I.
[1] MDTM the.pacific.pt.ii.720p.hdtv.x264-immerse.mkv
[1] 213 20100322030516
[1] PASV
[1] 227 Entering Passive Mode ()
[1] Opening data connection IP: PORT: 4937
[1] RETR the.pacific.pt.ii.720p.hdtv.x264-immerse.mkv
[1] 150 Opening BINARY mode data connection for the.pacific.pt.ii.720p.hdtv.x264-immerse.mkv (1565523860 bytes).
[1] 426 Connection closed: Connection timed out.
[1] PASV
[1] 227 Entering Passive Mode ()
[1] Opening data connection IP: PORT: 5445
[1] REST 0
[1] 350 Restarting at 0. Send STORE or RETRIEVE to initiate transfer.
[1] RETR the.pacific.pt.ii.720p.hdtv.x264-immerse.mkv
[1] 150 Opening BINARY mode data connection for the.pacific.pt.ii.720p.hdtv.x264-immerse.mkv (1565523860 bytes).

[Silly]

and extra information...when ioftpd crash system physical memory usese up to 99%. when i kill ioftpd.exe...the total memory usage drop back to 70%....is this kinda memory leak or something? i don't quite understand it..

[Yil]

That looks like the annoying lockup bug... You running win2k3? It seems to happen far more often on that OS especially with multi-core/processor setups. Win2k8/XP/win7 would be better choices if possible. The memory usage number you quoted doesn't really mean anything. You'd have to look at VM/Commit/Working Set details for the process to get a better idea of memory usage but I doubt that's the problem given how it's acting.

You've mentioned ioNinja, are you also using nxTools or other scripts? Do you do imdb, etc lookups? Have lots of mp3/movie uploads? Are most transfers using SSL? How many users online/transferring at a time? How much uptime do you get before you see the problem and anything in particular that seems likely to trigger it? Just trying to get more info to compare with others who have the problem.

If this problem is happening every day or so for you there are a few possible things to look forward to. The next feature release of ioFTPD will be trying openSSL to see if ditching the MS crypto libraries makes a difference since it seems to use the lock that gets stuck. It's an internal MS lock I can't touch which makes it annoying to debug.

There are 2 other things I just thought up that we could try though. I could perhaps make the server connect to itself every minute or so as part of the service loop or something. Thus when the server gets stuck it would figure it out quickly and either try and commit suicide (remember, it CAN'T exit gracefully, spawn processes, etc since it's an internal windows lock that is stuck and it's screws everything up), or report a problem/fail to report and see if we can't get the service manager to try and kill the process if you are running it as a service. I don't remember off the top of my head if it will do that, or just try to start a new copy which won't solve anything since the socket to listen on will already be held. But if it does the service manager could be used to restart it automatically if I can make it terminate which while not a very graceful solution would at least keep downtime to a minimum.

The other option to try is to use processor affinity to make ioFTPD only ever use one processor as this should reduce the chance of race conditions on the loader lock at the cost of lost performance. Since the server is so lightweight this might be worth the hit since it's usually I/O bound anyway and I'd trade stability for performance if it's a small hit.

[Silly]

That looks like the annoying lockup bug... You running win2k3? It seems to happen far more often on that OS especially with multi-core/processor setups. Win2k8/XP/win7 would be better choices if possible. The memory usage number you quoted doesn't really mean anything. You'd have to look at VM/Commit/Working Set details for the process to get a better idea of memory usage but I doubt that's the problem given how it's acting.

You've mentioned ioNinja, are you also using nxTools or other scripts? Do you do imdb, etc lookups? Have lots of mp3/movie uploads? Are most transfers using SSL? How many users online/transferring at a time? How much uptime do you get before you see the problem and anything in particular that seems likely to trigger it? Just trying to get more info to compare with others who have the problem.

If this problem is happening every day or so for you there are a few possible things to look forward to. The next feature release of ioFTPD will be trying openSSL to see if ditching the MS crypto libraries makes a difference since it seems to use the lock that gets stuck. It's an internal MS lock I can't touch which makes it annoying to debug.

There are 2 other things I just thought up that we could try though. I could perhaps make the server connect to itself every minute or so as part of the service loop or something. Thus when the server gets stuck it would figure it out quickly and either try and commit suicide (remember, it CAN'T exit gracefully, spawn processes, etc since it's an internal windows lock that is stuck and it's screws everything up), or report a problem/fail to report and see if we can't get the service manager to try and kill the process if you are running it as a service. I don't remember off the top of my head if it will do that, or just try to start a new copy which won't solve anything since the socket to listen on will already be held. But if it does the service manager could be used to restart it automatically if I can make it terminate which while not a very graceful solution would at least keep downtime to a minimum.

The other option to try is to use processor affinity to make ioFTPD only ever use one processor as this should reduce the chance of race conditions on the loader lock at the cost of lost performance. Since the server is so lightweight this might be worth the hit since it's usually I/O bound anyway and I'd trade stability for performance if it's a small hit.

thank you for your prompt relply yil

i am running io under win2k8 64bit..
For scripts, i was using ioA and ioNinja w/ imdb...not just happen when mp3/movie were uploading, but also some rar files. None transfers uses SSL. avg online user is about 10-15. But just only 2 or 3 uploads. The lockup bug randomly occurs. But one software would definitely trigger that bug is when I try to use uTorrent to fetch some files.
About the frequency, sometime were couple times a day, sometime were one day a freeze.

i will try to set affinity to one processor to see if that could solve that problem.

wish me luck..:)

[Silly]

alrite...set affinity doesn't work..

[Yil]

Hmm... I was hoping the idea of additional serialization imposed by processor affinity to a single core/processor would help but I guess not. If it locks up could you try to get a minidump via the task manager? I'd like to verify it's the loader lock getting stuck and what 3rd party modules might be loaded.

I do find it very interesting that uTorrent could make an impact. Any chance you are running something like netlimiter or a firewall with any sort of active filtering on connections? ioFTPD seems to highlight problems with 3rd party injected dlls.

[Silly]

Hmm... I was hoping the idea of additional serialization imposed by processor affinity to a single core/processor would help but I guess not. If it locks up could you try to get a minidump via the task manager? I'd like to verify it's the loader lock getting stuck and what 3rd party modules might be loaded.

I do find it very interesting that uTorrent could make an impact. Any chance you are running something like netlimiter or a firewall with any sort of active filtering on connections? ioFTPD seems to highlight problems with 3rd party injected dlls.

nah, i don't use any that kinda software...

for minidump...i had never tried. i will try to figure that out and then i will post them here

[Silly]

Hmm... I was hoping the idea of additional serialization imposed by processor affinity to a single core/processor would help but I guess not. If it locks up could you try to get a minidump via the task manager? I'd like to verify it's the loader lock getting stuck and what 3rd party modules might be loaded.

I do find it very interesting that uTorrent could make an impact. Any chance you are running something like netlimiter or a firewall with any sort of active filtering on connections? ioFTPD seems to highlight problems with 3rd party injected dlls.

dunno how to get mini dump, but i got full dump...

i will pm you where to download..

uploading..

about 3xmb

thx