[Investigated] ioB2 errors on rescan

[ko0ky]

Over the past few hours my site keeps crashing although ioftpd.exe is still running as a process the sitebot croaks and drops offline and I am unable to connect to the site. Getting the following message in my systemerror.log. Also as reported before, on some COMPLETES i will get duplicate stats or duplicate completes for the same release. I dont know if this is related but on this particular crash (posted below) I did notice the duplication of the COMPLETE and STATS.
Thanks in advance for any help, or let me know how I can futher assist the stomping of this bug.

09-10-2005 15:48:44 "c:\Site\ioFTPD\scripts\ioBanana\ioRESCAN.itcl" terminated abnormally
--------------------------- ErrorInfo ----------------------------
divide by zero
while executing
"expr double($valid) / double($files)"
(procedure "race" line 46)
invoked from within
"race "$parent" "$release" "$realdir" "$files" "1""
(procedure "rescan" line 86)
invoked from within
"rescan "[lindex $args 1]""
("RESCAN" arm line 5)
invoked from within
"switch [string toupper [lindex $args 0]] {

[godlike]

I can cofirm that, same thing happening here too :confused:

ioFTPD running but not connectable and even bot dies. Restart of ioFTPD is needed.

[Harm]

ioRESCAN is only called on rescan. It looks like your racefile got corrupted somehow. You can use SITE RESCAN in these specific directories to overwrite the racefiles with clean ones. Do not use SITE RESCAN <file> in such case, which is what you've most probably tried here.
Also the fact that the error is logged means the script is no longer hanging.

If I were you, I would make sure my users aren't allowed to resume and/or overwrite files. I would also review my ioFTPD.ini to make sure the threads configuration is optimal (see http://www.inicom.net/forum/showthread.php?t=13970).
Check your xferlog, it's likely that one file in each of these directories has been uploaded twice or more times.

Note: Godlike, could you also give details. I'm afraid that "I can confirm that" doesn't add anything to the discussion. There's like 50% chances your problem is different than ko0ky's.

[godlike]

Well my confirmation was meant in a way, like ioFTPD hangs, and bot pings out from irc and cant come back. Nut i do not have no logs of any kind because they are empty. So nothing finished and made a log i think. But i have left ioFTPD running that way for 12h and it still has not "unlocked" itself. I just have to go to the box and manualy restart ioFTPD to get it to work again.

This never happend with old iobv20. Same config as before, same ioa also. And upgraded iobanana on 20+ ioftpd`s but this error happens randomly on like 5-6 ftp`s at least once per day, and on ones with lots traffic, every few hours. Mostly annoying.

I would give more details but kind off hard because i cant realy see nothing what is happening in the background. And error logs empty. But i suspect it has maybe something to do when a file is uploaded that then something goes wrong. Because the fact is that it happens on "rush" hours on ioFTPD.

Tell me what to do and i will debug, it`s in my interest to get it resolved as soon as possible. It`s getting me nuts.

cya.

[ko0ky]

Yah to be honest. I screwed up when I pasted that information from the log file. My crash was on 13th and that log event is from 10th. I looked back at my logs and noticed no logs for the time it crashed.

this has never happened to me in the past and im using nothing but ioA, iobanana and ioftpd.

It's really annoying as the site before the recently upgrade was VERY VERY stable and it's not doing so great now. :(

[Harm]

1. Check you xferlog for two or more uploads of the same file. If you find some, the date and the time of the transfers will probably be nearly identical.

2. It would also be interesting to know what you have in your scheduler, your events and your ftp pre/post command events.

[ko0ky]

ill check the xferlog, I did already remove overwrite and resume access from the users. here's my scheduler and pre/post cmds. Strange, this wasn't a problem with the last setup I had using iobananav20. Why would it now be a problem? But thanks for giving suggestions to help pin it down. I have to review my irc logs and figure out what an upload looks like in the xferlog. I'll post what I find soon.

Reset = 0 0 * * &Reset
Service_Update = 10,30,50 * * * &Service_Update
DayStats = 55 22 * * TCL C:\site\ioFTPD\scripts\ioBanana\ioSTATS.itcl DAYSTATS
Spider = 0 3 * * TCL C:\site\ioFTPD\scripts\ioBanana\ioSC.itcl SPIDER
#AutoNuke = 0,10,20,30,40,50 * * * TCL c:\Site\ioFTPD\scripts\ioBanana\ioAUTONUKE.itcl
Newday = 0 22 * * EXEC ..\scripts\ioA\ioA.exe NEWDATE
Weekly = 0 0 * 6 EXEC ..\scripts\ioA\ioA.exe WEEKLYSET
searchlog = 0 0 * * EXEC ..\scripts\ioA\ioA.exe SEARCHLOGCREATE
reqwipe = 0 0 * * EXEC ..\scripts\ioA\ioA.exe REQWIPE

[FTP_Pre-Command_Events]
;list =
;stor =
;mkd =
mkd = TCL C:\site\ioFTPD\scripts\ioBanana\ioDUPE.itcl PREMKD

[FTP_Post-Command_Events]
;mkd =
;rmd =
;stor =
;dele =
cwd = TCL C:\site\ioFTPD\scripts\ioBanana\ioRULES.itcl CWD
dele = TCL C:\site\ioFTPD\scripts\ioBanana\ioPOSTDEL.itcl
mkd = TCL C:\site\ioFTPD\scripts\ioBanana\ioDUPE.itcl POSTMKD
rmd = TCL C:\site\ioFTPD\scripts\ioBanana\ioDUPE.itcl

[godlike]

Here you go my ioFTPD.ini aswell:

Reset = 0 0 * * &Reset
Service_Update = 10,30,50 * * * &Service_Update
Rotate_Log = 0 0 * * EXEC ..\scripts\ioBanana.exe rotatelog
Spider = 0 3 * * TCL c:\ioFTPD\scripts\ioBanana\ioSC.itcl SPIDER
searchlog = 30 10,20 * * EXEC ..\scripts\ioA\ioA.exe SEARCHLOGCREATE

[Events]
OnUploadComplete = TCL c:\ioFTPD\scripts\ioBanana\ioBANANA.itcl
OnUploadError = TCL c:\ioFTPD\scripts\ioBanana\ioERROR.itcl
OnLogIn = EXEC ..\scripts\ioA\ioA.exe logon

[FTP_Pre-Command_Events]
mkd = TCL c:\ioFTPD\scripts\ioBanana\ioDUPE.itcl PREMKD

[FTP_Post-Command_Events]
cwd = TCL c:\ioFTPD\scripts\ioBanana\ioRULES.itcl CWD
dele = TCL c:\ioFTPD\scripts\ioBanana\ioPOSTDEL.itcl
mkd = TCL c:\ioFTPD\scripts\ioBanana\ioDUPE.itcl POSTMKD
rmd = TCL c:\ioFTPD\scripts\ioBanana\ioDUPE.itcl

Thats all what it`s relevant i think. Now let me take a look at xferlog:

Last entries before the freeze, then i restarted ioFTPD in the morning, and log goes on ... pasted only lines around that time:

Wed Sep 14 01:26:12 2005 16 ip1 15000000 /TV-XVID/release1/CD2/release1.cd2.r34 b _ i r user1 ftp 1 * l
Wed Sep 14 01:26:29 2005 16 ip1 15000000 /TV-XVID/release1/CD2/release1.cd2.r35 b _ i r user1 ftp 1 * l
Wed Sep 14 01:26:53 2005 47 ip2 15000000 /TV-XVID/MIXED/release2/release2.r02 b _ o r user2 ftp 1 rave l
Wed Sep 14 08:39:56 2005 0 ip1 782 /TV-XVID/release3/release3.sfv b _ i r user1 ftp 1 * l

The thing is ioFTPD runs ok but nobody cant go on it. And logs for errors from ioFTPD are still empty after this...

I hope it helps you to find whats wrong, preferably quick, or else i will pull out all my hair soon :p

[godlike]

Some help about this issue.

Iobanana is totaly screwing with my ioftpd :(

I cant give more info than that i gave in previous post because there are no logs of any kind, when this happens. But fact is that it started happening after upgrade from iob20 to iobv2.

This is real pain in the ass.

Harm. Tell me what to do to debug this and i will do it.

cya.

[godlike]

One more thing i can give you as info.

I am 99,9% certain that this happend only when a file is uploading. Checked all the xferlogs on ioftpds, where this happens. And always there was something transfering, or was transfered in the moment of this happening. As visible also 2 posts back.

cya.

[Harm]

In fact, unless you're running other scripts that you found irrelevant (very unlikely, isn't it ? ;)), it could be a threading problem. ioFTPD kills the tcl interpreter if it for some reason it stops responding after a few mins (120 iirc), but I wonder what happens if it runs out of available threads for the interpreters.
Could you paste the [Thread] section of your ioFTPD.ini ?

You might also want to run Dr Watson to see when and how it crashes. It may not be usefull if it's only hanging like you said.

[godlike]

Here you go:

[Threads]
Process_Priority = NORMAL # Suggested process base priority (Idle/Normal/High/Realtime)
Io_Threads = 2 # Number of io threads
Worker_Threads = 1 # Number of worker threads
Worker_Fibers = 50 # Initial number of fibers
Encryption_Threads = 2 # Number of encryption threads

Tell me what can i change here and what would be optimal settings and i will see what happens then.

cya.

[Harm]

If I were you I would slightly increase the number of worker threads. The working fibers have disappeared a few betas ago btw.

Here is the configuration of my test server. There are rarely more than two concurrent users on this one so you might want to use more worker threads.
[Threads]
Process_Priority = NORMAL # Suggested process base priority (Idle/Normal/High/Realtime)
Io_Threads = 2 # Number of io threads
Worker_Threads = 5 # Number of worker threads
Encryption_Threads = 1 # Number of encryption threads
WindowName = ioFTPD::MessageWindow

darkone gave an explanation of all the types of threads and how you should configure them for optimal performance: http://www.inicom.net/forum/showthread.php?t=13970

[godlike]

Used your suggestion, and tried some more options. Result is same ...

What next ? :confused:

[Harm]

You have of course restarted ioFTPD after making the changes, haven't you ? What settings are you using now ? What cpu (and how many of them) are you using ? What's the average number of users connected to this server ?

[richto]

i had the same issue - just hangs or stops and says error when you try and rescan. When it hangs it tries to grab all the free memory on the box, which slows down everything.

I have to move the ioA rescan command so it was first to fix the issue.