nxTools Memory Leak?

[CoLt-[45]]

Not sure exactly what's happening here, but when i edit this in ioftpd.ini

[FTP_Pre-Command_Events]
stor = TCL ..\scripts\nxTools\nxDupe.itcl prestor
The memory has started to increased by approx 1mb or 2mb everytime a file is uploaded till ioftpd.exe has crashed. :(

When I disable it just like this

[FTP_Pre-Command_Events]
;stor = TCL ..\scripts\nxTools\nxDupe.itcl prestor
Works very stable, but I need this nice script for my site! :confused:

Any idea? Suggestion?

Cheers in advance

[neoxed]

nxTools doesn't "directly" allocate any memory, since it's a Tcl script. However, it may be caused an extension used by nxTools, such as SQLite. I've never encounted this problem nor heard of anything like it.

I'll look into it when I have time later this week. :)

[PaRaNoiD_JaCK]

I have this problem aswell. At the moment im just rebooting the ftpd every 24 hours to purge the memory allocated by ioftpd.

n00b question here:

By disabling

mkd = TCL ..\scripts\nxTools\nxDupe.tcl premkd
stor = TCL ..\scripts\nxTools\nxDupe.itcl prestor

what features are lost with the script?

[neoxed]

Which version of ioFTPD and Windows (service pack as well) are you two using?

The prestor and premkd events are used by the file and directory dupe checker, by disabling them dupe checking will not be enforced.

[PaRaNoiD_JaCK]

Running ioFTPD version 5-8-5r

Windows 2003 server standard SP1

[fobban]

Originally posted by NeoXed
nxTools doesn't "directly" allocate any memory, since it's a Tcl script. However, it may be caused an extension used by nxTools, such as SQLite. I've never encounted this problem nor heard of anything like it.

I'll look into it when I have time later this week. :)

Actually there is a similar problem with bounty's newdir script.
If you have a huuuuge dupedir file and make a site dupe you see that the size of dupedir.log is added ioFTPD's used memory.
I think one have to unset variables when you've finished using them.

[neoxed]

fobban: the file and directory names are stored in a SQLite database and accessed using the TclSQLite extension, not a read from text file.

On a side note: The variable doesn't have to be unset if it's a procedural/local variable, since Tcl's reference counting will destroy it once the function exits. If the variable is scoped in a namespace (i.e. the global namespace), it would have to be unset. This doesn't apply to nxTools, since databases are not accessed in the same manner that newdir accesses log files.

[Freestyler]

I see my memory usage grow slightly when i upload a release, since i started with a empty db.

I tested with the db of PaRaNoiD_JaCK and saw the memory usage grow from 11MB to 20MB after 15 uploads.

NXTools seems to be pretty tight, but the memory problem is a showstopper for me.

I also run on Windows 2003 SP1. Can we somehow help debugging?

[neoxed]

It seems this bug is specific to Windows 2003 SP1; I've only tested nxTools on Windows XP SP2 and Windows 2003 (no SP1).

Do any of you have any entries in logs\nxError.log or logs\SystemError.log?

[Freestyler]

Clearly im not ready with the setup. I get this line a lot in NxError.log:

04-17-2005 01:32:20 - LoadScript : unable to source "../scripts/nxAutoNuke.cfg": couldn't read file "../scripts/nxAutoNuke.cfg": no such file or directory

Two errors in systemlog:

04-16-2005 21:30:16 "..\scripts\nxTools\nxDupe.tcl" terminated abnormally
--------------------------- ErrorInfo ----------------------------
database is locked
while executing
"FileDb eval {INSERT INTO DupeFiles (TimeStamp,UserName,GroupName,FileName) VALUES($TimeStamp,$user,$group,$FileName)}"
(procedure "UpdateFiles" line 11)
invoked from within
"UpdateFiles $Action $VirtualPath"
(procedure "UpdateLog" line 9)
invoked from within
"UpdateLog "UPLD" $VirtualPath"
("upload" arm line 4)
in04-16-2005 21:39:16 "..\scripts\nxTools\nxDupe.tcl" terminated abnormally
--------------------------- ErrorInfo ----------------------------
database is locked
while executing
"FileDb eval {INSERT INTO DupeFiles (TimeStamp,UserName,GroupName,FileName) VALUES($TimeStamp,$user,$group,$FileName)}"
(procedure "UpdateFiles" line 11)
invoked from within
"UpdateFiles $Action $VirtualPath"
(procedure "UpdateLog" line 9)
invoked from within
"UpdateLog "UPLD" $VirtualPath"
("upload" arm line 4)
in

[CoLt-[45]]

Originally posted by NeoXed
It seems this bug is specific to Windows 2003 SP1; I've only tested nxTools on Windows XP SP2 and Windows 2003 (no SP1).

Do any of you have any entries in logs\nxError.log or logs\SystemError.log?

I get that problems even if i'm using Windows 2000 SP4 or Windows XP SP2

I've tried the new version, for some reason - i can't get it to work! :/ (of course i haven't finished my ioftpd setup :p)

There's one thing i'd like to know HOW do I add all the filenames into the database? - is there's a way round to it rather than having to add via ftp? - like a batch file or something?

Thanks

[PaRaNoiD_JaCK]

I get similar error as freestyler:

04-17-2005 07:21:44 "..\scripts\nxTools\nxDupe.tcl" terminated abnormally
--------------------------- ErrorInfo ----------------------------
database is locked
while executing
"FileDb eval {INSERT INTO DupeFiles (TimeStamp,UserName,GroupName,FileName) VALUES($TimeStamp,$user,$group,$FileName)}"
(procedure "UpdateFiles" line 11)
invoked from within
"UpdateFiles $Action $VirtualPath"
(procedure "UpdateLog" line 9)
invoked from within
"UpdateLog "UPLD" $VirtualPath"
("upload" arm line 4)
in

This error doesnt happen with every upload. The memory increase does tho.

:)

[neoxed]

Originally posted by CoLt-[45]
There's one thing i'd like to know HOW do I add all the filenames into the database? - is there's a way round to it rather than having to add via ftp? - like a batch file or something? Check the manual for the "SITE REBUILDLOGS" command.

Originally posted by CoLt-[45]
I get that problems even if i'm using Windows 2000 SP4 or Windows XP SP2ioFTPD recycles Tcl interpreters so you will notice an increase in memory each time a new interpreter is created. As far as I know, none of the interpreters are destroyed - so all of the Tcl code stays in memory. My guess is the number of Tcl interpreters is related to the number of worker threads.

Do any of you have Worker_Threads set to some extreme or unreasonable value?

http://www.ioftpd.com/board/showthread.php?s=&threadid=3970

Originally posted by darkone
Optimal amount for io threads, is 2x number of logical cpus.
Optimal amount for worker threads on site that runs lots of scripts, is ~half of max users online.
Optimal amount for encryption threads, is number of logical cpus.

[Freestyler]

This is what I use:

Io_Threads = 2 # Number of io threads
Worker_Threads = 5 # Number of worker threads
Encryption_Threads = 2 # Number of encryption threads

[neoxed]

Alright, turns out SQLite was infact the source of the memory leaks. More specifically, the Tcl extension to SQLite.

I built the latest CVS version of TclSQLite, which includes a fix for a memory leak discovered shortly after the release of SQLite v3.2.1. http://www.sqlite.org/cvstrac/chngview?cn=2435

You may download the updated libraries from here:
http://www.ioftpd.com/scripts/script.php?id=167

Edit: I posted the wrong ones, please redownload them.

[PaRaNoiD_JaCK]

The memory leak still happens here after updating the dll`s. Do I need to start with fresh db`s?

[neoxed]

There was another memory leak in the Tcl interface to SQLite found two days ago.

http://www.sqlite.org/cvstrac/chngview?cn=2464

I'm making a few changes to the dupe checker (hopefully to speed things up with _large_ databases). I'll post v0.9.10 with updated DLLs when I'm finished.

[PaRaNoiD_JaCK]

ok. I look forward to the update :D

Thanks

[neoxed]

nxTools v0.9.10 is out, be sure to download the new library package as well (v4).

http://www.ioftpd.com/scripts/script.php?id=167

[PaRaNoiD_JaCK]

All is working fine here except when i do SITE REQUESTS. I get the following:

[L] 200-.-[Request]--------------------------------------------------------------.
[L] 200-| Unable to open requests database. |
[L] 200-'------------------------------------------------------------------------'

Tried remaking the requests.db but still gives that error.

It`s no big problem. Just glad the memory leak is fixed.

Thanks , :D

[neoxed]

Damnit, I missed that - I had a space character out of place..grr. :)

Fixed in v0.9.10b, just replace nxRequest.tcl.

http://www.ioftpd.com/scripts/script.php?id=167

[PaRaNoiD_JaCK]

thanks :D