ioFTPD v6.3.5 Beta Released

[PopWeasel]

Yil, thanks for your well thought-out reply. You've brought up some interesting points. Would it simplify things for now if we're able to assume all file system changes (within site dirs) are made via the FTP itself? Perhaps that's the direction to go in to help simplify this in being implemented more easily/quickly. Later, a crontab-like script could be created, which when run nightly would act not unlike the current nightly spider scripts which thumb through the file system in search of obsolete symlinks, except in this case it would be checking existing files & directories against entries in the log file you mentioned above. Of course, these files and directories in the log file would need to be associated with their respective diskspace usage for it to give us what we're striving towards..

I also envision that (via a script?) this could be tied in to FTP_Pre-Command_Events and FTP_Post-Command_Events, at least for STOR and DELE. On STOR it would need to check if the user has available quota for storing, it would be helpful if ioFTPD has knowledge of the size of the file attempting to be stored at this point so it could more intelligently allow/deny the STOR attempt. On DELE it would add the size of the file(s) deleted back into the user's available disk quota.

Lastly, implemented as a site command to facilitate easier management, it might look something like 'site change billgates diskquota 30g', or 'site change team diskquota 100g'.

At the end of the day, whether this is all handled by scripts or internally, I think some foundation must be laid on the core ftpd side to support it. Thoughts?

[FTPServerTools]

My thought about this:
If io gets on a dir it hasnt read before it is reading the dir anyway. hence the dir is in the system cache. So running a script that adds the filesize together is very doable and fast. Then for a prent dir it needs to be recounted. Thus I conclude that a dir holds its own sum. For speed it may hold the sum of its children dirs.
Say X does a cwd ABC then a post cwd reads the dir, counts the files and stores it in .size (or better yet in a SQLite database). Any outside dir change can be caught by monitoring dir changes (yes there are api functions for it). But this doesnt work for network drives or ext2/3 drives and such.
So my quicky thought:
(OnPostCWD:count on current dir (actually OnPostLIST is more exact))
OnPostStor:add newfile size to dir of Stor
OnPreDel:mark file size for removal
OnPostDel:remove file from dir size
OnPostAppe:count on current Dir
OnPostList (and OnPostNlist):count files on current dir, grab all subdir sizes and add together
ReadDirectoryChangesW (see also http://www.codeproject.com/KB/files/DirCheck.aspx and other sites) do a recount when changed file is no longer locked, alternatively you can use the change journal of ntfs volumes.
The FILE_NOTIFY_CHANGE_SIZE and the FILE_NOTIFY_CHANGE_LAST_WRITE should give some clues... The ReadDirectoryChangesW tells me in the FILE_NOTIFY_INFORMATION structure if a file was deleted, modified, renamed or added. So that could be used by any external program, connected with for example a simple sqlite database.
I think I'd personally bet on the ReadDirectoryChangesW thingy then the journal system. That would at least work generically (sort of).
Then another alternative is to use the microsoft indexing service. I havent delpved in to it tho and I dont know if it holds all the file changes.
The third alternative which may be not so hard is to use detours or some other hooking library to hook any file create, file close, and file writes in certain preset dirs. Any change in those dirs is very likely to be passed through the main file i/o functions and can thus be hooked. A fileclose can therefore be made into an update, a fileopen can be made into a add new filename or do nothing, a file write can be a trigger to do a recount after the close..
Anyway I see some techy solutions.... Indexing, Journaling and Hooking can all work quite nicely outside of the ftpd. But all three suffer from no reactiveness on networked drives...

[PopWeasel]

Thanks for your input on this FTPServerTools, and it's good to see you again old friend..! I like what you had to say about caching and the idea of writing a hidden file in the directory which would store the sizes of files & child directories to help speed up checking for available disk quota when users go to upload. I think this approach is more directory-centric; instead of attaching disk quotas to users it attaches them to dirs, am I right? If so, I think this would work equally as well as attaching disk quota to users/groups, possibly even better so, because it would allow more flexibility (user would have a disk quota under his group dir but not necessarily when uploading into section dirs).

An idea for settings could be:
#write which dirs diskquota should apply to
diskquota_dirs: /DIR1 /DIR2...etc.

The site commands would look something like:
site diskquota /DIR1 200g
site diskquota /DIR1/group1 20g
site diskquota /DIR1/group2 50g

So, let's say you have this directory structure-
/DIR1
/DIR1/group1/
/DIR1/group2/
/DIR1/group3/
/DIR1/group4/
/DIR1/group5/
/DIR1/group6/
/DIR1/group7/

...you have some nice flexibility here because you've specified that your main groups section dir will have a limit imposed of 200g (GB). The Group1 dir has been alotted 20GB, the Group2 dir has been alotted 50GB. Group3 through Group7 will share the remainder of the 130GB of diskquota space that's still available underneath /DIR1.

Also, if the site commands I mentioned above wouldn't work for some reason (maybe the script doesn't like slashes ), then it could be written in a configuration file:

#write your diskquota alottments here
diskquota_dir_size /DIR1 200g
diskquota_dir_size /DIR1/group1 20g
diskquota_dir_size /DIR1/group2 50g
etc...

My thought on Microsoft Indexing Service is that I wonder if it will pose file locking problems. For instance, Google Desktop and MS Indexing Service both used to get in the way when I used to be moving files around in Windows Explorer. I don't know if they've improved to not do that anymore, I stopped using them because it was too annoying..

As for ext2/3, reiserfs, etc., I think we can safely eliminate those from the equation since Windows doesn't read them..that I'm aware of anyways. And I think we can safely assume that anyone wanting these functions will need to be using local disks and that they'll need to accept the network drives limitation. Personally I'm perfectly fine with that.

You guys both know more about the nuts-and-bolts end of it- what's needed from the core ftpd in order to support this?

Anyone else have any thoughts?

Someone may want to split off these 4 posts or so into a new topic for this
Thanks.


Edit: I count 8 posts including this one that could be moved to their own new topic, starting with my post of 02-11-2008, 09:52 AM if anyone is up for it

[o_dog]

just a small request, i would like it to resolve symlinks in newdir log.
/today/ -> /section/today/

if someone creates /today/new it should log /section/today/new
now it just logs /today/new as pwd.

[hukker]

yil hey, just wondering how its going with the new release, any progress with the psycho bug which locks ioftpd...


anyways have a nice day

[Flow]

hey Yil, whare you at now? whats status on the 6.4 -stable release?

[Yil]

No luck on lockup issue as it's nearly impossible to trigger for me and it increasingly look like nothing is being done wrong in the server code itself. Still holding out hope to solve that...

The only other 2 known "bugs" are site adduser and site addip not accepting hostmasks from non-M accounts when in backward compatibility mode (i.e. no Secure_Ip rules are defined). Trivial fix. And the site crashes when dealing with pathnames longer than 260 characters in the recursive routines which I've sorta fixed by ignoring those files, but should fix correctly...

If you know of a "bug" and not a new feature request speak up

[thomas74]

Ah, so I see more users are having the same problem with lockup.

I had this lockup happen twice during 14 days. I can't see no pattern for it. I really hope you're able to locate the bug.

edit: System is Vista.

[Yil]

Lockup issue survey:

1) Has anyone running ioFTPD without any addons had the server lockup?

2) If addons are involved, are they tcl scripts or external apps?

3) If tcl is involved, does the script make use of sockets? I.e. an imdb script that talks to imdb directly and not through curl.

4) What about 3rd party libraries such as neoxed's share DB or mySQL libs?

When replying include ioFTPD version, addon scripts you are using, and what operating system and how many processors/cores you have.

It's looking increasingly likely that this is some bizarre issue with MS libraries. In particular tcl's use of winsock and/or 3rd party libraries. Since I can confirm the lockups I have information on involve the loader lock it's possible various static runtimes are conflicting or the static runtime is having issues with the dynamic runtime on some operating systems...

I'm looking into recompiling everything against dynamic runtimes and distributing the MS redistributable versions for people to install to see if that fixes things...

[Flow]

Hmm.... what happens when these lockup occurs?.. im curiuos ... cant connect? cant delete files?

I get WarChive error now and then but that i guess is not ioFTPD fault.

[Yil]

It depends since you can still do things that don't require you to acquire the loader lock.

Thus you can login (provided you don't have an ident in your hostmask since ident lookup will fail) since the sockets for this were pre-allocated. You can't get directory listings (except via stat -l perhaps) or transfer files though because those require new sockets.

You can't issue any command that ends up creating a process so simple tcl scripts might work, but external processes will just hang.

Site shutdown doesn't work.

Site crashnow amazingly doesn't work either and that's how you can tell you have this problem. It appears that exception handling or simply opening a text file acquires the loader lock ! I specifically wrote the exception handler to not attempt any function in the early part of the routine that is documented to acquire the loader lock. Thus I do no process address lookups, dll loading, etc... All of that was done ahead of time for just this reason and yet I still can't get around this problem.

Basically after a while the process totally locks up, and only killing it from the task manager will cause it to exit.

[thomas74]

I run the latest build of ioFTPD, latest nxTools and latest ioNinja.

[FTPServerTools]

Thanks for your view on this Popweasel... And good to see you are still going strong..

[costanza]

Great script , everything run with no problem.

Great job


ioFTPD-v6.3.5+ioninja v6.3.5.zip+esmnewdir+windows 2003 sp2

[thomas74]

Any progress on that lockup bug ?