Go Back   FlashFXP Forums > > > >

ioFTPD General New releases, comments, questions regarding the latest version of ioFTPD.

Reply
 
Thread Tools Rate Thread Display Modes
Old 02-15-2008, 08:31 PM   #166
PopWeasel
Senior Member
ioFTPD Scripter
 
Join Date: Feb 2004
Posts: 181
Default

Yil, thanks for your well thought-out reply. You've brought up some interesting points. Would it simplify things for now if we're able to assume all file system changes (within site dirs) are made via the FTP itself? Perhaps that's the direction to go in to help simplify this in being implemented more easily/quickly. Later, a crontab-like script could be created, which when run nightly would act not unlike the current nightly spider scripts which thumb through the file system in search of obsolete symlinks, except in this case it would be checking existing files & directories against entries in the log file you mentioned above. Of course, these files and directories in the log file would need to be associated with their respective diskspace usage for it to give us what we're striving towards..

I also envision that (via a script?) this could be tied in to FTP_Pre-Command_Events and FTP_Post-Command_Events, at least for STOR and DELE. On STOR it would need to check if the user has available quota for storing, it would be helpful if ioFTPD has knowledge of the size of the file attempting to be stored at this point so it could more intelligently allow/deny the STOR attempt. On DELE it would add the size of the file(s) deleted back into the user's available disk quota.

Lastly, implemented as a site command to facilitate easier management, it might look something like 'site change billgates diskquota 30g', or 'site change team diskquota 100g'.

At the end of the day, whether this is all handled by scripts or internally, I think some foundation must be laid on the core ftpd side to support it. Thoughts?

Last edited by PopWeasel; 02-15-2008 at 08:44 PM.
PopWeasel is offline   Reply With Quote
Old 02-17-2008, 08:16 PM   #167
FTPServerTools
Senior Member
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: Sep 2002
Posts: 543
Default

My thought about this:
If io gets on a dir it hasnt read before it is reading the dir anyway. hence the dir is in the system cache. So running a script that adds the filesize together is very doable and fast. Then for a prent dir it needs to be recounted. Thus I conclude that a dir holds its own sum. For speed it may hold the sum of its children dirs.
Say X does a cwd ABC then a post cwd reads the dir, counts the files and stores it in .size (or better yet in a SQLite database). Any outside dir change can be caught by monitoring dir changes (yes there are api functions for it). But this doesnt work for network drives or ext2/3 drives and such.
So my quicky thought:
(OnPostCWD:count on current dir (actually OnPostLIST is more exact))
OnPostStor:add newfile size to dir of Stor
OnPreDel:mark file size for removal
OnPostDel:remove file from dir size
OnPostAppe:count on current Dir
OnPostList (and OnPostNlist):count files on current dir, grab all subdir sizes and add together
ReadDirectoryChangesW (see also http://www.codeproject.com/KB/files/DirCheck.aspx and other sites) do a recount when changed file is no longer locked, alternatively you can use the change journal of ntfs volumes.
The FILE_NOTIFY_CHANGE_SIZE and the FILE_NOTIFY_CHANGE_LAST_WRITE should give some clues... The ReadDirectoryChangesW tells me in the FILE_NOTIFY_INFORMATION structure if a file was deleted, modified, renamed or added. So that could be used by any external program, connected with for example a simple sqlite database.
I think I'd personally bet on the ReadDirectoryChangesW thingy then the journal system. That would at least work generically (sort of).
Then another alternative is to use the microsoft indexing service. I havent delpved in to it tho and I dont know if it holds all the file changes.
The third alternative which may be not so hard is to use detours or some other hooking library to hook any file create, file close, and file writes in certain preset dirs. Any change in those dirs is very likely to be passed through the main file i/o functions and can thus be hooked. A fileclose can therefore be made into an update, a fileopen can be made into a add new filename or do nothing, a file write can be a trigger to do a recount after the close..
Anyway I see some techy solutions.... Indexing, Journaling and Hooking can all work quite nicely outside of the ftpd. But all three suffer from no reactiveness on networked drives...
FTPServerTools is offline   Reply With Quote
Old 02-17-2008, 11:03 PM   #168
PopWeasel
Senior Member
ioFTPD Scripter
 
Join Date: Feb 2004
Posts: 181
Default

Thanks for your input on this FTPServerTools, and it's good to see you again old friend..! I like what you had to say about caching and the idea of writing a hidden file in the directory which would store the sizes of files & child directories to help speed up checking for available disk quota when users go to upload. I think this approach is more directory-centric; instead of attaching disk quotas to users it attaches them to dirs, am I right? If so, I think this would work equally as well as attaching disk quota to users/groups, possibly even better so, because it would allow more flexibility (user would have a disk quota under his group dir but not necessarily when uploading into section dirs).

An idea for settings could be:
#write which dirs diskquota should apply to
diskquota_dirs: /DIR1 /DIR2...etc.

The site commands would look something like:
site diskquota /DIR1 200g
site diskquota /DIR1/group1 20g
site diskquota /DIR1/group2 50g

So, let's say you have this directory structure-
/DIR1
/DIR1/group1/
/DIR1/group2/
/DIR1/group3/
/DIR1/group4/
/DIR1/group5/
/DIR1/group6/
/DIR1/group7/

...you have some nice flexibility here because you've specified that your main groups section dir will have a limit imposed of 200g (GB). The Group1 dir has been alotted 20GB, the Group2 dir has been alotted 50GB. Group3 through Group7 will share the remainder of the 130GB of diskquota space that's still available underneath /DIR1.

Also, if the site commands I mentioned above wouldn't work for some reason (maybe the script doesn't like slashes ), then it could be written in a configuration file:

#write your diskquota alottments here
diskquota_dir_size /DIR1 200g
diskquota_dir_size /DIR1/group1 20g
diskquota_dir_size /DIR1/group2 50g
etc...

My thought on Microsoft Indexing Service is that I wonder if it will pose file locking problems. For instance, Google Desktop and MS Indexing Service both used to get in the way when I used to be moving files around in Windows Explorer. I don't know if they've improved to not do that anymore, I stopped using them because it was too annoying..

As for ext2/3, reiserfs, etc., I think we can safely eliminate those from the equation since Windows doesn't read them..that I'm aware of anyways. And I think we can safely assume that anyone wanting these functions will need to be using local disks and that they'll need to accept the network drives limitation. Personally I'm perfectly fine with that.

You guys both know more about the nuts-and-bolts end of it- what's needed from the core ftpd in order to support this?

Anyone else have any thoughts?

Someone may want to split off these 4 posts or so into a new topic for this
Thanks.


Edit: I count 8 posts including this one that could be moved to their own new topic, starting with my post of 02-11-2008, 09:52 AM if anyone is up for it

Last edited by PopWeasel; 02-17-2008 at 11:10 PM.
PopWeasel is offline   Reply With Quote
Old 02-21-2008, 11:20 PM   #169
o_dog
Senior Member
 
Join Date: May 2007
Posts: 692
Default

just a small request, i would like it to resolve symlinks in newdir log.
/today/ -> /section/today/

if someone creates /today/new it should log /section/today/new
now it just logs /today/new as pwd.
__________________
ioNiNJA
o_dog is offline   Reply With Quote
Old 02-26-2008, 01:10 PM   #170
hukker
Senior Member
FlashFXP Registered User
ioFTPD Foundation User
 
Join Date: Jun 2004
Posts: 165
Default

yil hey, just wondering how its going with the new release, any progress with the psycho bug which locks ioftpd...


anyways have a nice day
hukker is offline   Reply With Quote
Old 02-29-2008, 04:06 PM   #171
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default

hey Yil, whare you at now? whats status on the 6.4 -stable release?
Flow is offline   Reply With Quote
Old 02-29-2008, 04:30 PM   #172
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

No luck on lockup issue as it's nearly impossible to trigger for me and it increasingly look like nothing is being done wrong in the server code itself. Still holding out hope to solve that...

The only other 2 known "bugs" are site adduser and site addip not accepting hostmasks from non-M accounts when in backward compatibility mode (i.e. no Secure_Ip rules are defined). Trivial fix. And the site crashes when dealing with pathnames longer than 260 characters in the recursive routines which I've sorta fixed by ignoring those files, but should fix correctly...

If you know of a "bug" and not a new feature request speak up
Yil is offline   Reply With Quote
Old 03-16-2008, 05:06 PM   #173
thomas74
Junior Member
 
Join Date: Dec 2007
Posts: 14
Default random lockup

Ah, so I see more users are having the same problem with lockup.

I had this lockup happen twice during 14 days. I can't see no pattern for it. I really hope you're able to locate the bug.

edit: System is Vista.
thomas74 is offline   Reply With Quote
Old 03-18-2008, 10:20 PM   #174
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Lockup issue survey:

1) Has anyone running ioFTPD without any addons had the server lockup?

2) If addons are involved, are they tcl scripts or external apps?

3) If tcl is involved, does the script make use of sockets? I.e. an imdb script that talks to imdb directly and not through curl.

4) What about 3rd party libraries such as neoxed's share DB or mySQL libs?

When replying include ioFTPD version, addon scripts you are using, and what operating system and how many processors/cores you have.

It's looking increasingly likely that this is some bizarre issue with MS libraries. In particular tcl's use of winsock and/or 3rd party libraries. Since I can confirm the lockups I have information on involve the loader lock it's possible various static runtimes are conflicting or the static runtime is having issues with the dynamic runtime on some operating systems...

I'm looking into recompiling everything against dynamic runtimes and distributing the MS redistributable versions for people to install to see if that fixes things...
Yil is offline   Reply With Quote
Old 03-19-2008, 01:39 AM   #175
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default

Hmm.... what happens when these lockup occurs?.. im curiuos ... cant connect? cant delete files?

I get WarChive error now and then but that i guess is not ioFTPD fault.
Flow is offline   Reply With Quote
Old 03-20-2008, 12:23 PM   #176
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

It depends since you can still do things that don't require you to acquire the loader lock.

Thus you can login (provided you don't have an ident in your hostmask since ident lookup will fail) since the sockets for this were pre-allocated. You can't get directory listings (except via stat -l perhaps) or transfer files though because those require new sockets.

You can't issue any command that ends up creating a process so simple tcl scripts might work, but external processes will just hang.

Site shutdown doesn't work.

Site crashnow amazingly doesn't work either and that's how you can tell you have this problem. It appears that exception handling or simply opening a text file acquires the loader lock ! I specifically wrote the exception handler to not attempt any function in the early part of the routine that is documented to acquire the loader lock. Thus I do no process address lookups, dll loading, etc... All of that was done ahead of time for just this reason and yet I still can't get around this problem.

Basically after a while the process totally locks up, and only killing it from the task manager will cause it to exit.
Yil is offline   Reply With Quote
Old 03-20-2008, 01:36 PM   #177
thomas74
Junior Member
 
Join Date: Dec 2007
Posts: 14
Default

I run the latest build of ioFTPD, latest nxTools and latest ioNinja.
thomas74 is offline   Reply With Quote
Old 03-20-2008, 07:43 PM   #178
FTPServerTools
Senior Member
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: Sep 2002
Posts: 543
Default

Thanks for your view on this Popweasel... And good to see you are still going strong..
FTPServerTools is offline   Reply With Quote
Old 03-22-2008, 05:34 AM   #179
costanza
Junior Member
FlashFXP Registered User
ioFTPD Foundation User
 
Join Date: Jul 2005
Posts: 2
Default

Great script , everything run with no problem.

Great job


ioFTPD-v6.3.5+ioninja v6.3.5.zip+esmnewdir+windows 2003 sp2

Last edited by costanza; 03-22-2008 at 05:35 AM. Reason: add
costanza is offline   Reply With Quote
Old 03-24-2008, 04:56 PM   #180
thomas74
Junior Member
 
Join Date: Dec 2007
Posts: 14
Default

Any progress on that lockup bug ?
thomas74 is offline   Reply With Quote
Reply

Tags
easier, hostmasks, make, problems, support

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 03:07 PM.

Parts of this site powered by vBulletin Mods & Addons from DragonByte Technologies Ltd. (Details)