Go Back   FlashFXP Forums > > > >

ioFTPD General New releases, comments, questions regarding the latest version of ioFTPD.

Reply
 
Thread Tools Rate Thread Display Modes
Old 04-22-2008, 01:01 AM   #1
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Doh! You remember that lockup bug? Well, I spoke too soon. On the 4 processor system it happened again. The "good" news is I found a pattern! Every single pure lockup dump (over 10 now) has been the result of I believe an ioFTPD timeout that forces the socket closed to cancel all the outstanding requests.

The interesting thing here is I don't see how this code can be executed twice since there is a test for that but it appears to be since the timeout return code means the timeout happened but the callstack is not from that call... What's tricky here, is this code is calling winsock library functions that should return errors. They are never supposed to lock up the entire process. So while ioFTPD may be callling them in some improper way, the "lockup" bug is actually a winsock bug. That's my story and I'm sticking to it!

Having found the common thread, and the timeout code forcing a closesocket, I've learned some stuff and I should be able to duplicate the problem locally now and thus be able to solve it.

I just want to get this thing stable so I can start breaking it with new features again

FYI: 6.4 will eventually fix this problem, and 6.5 will introduce changes to the userfile for some new options that might break old compiled scripts. TCL based scripts will likely work fine though. I might be able to offer some sort of compatibility mode and hide the new userfile fields or something, but no promises.
Yil is offline   Reply With Quote
Old 04-22-2008, 09:04 AM   #2
Zer0Racer
Senior Member
ioFTPD Scripter
 
Join Date: Oct 2002
Posts: 703
Default ioFTPD v6.5.x discussion

Great news, sort of Winsock is M$ right? Now how is that supposed to ever get fixed? Do different versions of windows use different versions of winsock? Are they all affected?

Question: any plans on removing the telnet/http part of ioftpd? Is it hard to do? Imho they are not needed, not documented and not "finished".
Zer0Racer is offline   Reply With Quote
Old 04-22-2008, 10:26 AM   #3
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Zero: Very good question on the telnet/http part. I was wondering when I would do it too! Probably the next time it gives me any trouble.
Yil is offline   Reply With Quote
Old 04-22-2008, 11:08 AM   #4
Zer0Racer
Senior Member
ioFTPD Scripter
 
Join Date: Oct 2002
Posts: 703
Default

Just thought it might be a good idea to try and remove those before adding alot of new features. ioFTPD still has to be "light-weight"
Zer0Racer is offline   Reply With Quote
Old 04-23-2008, 02:46 AM   #5
FTPServerTools
Senior Member
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: Sep 2002
Posts: 543
Default

I second that. telnet and http (specifically http) is doable per plugin I think.
FTPServerTools is offline   Reply With Quote
Old 04-24-2008, 01:08 PM   #6
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default ioFTPD v6.5.x discussion

ugh, 6.5 will break nxTools ? : /
Flow is offline   Reply With Quote
Old 04-24-2008, 02:57 PM   #7
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Flow: I've got a few goals for 6.5 but the first is I'd like to eliminate scripts accessing ioFTPD configuration files. This means I'll provide a TCL extension and a shared memory solution for access to VFS configuration data, etc. I think I'll also extend the interface a little to support resolving merged directories so scripts can handle that better in the future when disk spanning gets introduced.

The second, and much more serious problem, is I'm going to add fields to the userfile. I made sure to add them to the end of the userfile on purpose to minimize breakage in any 3rd party executable scripts requesting the userfile over the data exchange interface. Compiled scripts that just read existing information from the userfile may work depending on how they allocate buffer space, but those that try to update/store data in userfiles will probably cause problems... TCL scripts usually use the ascii2bin and bin2ascii and thus should work without any changes.

In particular, here are the new fields I'm playing with. Chime in now if you've got a suggestion as these are not permanent yet!

Code:
CreatorUid   - uid of user who created account
CreatorName  - name of creating user in case deleted
CreatedOn    - time account created
LogonCount   - number of successful logins
LogonTime    - time of last successful login
MaxUploads   - number of simultaneous uploads, -1 = unlimited, 0 = denied
MaxDownloads - number of simultaneous downloads, -1 = unlimited, 0 = denied
Opaque       - 256 byte opaque character field for use by scripts,
               sitebot, etc.  Format is "tag" "value" pairs suitable for
               input to TCL.
The created and logon stuff are just useful to have. The Max Up/Down will eliminate the need for external dual leech scripts and allow you to give someone 1 or 2 download slots but 5-10 logins which they can use to mass upload. The Opaque field I view would be used by large scripts and specifically sitebots to stash info like online nicks, etc. I've toyed with the idea of a last created dir, last upload/download time, and official nick support by giving it a whole field. Let me know what you think, or if I've missed something.

FTPServerTools: You have one of the few actively maintained executable addons, so I'm particularly interested in what you need done if you couldn't access configuration files, and what userfile changes mean to you.

nxShareDb: will require a few simple changes to store/retrieve the new fields but should be trivial to do and of course a recompile. The ascii2userfile function already provides default values for userfiles who don't have the new fields so you don't need to worry about any of that.

I'm also thinking of adding support for handling bad and/or abandoned symlinks. Site wipe scripts, automatic space making scripts like WarChive, etc just go ahead and delete directories and the zipscript never gets a chance to cleanup sorted symlinks, incomplete tags, etc. So I'm toying with 2 solutions. Both ideas use the feature that allows you to store extra data in the .ioFTPD file alongside special permissions and symbolic link information like ioSFV uses. I'd make VFS ID 2 and/or 3 contain a list of symbolic links pointing to this dir which the zipscript and the built-in site symlink command can update. Then everyone knows where to look to cleanup those links on deletion or perhaps the server just does it automatically! The other idea is to stash the name of a script that wishes to run when the dir gets deleted/moved/etc. This is a bit more flexible and powerful, and perhaps a special case of a built-in script can handle manually created symlinks. Or perhaps I just do both... call the script first, then automatically cleanup whatever symlinks are still left in the list of known symlinks.

Neither of these cleanup after manually deleted/moved directories outside the FTP, so a regular symlink cleaner would probably still be needed, but this should seriously resolve a lot of wasted disk activity running around looking for links as that could be run on demand or every few days.
Yil is offline   Reply With Quote
Old 04-24-2008, 03:06 PM   #8
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Oh, the timeout bit about the lockup bug was a red herring. It just means the timer went off and updated the error code, but that's because the thread was already stuck and wasn't the cause of the problem.

However, I've figured out some new ways to gather more information but I'm back to waiting for the multi processor machine to lockup again which it conveniently isn't so it can taunt me...
Yil is offline   Reply With Quote
Old 05-05-2008, 10:26 PM   #9
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Quick update: I decided to maintain compatibility with older scripts and kept the existing shared memory interface with a hacked up byte compatible userfile that I convert back and forth to the newer form so things like ioA should still work but without access to the new features. I added 4 new shared memory commands to open/lock/etc the new userfile structure for scripts that can be recompiled.
Yil is offline   Reply With Quote
Old 05-06-2008, 01:19 PM   #10
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default

Yil, you think you can pinpoint a -stable release and save those src code files for future fix before adding new stuff to it? (i, guessing when that "lockup" issue which i never had is fixed, right?)
Flow is offline   Reply With Quote
Old 05-06-2008, 11:09 PM   #11
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Flow: This, the 6.4.X is the stable branch [note the lack of a BETA in the title]

I should probably have started a v6.5 thread for my comments about the upcoming 6.5 release, but I think you're going to really like it
Yil is offline   Reply With Quote
Old 05-07-2008, 10:18 AM   #12
FTPServerTools
Senior Member
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: Sep 2002
Posts: 543
Default

Well I actually dont read the userfiles myself. My main plugins are simple commandline based plugins that get group/user/directory/filename/speed/size/.. parameters as arguments from iotpd. So as long as that is still possible I have very little to do.
Now I do not fiddle in the .ioftpd files myself with FTPLogger and OnDirCreated and OnDirDeleted, I simply check if these plugins are runnign as a subprocess in ioftpd. If so then I know I can use shared memory to recache a directory as my seperate ioRecache does. So it forces io to simply reread the dir and make a new .ioftpd file as such:
Code:
....
	//Now mark the dir as dirty
	lpMessage->dwIdentifier	= DC_DIRECTORY_MARKDIRTY;
    strcpy(PathToRefresh,Path);

	DWORD dwRes=SendMessage(hIoFTPDWindow, WM_SHMEM, NULL, (LPARAM)hMemory);
	if (!dwRes) WaitForSingleObject(hEvent,2000);

	//	Unlock shared memory
	ShMem_Unlock(lpMessage);
...
There is a little piece in ftp logger that force deletes a .ioftpd file if needed, and I use the printf("!vfs:chattr 1 \"%s\" \"%s\"",dir,TagType); on the tag that is created to set the tag owner and rights to the owner and rights of the last uploaded and complete file. I use the USER and GROUP environment variables as well as the UID and GID. I can easily do that due to the check of being a subprocess of ioftpd.
Now the other apps, ioSiteKill, ioRecache, ioSiteWho, ioGUI, ioVFSDLL, ioSymlinkRemover.
ioSymlinkRemover has never been publically released that one altered the .ioftpd file itself, so if it changes I dont really mind. I can simply recode it, as long as I know the structure. But I see no real reason to use it if there is another better mechanism.
ioVFSDLL is never released and is a dll to handle the vfs filesystem and the .ioftpd files so the same goes here.
ioSiteWho does read the user and group files, it basically uses shared memory to loop through all the active users, then grabs their structure and then grabs the needed group and user data from the group and userfiles. Those text data's it uses to do pattern matching after which it can decide to show some info. Basically it is the well known sitewho but then somewhat extended.
THen there is ioSiteKill which was build for ioFTPD 5.x it also loops using shared mem through the connections, then grabs user and group data from the appropriate user and group files. Then also using a wildcard and a build up string I kill a connection e.g. sitekill user!=sitebot path=/test/dummy/* "idletime>120"
So basically for a user I get the full user data from its user name, then get the groupname and then get the groups he is in and then also get the admin groups. The kill then finally happens with a SendMessage using WM_Kill and the iPos (the running number in the connections loop).
ioRecache is rather simple, it allocates the shared memory and then where needed (from the parameters) does a SendMessage with WM_SHMEM which basically forces io to reread the dir and remake the ioftpd file.

So basically if you make big changes then I simply branch off a new version of my tools. Tho I may drop support for older versions if people dont use that anymore.. So far ioSiteKill is the most complicated one.
The ideltime and so I just retrieve from io's memory structure.

Now if it were up to me I would definitely make it simpler, act like it is a database. So the following routines would be my choice.
Give total number of connections
Give connection info for connection(x) including username
Give userinfo for user(y) including group(s)
Give groupinfo for group(z)
That would handle most loops. Knowing that whilst one program loops a user can logout you'd need an active marker at a connection but that was already there in some way in version 4.x.
Alternatively you could make semaphore like commands..
Lock Connection, Lock User, Lock Group and build command queues to a connection, user and group. So if connection 1 ups a file it locks the connection, then finds the user, locks the user, adds some data, unlocks the user, locks the groups, changes some data where needed, unlocks the groups, unlocks the connection. Then another connection of that same user can have an upload with its own changes next in the queue. All that you'd need in io is to avoid deadlocking which a timeout would handle.
Storing data in a database, an ini file, the rgistry, remotely in mysql or whereever would only need some plugin to read/write/update/delete. And it could even be simply written in tcl or lua or vbscript or powershell. Anyway those are just my ideas of course (I got tons of ideas as usual..).
I had build my tools on purpose to support more ftp servers and to be as flexible as possible so only the tools that start with io are ioftpd only...
FTPServerTools is offline   Reply With Quote
Old 05-07-2008, 03:40 PM   #13
Zer0Racer
Senior Member
ioFTPD Scripter
 
Join Date: Oct 2002
Posts: 703
Default ioFTPD v6.5.x discussion

Moved some posts from v6.4.3 thread to here.

Edit: and copied the initial post from Yil that started the discussion

Last edited by Zer0Racer; 05-07-2008 at 03:48 PM.
Zer0Racer is offline   Reply With Quote
Old 05-08-2008, 02:05 AM   #14
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Thanks a lot for that really in depth review of how you use the shared memory interface. The good news is, you won't have to change anything to keep all existing functionality and everything existing will still work now that I've decided keeping compatibility was a high priority again. I'm running an untouched ioGUI in shared memory mode for testing and it appears to be fine and it uses a LOT of shared memory functionality.

If you continue to recompile against 6.4 headers you don't need to do anything at all, if you switch to 6.5 headers you'll need to either change UserFile references to UserFile_Old in your code, or change to the newer DC_NEW_USERFILE_[OPEN,CLOSE,etc] identifiers to manipulate the new UserFile structures which would then give you access to all the new fields and 25 sections instead of 10.

If you can let me in on your custom ioSiteWho additions I'll see what I can do. I've added some new features to site swho, and provided a new site who that basically does all SiteWho.exe did but is designed for regular users if you allow it in ioFTPD.ini. You can make it hide users, hide paths, show just uploaders, downloaders, idlers or just a summary. It uses new text/ftp/Who.* files so people can customize the output although I've updated more than half the files in there for the 6.5 release. I've even added support for conditional cookies so you can print some info, and hide other stuff based on what cookies evaluate to. Kinda cool stuff, but if there is something you use that you don't see let me know and I'll add it

I'm also planning on going the glftpd route with site deluser only marking an account as deleted and providing an optional message to show the user if they try to login. You'll have to use site purge to actually remove the user and of course you can use site readd to restore the user and you can only purge people lower than you. So 1 flagged users can delete but not purge other 1 users, but can do whatever to group admins, etc...
Yil is offline   Reply With Quote
Old 05-09-2008, 10:58 AM   #15
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default

How about implementing "mode-z" standard?
Flow is offline   Reply With Quote
Reply

Tags
bug, code, ioftpd, lockup, timeout

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 02:31 PM.

Parts of this site powered by vBulletin Mods & Addons from DragonByte Technologies Ltd. (Details)