ioFTPD v6.5.x discussion

[Yil]

:(:(:( Doh! You remember that lockup bug? Well, I spoke too soon. On the 4 processor system it happened again. The "good" news is I found a pattern! Every single pure lockup dump (over 10 now) has been the result of I believe an ioFTPD timeout that forces the socket closed to cancel all the outstanding requests.

The interesting thing here is I don't see how this code can be executed twice since there is a test for that but it appears to be since the timeout return code means the timeout happened but the callstack is not from that call... What's tricky here, is this code is calling winsock library functions that should return errors. They are never supposed to lock up the entire process. So while ioFTPD may be callling them in some improper way, the "lockup" bug is actually a winsock bug. That's my story and I'm sticking to it!

Having found the common thread, and the timeout code forcing a closesocket, I've learned some stuff and I should be able to duplicate the problem locally now and thus be able to solve it.

I just want to get this thing stable so I can start breaking it with new features again :)

FYI: 6.4 will eventually fix this problem, and 6.5 will introduce changes to the userfile for some new options that might break old compiled scripts. TCL based scripts will likely work fine though. I might be able to offer some sort of compatibility mode and hide the new userfile fields or something, but no promises.

[Zer0Racer]

Great news, sort of :) Winsock is M$ right? Now how is that supposed to ever get fixed? Do different versions of windows use different versions of winsock? Are they all affected?

Question: any plans on removing the telnet/http part of ioftpd? Is it hard to do? Imho they are not needed, not documented and not "finished".

[Yil]

Zero: Very good question on the telnet/http part. I was wondering when I would do it too! Probably the next time it gives me any trouble. :)

[Zer0Racer]

Just thought it might be a good idea to try and remove those before adding alot of new features. ioFTPD still has to be "light-weight" :p

[FTPServerTools]

I second that. telnet and http (specifically http) is doable per plugin I think.

[Flow]

ugh, 6.5 will break nxTools ? : /

[Yil]

Flow: I've got a few goals for 6.5 but the first is I'd like to eliminate scripts accessing ioFTPD configuration files. This means I'll provide a TCL extension and a shared memory solution for access to VFS configuration data, etc. I think I'll also extend the interface a little to support resolving merged directories so scripts can handle that better in the future when disk spanning gets introduced.

The second, and much more serious problem, is I'm going to add fields to the userfile. I made sure to add them to the end of the userfile on purpose to minimize breakage in any 3rd party executable scripts requesting the userfile over the data exchange interface. Compiled scripts that just read existing information from the userfile may work depending on how they allocate buffer space, but those that try to update/store data in userfiles will probably cause problems... TCL scripts usually use the ascii2bin and bin2ascii and thus should work without any changes.

In particular, here are the new fields I'm playing with. Chime in now if you've got a suggestion as these are not permanent yet!

CreatorUid - uid of user who created account
CreatorName - name of creating user in case deleted
CreatedOn - time account created
LogonCount - number of successful logins
LogonTime - time of last successful login
MaxUploads - number of simultaneous uploads, -1 = unlimited, 0 = denied
MaxDownloads - number of simultaneous downloads, -1 = unlimited, 0 = denied
Opaque - 256 byte opaque character field for use by scripts,
sitebot, etc. Format is "tag" "value" pairs suitable for
input to TCL.



The created and logon stuff are just useful to have. The Max Up/Down will eliminate the need for external dual leech scripts and allow you to give someone 1 or 2 download slots but 5-10 logins which they can use to mass upload. The Opaque field I view would be used by large scripts and specifically sitebots to stash info like online nicks, etc. I've toyed with the idea of a last created dir, last upload/download time, and official nick support by giving it a whole field. Let me know what you think, or if I've missed something.

FTPServerTools: You have one of the few actively maintained executable addons, so I'm particularly interested in what you need done if you couldn't access configuration files, and what userfile changes mean to you.

nxShareDb: will require a few simple changes to store/retrieve the new fields but should be trivial to do and of course a recompile. The ascii2userfile function already provides default values for userfiles who don't have the new fields so you don't need to worry about any of that.

I'm also thinking of adding support for handling bad and/or abandoned symlinks. Site wipe scripts, automatic space making scripts like WarChive, etc just go ahead and delete directories and the zipscript never gets a chance to cleanup sorted symlinks, incomplete tags, etc. So I'm toying with 2 solutions. Both ideas use the feature that allows you to store extra data in the .ioFTPD file alongside special permissions and symbolic link information like ioSFV uses. I'd make VFS ID 2 and/or 3 contain a list of symbolic links pointing to this dir which the zipscript and the built-in site symlink command can update. Then everyone knows where to look to cleanup those links on deletion or perhaps the server just does it automatically! The other idea is to stash the name of a script that wishes to run when the dir gets deleted/moved/etc. This is a bit more flexible and powerful, and perhaps a special case of a built-in script can handle manually created symlinks. Or perhaps I just do both... call the script first, then automatically cleanup whatever symlinks are still left in the list of known symlinks.

Neither of these cleanup after manually deleted/moved directories outside the FTP, so a regular symlink cleaner would probably still be needed, but this should seriously resolve a lot of wasted disk activity running around looking for links as that could be run on demand or every few days.

[Yil]

Oh, the timeout bit about the lockup bug was a red herring. It just means the timer went off and updated the error code, but that's because the thread was already stuck and wasn't the cause of the problem.

However, I've figured out some new ways to gather more information but I'm back to waiting for the multi processor machine to lockup again which it conveniently isn't so it can taunt me...

[Yil]

Quick update: I decided to maintain compatibility with older scripts and kept the existing shared memory interface with a hacked up byte compatible userfile that I convert back and forth to the newer form so things like ioA should still work but without access to the new features. I added 4 new shared memory commands to open/lock/etc the new userfile structure for scripts that can be recompiled.

[Flow]

Yil, you think you can pinpoint a -stable release and save those src code files for future fix before adding new stuff to it? (i, guessing when that "lockup" issue which i never had is fixed, right?)

[Yil]

Flow: This, the 6.4.X is the stable branch [note the lack of a BETA in the title] :)

I should probably have started a v6.5 thread for my comments about the upcoming 6.5 release, but I think you're going to really like it :)

[FTPServerTools]

Well I actually dont read the userfiles myself. My main plugins are simple commandline based plugins that get group/user/directory/filename/speed/size/.. parameters as arguments from iotpd. So as long as that is still possible I have very little to do.
Now I do not fiddle in the .ioftpd files myself with FTPLogger and OnDirCreated and OnDirDeleted, I simply check if these plugins are runnign as a subprocess in ioftpd. If so then I know I can use shared memory to recache a directory as my seperate ioRecache does. So it forces io to simply reread the dir and make a new .ioftpd file as such:

....
//Now mark the dir as dirty
lpMessage->dwIdentifier = DC_DIRECTORY_MARKDIRTY;
strcpy(PathToRefresh,Path);

DWORD dwRes=SendMessage(hIoFTPDWindow, WM_SHMEM, NULL, (LPARAM)hMemory);
if (!dwRes) WaitForSingleObject(hEvent,2000);

// Unlock shared memory
ShMem_Unlock(lpMessage);
...

There is a little piece in ftp logger that force deletes a .ioftpd file if needed, and I use the printf("!vfs:chattr 1 \"%s\" \"%s\"",dir,TagType); on the tag that is created to set the tag owner and rights to the owner and rights of the last uploaded and complete file. I use the USER and GROUP environment variables as well as the UID and GID. I can easily do that due to the check of being a subprocess of ioftpd.
Now the other apps, ioSiteKill, ioRecache, ioSiteWho, ioGUI, ioVFSDLL, ioSymlinkRemover.
ioSymlinkRemover has never been publically released that one altered the .ioftpd file itself, so if it changes I dont really mind. I can simply recode it, as long as I know the structure. But I see no real reason to use it if there is another better mechanism.
ioVFSDLL is never released and is a dll to handle the vfs filesystem and the .ioftpd files so the same goes here.
ioSiteWho does read the user and group files, it basically uses shared memory to loop through all the active users, then grabs their structure and then grabs the needed group and user data from the group and userfiles. Those text data's it uses to do pattern matching after which it can decide to show some info. Basically it is the well known sitewho but then somewhat extended.
THen there is ioSiteKill which was build for ioFTPD 5.x it also loops using shared mem through the connections, then grabs user and group data from the appropriate user and group files. Then also using a wildcard and a build up string I kill a connection e.g. sitekill user!=sitebot path=/test/dummy/* "idletime>120"
So basically for a user I get the full user data from its user name, then get the groupname and then get the groups he is in and then also get the admin groups. The kill then finally happens with a SendMessage using WM_Kill and the iPos (the running number in the connections loop).
ioRecache is rather simple, it allocates the shared memory and then where needed (from the parameters) does a SendMessage with WM_SHMEM which basically forces io to reread the dir and remake the ioftpd file.

So basically if you make big changes then I simply branch off a new version of my tools. Tho I may drop support for older versions if people dont use that anymore.. So far ioSiteKill is the most complicated one.
The ideltime and so I just retrieve from io's memory structure.

Now if it were up to me I would definitely make it simpler, act like it is a database. So the following routines would be my choice.
Give total number of connections
Give connection info for connection(x) including username
Give userinfo for user(y) including group(s)
Give groupinfo for group(z)
That would handle most loops. Knowing that whilst one program loops a user can logout you'd need an active marker at a connection but that was already there in some way in version 4.x.
Alternatively you could make semaphore like commands..
Lock Connection, Lock User, Lock Group and build command queues to a connection, user and group. So if connection 1 ups a file it locks the connection, then finds the user, locks the user, adds some data, unlocks the user, locks the groups, changes some data where needed, unlocks the groups, unlocks the connection. Then another connection of that same user can have an upload with its own changes next in the queue. All that you'd need in io is to avoid deadlocking which a timeout would handle.
Storing data in a database, an ini file, the rgistry, remotely in mysql or whereever would only need some plugin to read/write/update/delete. And it could even be simply written in tcl or lua or vbscript or powershell. Anyway those are just my ideas of course (I got tons of ideas as usual..).
I had build my tools on purpose to support more ftp servers and to be as flexible as possible so only the tools that start with io are ioftpd only... :)

[Zer0Racer]

Moved some posts from v6.4.3 thread to here.

Edit: and copied the initial post from Yil that started the discussion

[Yil]

Thanks a lot for that really in depth review of how you use the shared memory interface. The good news is, you won't have to change anything to keep all existing functionality and everything existing will still work now that I've decided keeping compatibility was a high priority again. I'm running an untouched ioGUI in shared memory mode for testing and it appears to be fine and it uses a LOT of shared memory functionality.

If you continue to recompile against 6.4 headers you don't need to do anything at all, if you switch to 6.5 headers you'll need to either change UserFile references to UserFile_Old in your code, or change to the newer DC_NEW_USERFILE_[OPEN,CLOSE,etc] identifiers to manipulate the new UserFile structures which would then give you access to all the new fields and 25 sections instead of 10.

If you can let me in on your custom ioSiteWho additions I'll see what I can do. I've added some new features to site swho, and provided a new site who that basically does all SiteWho.exe did but is designed for regular users if you allow it in ioFTPD.ini. You can make it hide users, hide paths, show just uploaders, downloaders, idlers or just a summary. It uses new text/ftp/Who.* files so people can customize the output although I've updated more than half the files in there for the 6.5 release. I've even added support for conditional cookies so you can print some info, and hide other stuff based on what cookies evaluate to. Kinda cool stuff, but if there is something you use that you don't see let me know and I'll add it :)

I'm also planning on going the glftpd route with site deluser only marking an account as deleted and providing an optional message to show the user if they try to login. You'll have to use site purge to actually remove the user and of course you can use site readd to restore the user and you can only purge people lower than you. So 1 flagged users can delete but not purge other 1 users, but can do whatever to group admins, etc...

[Flow]

How about implementing "mode-z" standard?

[Yil]

I suppose finding a GNU based compression library wouldn't be that difficult or since it's pretty simple I could maybe even roll my own, but I'm not sure it's worth implementing. Directory listings get great compression but they really aren't very big and given that almost everything else is already compressed it's a lot of extra computing costs for no gain as I think it defaults to on for all transfers once enabled.

Do you find that servers that support mode-z are giving you good compression ratios on some types of files? Are they big files?

[Yil]

Quick update: I think I've finished coding for v6.5.0. There are really a LOT of changes for this release because I was hoping to get most of the userfile additions done at once and that meant new cookies, new site change commands, etc for all of new fields...

It's probably still several days away, but hopefully you should see it soon.

[Flow]

Yey, good news.
/me love new toys

:p

[Zer0Racer]

It's good to know that "soon" actually means soon nowadays ^^