ioFTPD v6.4.3 Released

[Yil]

Lots of new features and hopefully a fix for the lockup bug...

Let me know how it works.

Latest Version:
ioFTPD-v6.4.3.zip (http://home.comcast.net/~yil/ioFTPD-v6.4.3.zip)

Source:
ioFTPD-v6.4.0-src.zip (http://home.comcast.net/~yil/ioFTPD-v6.4.0-src.zip)

[Yil]

v6.4.0 Release Notes:

*** File Modifications:

1) File system\ioFTPD.exe changed. Version 6.4.0.0

2) File system\ioFTPD.pdb changed.

3) File system\tcl84t.dll and system\tcl84t.pdb changed.

4) File system\ioFTPD.ini changed (Out_Ports, NoFxpOut, NoFxpIn added).

5) FILES: text\ftp\[AllUp,AllDn,MonthUp,MonthDn,WkUp,WkDn,DayUp,DayDn].Footer
changed.


*** New Features:

6) New ioFTPD.ini option (Out_Ports under Devices like [Any]) : This option
allows you to control which ports the server uses for outgoing connections.
If Out_Ports is undefined that means use the old default of Port-1 for the
service initiating the connection. However to avoid "Connection closed:
Only one usage of each socket address (protocol/network address/port) is
normally permitted" errors caused by the receiving server or FTP client
not having a large enough port range you can specify additional local
ports to use. An Out_Ports of 0 means use any port which for almost all
cases eliminates the problem and is the new prefered settings unless you
have a router/gateway that needs you to limit the outgoing port range.

7) It is now possible to view and/or modify the Default.User and Default.Group
settings from within ioFTPD. Simply refer to them as /Default.User or
/Default.Group in any "site change" command, or "site chgrp" to change
the default groups for new users. Currently the only thing you can't set
are IPs for the Default.User although if there is a need this can be
added. To view the current defaults use "site uinfo /Default.User" or
"site ginfo /Default.Group".

8) New site change command (site change <who> nonleech <ratio> [<section>]).
This command is very similar to site change ratio except it will only
change the ratio of users who don't have leech on the section already.
This turns out to be really useful if you used to use 1:3 but now want
to go to 1:4 or something and using "site change * ratio 4" would force
everyone to 1:4 but you probably didn't want siteops, etc to be modified.
Modification of the Default.User or Default.Group setting requires the M
flag.

9) New ioFTPD.ini options (NoFxpIn, NoFxpOut under [VFS]). You can now
specify file paths to deny FXP to/from along with regular permissions
expressions to specify matching users. Thus "NoFxpOut /MP3/* 3" would
make all 3 flag users unable to FXP out from the /MP3 section while
allowing everyone else like siteops who presumably don't have the 3 flag.
The actual test is performed only after the connection is established
because only then is it possible to determine whether the transfer is an
FXP or not... You can still deny a user any FXP no matter what these
settings are via the user flags 'f' and 'F' which take precedance.

10) New "site stat" options (NoZeros and Zeros). When NoZeros is specified
entries that have a 0 in the sorted field will no long be shown. You can
specify the Zeros option to get the current style of output that includes
0 fields. NOTE: The new style of NoZeros might become the default
behavior in the future hence the need for the Zero option which is
currently useless.

11) New "site stat" feature (summations). The requested type of stat
(AllUp, AllDn, MonthUp, MonthDn, WkUp, WkDn, DayUp, or DayDn) now has
a summation counter available for bytes, files, and time that can be
accessed in the associated .Footer file. The %[pos] cookie will be set
to the total number of userfiles that matched in the Footer as well.

12) New "site stat" feature (count 0). Specifying a count of 0 previously
would imply the default of the top 10 matches, but it now means don't
show any matches at all which is useful when you're just interested in
viewing the total via the new summation feature.

Some site stat examples assuming the default text/* files.
NOTE: day stats reset at midnight UTC, weeks at the start of whatever day
you defined as WeeklyReset in ioFTPD.ini, months at the start of
whatever day you defined as MonthlyReset in ioFTPD.ini.
NOTE: Section 0, Byte order, Week Up, and top 10 are the defaults.

site stats monthdn
-- shows the monthly download statistics for the top 10 downloaders by
bytes for the current month along with the total of all downloaders.
site stats section 1 monthdn
-- shows the monthly download statistics in section 1 for the top 10
downloaders for the current month in section 1 and the total of
all downloads in section 1.
site stats count -1 wkup nozeros
-- shows the weekly upload statistics for everybody who uploaded at
least one byte in the current week along with the totals.
site stats alldn limit ".1 .G" count 0
-- show just the total of all the downloads for users with the 1 and/or
G flags.
site stats monthdn files limit ".3 !=GroupA" count -1 nozeros
-- show the monthly download statistics for all users who downloaded at
least 1 file who have the 3 flag but are not in GroupA along with
the monthly totals.

13) New site command (site ioverify [fix]). This command will walk through
the in memory user and group tables and examine all users and groups
and report any inconsistencies found. This command requires the M flag.
If the [fix] argument is supplied it will correct the number of users
in a group if it differs from the number actually discovered. Useful
for deleting a group whose user count is wrong since a group without
users but a user count greater than 0 cannot be deleted.
Command is untested with the shared db modules loaded, but should work.

14) New limited startup error reporting! When ioFTPD is not being run as a
service the server will now pop up an error message indicating why it
failed to start. Except for not being able to read/find the config file
and a few other critical errors the information isn't very detailed yet.
It will report which of the 30 or so modules caused the failure though
and this is much more than we had before!


*** Functionality changes:

15) Added code to IdDataBase_Init to report and reject duplicate id/names,
create a log entry in the SysOp log to report user/groups which are
deleted by a module during initialization (presumably someone deleted
a user/group in a shared database so the id is no longer valid), and
report unparsable lines to the error log.

16) Added code to the generic field parser used by Ascii2GroupFile and
Ascii2UserFile to report lines that couldn't be parsed to the error log.
Unfortunately the Ascii2* functions are exported and hence can't take
extra arguments, and to make things worse the ioFTPD calls are from
functions that don't know the group/user id or name so extra arguments
to a duplicate function aren't possible without lots of changes.
Therefore these errors will be relatively useless for pinpointing the
problem since you'll get the line but nothing else! However seeing
errors in the logfile is an indication a script or something is
having problems.

17) NOOPs no longer update the last action field of the client data
structure if an active transfer is in progress, but does reset the
associated idle timer. This idle timer is used in swho and external
scripts as a record of when the last input from the client was received.
It is not the new internal idle timer used to determine when to kick a
user according to the idle rules. The end result of this is that when
transfering a large file the last action will displayed for a user will
not be a NOOP sent as a keep-alive but the actual transfer command
itself. By keeping the NOOPs being recorded when no transfer is in
progress there should be no impact on external idle kick scripts.

[Yil]

*** Bug Fixes

18) Possible "lockup" bug solution. There were 2 places in the code which
closed a socket and then removed the socket id from a list searched
in a windows callback. It should have been removed first and then
closed to avoid a race condition.

19) Secure_Ip didn't properly initialize a variable and would thus fail the
test for backwards compatibility (no rules defined at all) so it would
reject all addip attempts from non-master accounts.

20) Ident responses now update the static data structure so site swho, shared
memory commands, etc will no longer always have the default "*" in the
ident field.

21) Very rarely the etc/UserIdTable and etc/GroupIdTable files get corrupted
somehow. I assume this is a result of a power failure or a system crash.
I've added code to force the temp file to be flushed to disk along with
it's metadata and asked for the file rename to be immediately written
as well. The rename should be an atomic operation where it either works
or the OS rolls the rename back, so the only thing I can think of is that
the new file wasn't committed to disk but the rename was. Hopefully this
should fix an extremely rare, but very bad, problem.

22) UserIpHostMaskInit would crash if no users were defined at all.

23) In some cases the hostname lookup for the HOST= or BIND= parameters would
timeout or temporarily fail and you would see the following log entries:
[timestamp] Unable to read/resolve host address: xyz.com
[timestamp] Device 'Any' failed to start due to HOST/BIND/PORT error.
Nothing special needs to be done if this occurs at startup since the
next ConfigUpdate event [every 10 minutes by default] will hopefully
fix things when the DNS server returns. On an already active server when
this happens during a ConfigUpdate event the server would stop listening
for new connections until the hostname could be resolved at a future
ConfigUpdate. This is almost always the wrong thing to do and instead it
will now either bind to all addresses, and/or use the previous HOST=
resolved address. It will still report the problem, but this should
allow the server to continue accepting new logins provided the IP really
hasn't changed.

24) Rewrote the logic to avoid strcpy_s and instead use stcncpy_s in the
RecursiveAction function to avoid situations where path+filesnames which
are longer than 260 characters would trigger the safety exception and
cause the server to stop. Instead a log entry with the first 260
characters of the offending directory or file is printed in the
Error.log file.

25) Fixed a bug which resulted in a crash if an unknown user tried to login
and Require_Encrypted_Auth wasn't defined in ioFTPD.ini.

[Zer0Racer]

Nice update! :)

Tried the new stuff for Default.User and I get:
[11:14:21] [L] site change /Default.User flags +L
[11:14:21] [L] 200-Changed flags for user '(null)' from '3' to '3L'.
[11:14:21] [L] 200-Default.User error: Directory/File already exists.
[11:14:21] [L] 200 change Command successful.
[11:15:13] [L] site uinfo /Default.User
[11:15:20] [L] Connection lost: ioFTPD
Yes, it crashed but it did however change the flags in the file.

/ZR

[Yil]

v6.4.1 Release Notes:

*** File Modifications:

1) File system\ioFTPD.exe changed. Version 6.4.1.0

2) File system\ioFTPD.pdb changed.


*** Bug Fixes

3) "site uinfo" called the wrong user close function for regular accounts
which lead to crashes.

4) Eliminated "200-Default.User error: Directory/File already exists" style
bogus messages for Default.User and Default.Group site change commands.

5) Changed 'null' user name in site change replies to "Default.User".

6) Changed regular site group change commands being executed twice.

7) site change commands on Default.User would return "500 Default.Group:
The operation completed successfully." messages instead of "200 change
Command Successful" messages.

[Yil]

If you grabbed 6.4.1 within like 10 minutes of me posting it and use ioSFV, go grab it again. I forgot to add the 6.4.1 version info onto the end of the file that ioSFV likes to read.

[Zer0Racer]

Thanks! Seems to work now. Maybe you could do the same with the "bogus messages" for chgrp?[17:00:23] [L] site chgrp /Default.User user
[17:00:24] [L] 200-Group 'user' has been successfully added to Default.User.
[17:00:24] [L] 500 chgrp: Directory/File already exists.
Haven't used this new function enough to spot other things that might be wrong, but I will keep you posted.

[Yil]

Yea, I'll fix that soon. But since it's not critical let's see what else turns up in the next day or two...

[Zer0Racer]

Of course.

[Yil]

Oh one other tidbit. I'm starting to think there is an error inside the symbolic link resolving logic. Having trouble pinning it down since I can't reproduce it yet.

What I know so far is that app verifier has produced 2 warnings at the same line on a heavily used FTP so that's more than luck I would think. The symbolic link is one of those /latest-xyz links so it's being deleted quickly which may be the cause of the problem and of course makes it hard to debug.

This applies to all versions of ioFTPD (though perhaps only 6.2+ ish) so for the moment there may be some benefit to not using /Latest style symbolic links...

[Yil]

With the release of 6.4.2 all known and reported bugs have hopefully been fixed. So if you spot something wrong, let me know because I think it's working :)


v6.4.2 Release Notes:

*** File Modifications:

1) File system\ioFTPD.exe changed. Version 6.4.2.0

2) File system\ioFTPD.pdb changed.


*** Bug Fixes

3) "site chgrp /Default.User": Eliminated "500 chgrp: Directory/File already
exists" bogus error message.

4) Fixed a v0.5+ memory corruption issue with symbolic links that reference
other symbolic links. Example: /MP3/0101/XYZ is a real dir, /MP3/today ->
/MP3/0101 and /Latest -> /MP3/today/XYZ. If you try to access /Latest
it will likely work but corrupt memory.

[Flow]

Thanks Yil !

[Yil]

Yet another fix to the lockup/crash bug. I think this should at least mask it. However, there is one final option, remove the offending system call as it's not entirely needed and we'll go that route if this isn't stable for people...

v6.4.3 Release Notes:

*** File Modifications:

1) File system\ioFTPD.exe changed. Version 6.4.3.0

2) File system\ioFTPD.pdb changed.


*** Bug Fixes

3) When an active mode (outgoing) connections fails to locally bind a socket
it silently tries a few more times before reporting a failure. However,
it failed to clear the windows socket notifications and internal timer.
The notificiation not being cleared is probably yet another cause of the
lockup/crash bug besides the ones I already fixed.

4) I've now also special cased the lockup/crash bug so it might be "fixed".
However, as an indication that there is still an underlying problem
with the notification code it will output to the error log
"AsyncSelectProc: Invalid socket error". Please let me know if you
see these cropping up.

[Yil]

Good and bad news: The good, no great, news is the server appears stable! The bad news is you will see the occasional new "AsyncSelectProc: Invalid socket error." messages in error.log on a busy server. I'm learning more about how/why we get this but remember this isn't an ERROR anymore, just something that I didn't think should happen that clearly is. It's being handled in a safe way now, just kinda annoying if you happen to see a bunch of those in the error log...

[exitwin98]

Thanks Yil !

[Yil]

:(:(:( Doh! You remember that lockup bug? Well, I spoke too soon. On the 4 processor system it happened again. The "good" news is I found a pattern! Every single pure lockup dump (over 10 now) has been the result of I believe an ioFTPD timeout that forces the socket closed to cancel all the outstanding requests.

The interesting thing here is I don't see how this code can be executed twice since there is a test for that but it appears to be since the timeout return code means the timeout happened but the callstack is not from that call... What's tricky here, is this code is calling winsock library functions that should return errors. They are never supposed to lock up the entire process. So while ioFTPD may be callling them in some improper way, the "lockup" bug is actually a winsock bug. That's my story and I'm sticking to it!

Having found the common thread, and the timeout code forcing a closesocket, I've learned some stuff and I should be able to duplicate the problem locally now and thus be able to solve it.

I just want to get this thing stable so I can start breaking it with new features again :)

FYI: 6.4 will eventually fix this problem, and 6.5 will introduce changes to the userfile for some new options that might break old compiled scripts. TCL based scripts will likely work fine though. I might be able to offer some sort of compatibility mode and hide the new userfile fields or something, but no promises.

[Zer0Racer]

I moved the posts about upcoming v6.5.x to a new thread.

[monk-]

is cap sensitivity still working on the vfs, cos i got ppl trading with /mp3/ while the sfv dir is /MP3/

[monk-]

i figured this is a new functionality

9) ioFTPD is now case insensitive when dealing with mountpoints. File and
directory names were already case insensitive, but mountpoints weren't.

but can u plz make this optional, so i can make it work like gl again, and dont have to make the botscript work case insensitive

[Yil]

Hmm, which botscript is giving you problems? Since every other component of the path is case insensitive it probably doesn't make sense to make just the mountpoints sensitive. Also, if it's reading the vfs file, it's probably one of those scripts I want to be aware of since I'm trying to find ways to support scripts without letting them see server config files by adding tcl or shared memory routines to cover these cases. If there isn't an easy way for me to fix the script or whatever I can probably add an option for you, but I'd rather not do that if possible.

[monk-]

i think its with any dszbot,
now im using an old version of the ported PZS-NG dszbot by o_dog (v0.32?)
and i temp fixed it by simply adding a small caps section:


# What sections are we announcing for?
set sections "MP3 mp3"

# Set up paths for all the sections (wildcards).
set paths(MP3) "/MP3/*"
set chanlist(MP3) $spamchan

set paths(mp3) "/mp3/*"
set chanlist(mp3) $spamchan


and dszbot, and I think any other race announce botscript, uses the ioftpd.log file
and are all cap sensitive


05-10-2008 04:36:11 RACE_AUDIO: /MP3/0510/Some_Mp3_Release-HERE/ racer1 iND {racer3 racer4 racer5} Some_Mp3_Release-HERE 279 04-blabla.mp3 >>unimportant log stuff
05-10-2008 04:36:24 RACE_AUDIO: /mp3/0510/Some_Mp3_Release-HERE/ racer2 iND {racer1 racer3 racer4 racer5} Some_Mp3_Release-HERE 212 02-blabla.mp3 >>unimportant log stuff


since racers are now able to race any case insensitive path, it also is being added with that path to the log

edit: oh, i forgot ioNINJA makes those log entries
and ninja uses the $pwd var

[whocarez2k5]

It's a long time Ã* i have been here so don't know if it was already mentioned.
Is there a way to get total disc space (site free) in ioFTPD?
Now i only get root space and i would like to see total space (C: \ D: \ E: \ F: etc )

[Yil]

The %[Free()] cookie takes real paths and gives the free space on that particular drive. The "site freespace" command is similar but reports the space based on a VFS path rather than a real one.

There are scripts out there that report drive letter and free space, or you could just build a file with things like Drive c = %[Free(c:\)] and create that as a new site command...