ioFTPD v6.4.3 Released

[Yil]

Lots of new features and hopefully a fix for the lockup bug...

Let me know how it works.

Latest Version:
ioFTPD-v6.4.3.zip

Source:
ioFTPD-v6.4.0-src.zip

[Yil]

v6.4.0 Release Notes:

*** File Modifications:

1) File system\ioFTPD.exe changed. Version 6.4.0.0

2) File system\ioFTPD.pdb changed.

3) File system\tcl84t.dll and system\tcl84t.pdb changed.

4) File system\ioFTPD.ini changed (Out_Ports, NoFxpOut, NoFxpIn added).

5) FILES: text\ftp\[AllUp,AllDn,MonthUp,MonthDn,WkUp,WkDn,DayUp,DayDn].Footer
   changed.


*** New Features:

6) New ioFTPD.ini option (Out_Ports under Devices like [Any]) : This option
   allows you to control which ports the server uses for outgoing connections.
   If Out_Ports is undefined that means use the old default of Port-1 for the
   service initiating the connection.  However to avoid "Connection closed:
   Only one usage of each socket address (protocol/network address/port) is
   normally permitted" errors caused by the receiving server or FTP client
   not having a large enough port range you can specify additional local
   ports to use.  An Out_Ports of 0 means use any port which for almost all
   cases eliminates the problem and is the new prefered settings unless you
   have a router/gateway that needs you to limit the outgoing port range.

7) It is now possible to view and/or modify the Default.User and Default.Group
   settings from within ioFTPD.  Simply refer to them as /Default.User or
   /Default.Group in any "site change" command, or "site chgrp" to change
   the default groups for new users.  Currently the only thing you can't set
   are IPs for the Default.User although if there is a need this can be
   added.  To view the current defaults use "site uinfo /Default.User" or
   "site ginfo /Default.Group".

8) New site change command (site change <who> nonleech <ratio> [<section>]).
   This command is very similar to site change ratio except it will only
   change the ratio of users who don't have leech on the section already.
   This turns out to be really useful if you used to use 1:3 but now want
   to go to 1:4 or something and using "site change * ratio 4" would force
   everyone to 1:4 but you probably didn't want siteops, etc to be modified.
   Modification of the Default.User or Default.Group setting requires the M
   flag.

9) New ioFTPD.ini options (NoFxpIn, NoFxpOut under [VFS]).  You can now
   specify file paths to deny FXP to/from along with regular permissions
   expressions to specify matching users.  Thus "NoFxpOut /MP3/* 3" would
   make all 3 flag users unable to FXP out from the /MP3 section while
   allowing everyone else like siteops who presumably don't have the 3 flag.
   The actual test is performed only after the connection is established
   because only then is it possible to determine whether the transfer is an
   FXP or not...  You can still deny a user any FXP no matter what these
   settings are via the user flags 'f' and 'F' which take precedance.

10) New "site stat" options (NoZeros and Zeros).  When NoZeros is specified
    entries that have a 0 in the sorted field will no long be shown.  You can
    specify the Zeros option to get the current style of output that includes
    0 fields.  NOTE: The new style of NoZeros might become the default
    behavior in the future hence the need for the Zero option which is
    currently useless.

11) New "site stat" feature (summations).  The requested type of stat
    (AllUp, AllDn, MonthUp, MonthDn, WkUp, WkDn, DayUp, or DayDn) now has
    a summation counter available for bytes, files, and time that can be
    accessed in the associated .Footer file.  The %[pos] cookie will be set
    to the total number of userfiles that matched in the Footer as well.

12) New "site stat" feature (count 0).  Specifying a count of 0 previously
    would imply the default of the top 10 matches, but it now means don't
    show any matches at all which is useful when you're just interested in
    viewing the total via the new summation feature.

    Some site stat examples assuming the default text/* files.
    NOTE: day stats reset at midnight UTC, weeks at the start of whatever day
          you defined as WeeklyReset in ioFTPD.ini, months at the start of
          whatever day you defined as MonthlyReset in ioFTPD.ini.
    NOTE: Section 0, Byte order, Week Up, and top 10 are the defaults.
 
    site stats monthdn
      -- shows the monthly download statistics for the top 10 downloaders by
         bytes for the current month along with the total of all downloaders.
    site stats section 1 monthdn
      -- shows the monthly download statistics in section 1 for the top 10
         downloaders for the current month in section 1 and the total of
         all downloads in section 1.
    site stats count -1 wkup nozeros
      -- shows the weekly upload statistics for everybody who uploaded at
         least one byte in the current week along with the totals.
    site stats alldn limit ".1 .G" count 0
      -- show just the total of all the downloads for users with the 1 and/or
         G flags.
    site stats monthdn files limit ".3 !=GroupA" count -1 nozeros
      -- show the monthly download statistics for all users who downloaded at
         least 1 file who have the 3 flag but are not in GroupA along with
         the monthly totals.

13) New site command (site ioverify [fix]).  This command will walk through
    the in memory user and group tables and examine all users and groups
    and report any inconsistencies found.  This command requires the M flag.
    If the [fix] argument is supplied it will correct the number of users
    in a group if it differs from the number actually discovered.  Useful
    for deleting a group whose user count is wrong since a group without
    users but a user count greater than 0 cannot be deleted.
    Command is untested with the shared db modules loaded, but should work.

14) New limited startup error reporting!  When ioFTPD is not being run as a
    service the server will now pop up an error message indicating why it
    failed to start.  Except for not being able to read/find the config file
    and a few other critical errors the information isn't very detailed yet.
    It will report which of the 30 or so modules caused the failure though
    and this is much more than we had before!


*** Functionality changes:

15) Added code to IdDataBase_Init to report and reject duplicate id/names,
    create a log entry in the SysOp log to report user/groups which are
    deleted by a module during initialization (presumably someone deleted
    a user/group in a shared database so the id is no longer valid), and
    report unparsable lines to the error log.

16) Added code to the generic field parser used by Ascii2GroupFile and
    Ascii2UserFile to report lines that couldn't be parsed to the error log.
    Unfortunately the Ascii2* functions are exported and hence can't take
    extra arguments, and to make things worse the ioFTPD calls are from
    functions that don't know the group/user id or name so extra arguments
    to a duplicate function aren't possible without lots of changes.
    Therefore these errors will be relatively useless for pinpointing the
    problem since you'll get the line but nothing else!  However seeing
    errors in the logfile is an indication a script or something is
    having problems.

17) NOOPs no longer update the last action field of the client data
    structure if an active transfer is in progress, but does reset the
    associated idle timer.  This idle timer is used in swho and external
    scripts as a record of when the last input from the client was received.
    It is not the new internal idle timer used to determine when to kick a
    user according to the idle rules.  The end result of this is that when
    transfering a large file the last action will displayed for a user will
    not be a NOOP sent as a keep-alive but the actual transfer command
    itself.  By keeping the NOOPs being recorded when no transfer is in
    progress there should be no impact on external idle kick scripts.

[Yil]

*** Bug Fixes

18) Possible "lockup" bug solution.  There were 2 places in the code which
    closed a socket and then removed the socket id from a list searched
    in a windows callback.  It should have been removed first and then
    closed to avoid a race condition.

19) Secure_Ip didn't properly initialize a variable and would thus fail the
    test for backwards compatibility (no rules defined at all) so it would
    reject all addip attempts from non-master accounts.

20) Ident responses now update the static data structure so site swho, shared
    memory commands, etc will no longer always have the default "*" in the
    ident field.

21) Very rarely the etc/UserIdTable and etc/GroupIdTable files get corrupted
    somehow.  I assume this is a result of a power failure or a system crash.
    I've added code to force the temp file to be flushed to disk along with
    it's metadata and asked for the file rename to be immediately written
    as well.  The rename should be an atomic operation where it either works
    or the OS rolls the rename back, so the only thing I can think of is that
    the new file wasn't committed to disk but the rename was.  Hopefully this
    should fix an extremely rare, but very bad, problem.

22) UserIpHostMaskInit would crash if no users were defined at all.

23) In some cases the hostname lookup for the HOST= or BIND= parameters would
    timeout or temporarily fail and you would see the following log entries:
      [timestamp] Unable to read/resolve host address: xyz.com
      [timestamp] Device 'Any' failed to start due to HOST/BIND/PORT error.
    Nothing special needs to be done if this occurs at startup since the
    next ConfigUpdate event [every 10 minutes by default] will hopefully
    fix things when the DNS server returns.  On an already active server when
    this happens during a ConfigUpdate event the server would stop listening
    for new connections until the hostname could be resolved at a future
    ConfigUpdate.  This is almost always the wrong thing to do and instead it
    will now either bind to all addresses, and/or use the previous HOST=
    resolved address.  It will still report the problem, but this should
    allow the server to continue accepting new logins provided the IP really
    hasn't changed.

24) Rewrote the logic to avoid strcpy_s and instead use stcncpy_s in the
    RecursiveAction function to avoid situations where path+filesnames which
    are longer than 260 characters would trigger the safety exception and
    cause the server to stop.  Instead a log entry with the first 260
    characters of the offending directory or file is printed in the
    Error.log file.

25) Fixed a bug which resulted in a crash if an unknown user tried to login
    and Require_Encrypted_Auth wasn't defined in ioFTPD.ini.

[Zer0Racer]

Nice update!

Tried the new stuff for Default.User and I get:

[11:14:21] [L] site change /Default.User flags +L
[11:14:21] [L] 200-Changed flags for user '(null)' from '3' to '3L'.
[11:14:21] [L] 200-Default.User error: Directory/File already exists.
[11:14:21] [L] 200 change Command successful.
[11:15:13] [L] site uinfo /Default.User
[11:15:20] [L] Connection lost: ioFTPD

Yes, it crashed but it did however change the flags in the file.

/ZR

[Yil]

v6.4.1 Release Notes:

*** File Modifications:

1) File system\ioFTPD.exe changed. Version 6.4.1.0

2) File system\ioFTPD.pdb changed.


*** Bug Fixes

3) "site uinfo" called the wrong user close function for regular accounts
   which lead to crashes.

4) Eliminated "200-Default.User error: Directory/File already exists" style
   bogus messages for Default.User and Default.Group site change commands.

5) Changed 'null' user name in site change replies to "Default.User".

6) Changed regular site group change commands being executed twice.

7) site change commands on Default.User would return "500 Default.Group:
   The operation completed successfully." messages instead of "200 change
   Command Successful" messages.

[Yil]

If you grabbed 6.4.1 within like 10 minutes of me posting it and use ioSFV, go grab it again. I forgot to add the 6.4.1 version info onto the end of the file that ioSFV likes to read.

[Zer0Racer]

Thanks! Seems to work now. Maybe you could do the same with the "bogus messages" for chgrp?

[17:00:23] [L] site chgrp /Default.User user
[17:00:24] [L] 200-Group 'user' has been successfully added to Default.User.
[17:00:24] [L] 500 chgrp: Directory/File already exists.

Haven't used this new function enough to spot other things that might be wrong, but I will keep you posted.

[Yil]

Yea, I'll fix that soon. But since it's not critical let's see what else turns up in the next day or two...

[Zer0Racer]

Of course.

[Yil]

Oh one other tidbit. I'm starting to think there is an error inside the symbolic link resolving logic. Having trouble pinning it down since I can't reproduce it yet.

What I know so far is that app verifier has produced 2 warnings at the same line on a heavily used FTP so that's more than luck I would think. The symbolic link is one of those /latest-xyz links so it's being deleted quickly which may be the cause of the problem and of course makes it hard to debug.

This applies to all versions of ioFTPD (though perhaps only 6.2+ ish) so for the moment there may be some benefit to not using /Latest style symbolic links...

[Yil]

With the release of 6.4.2 all known and reported bugs have hopefully been fixed. So if you spot something wrong, let me know because I think it's working

v6.4.2 Release Notes:

*** File Modifications:

1) File system\ioFTPD.exe changed. Version 6.4.2.0

2) File system\ioFTPD.pdb changed.


*** Bug Fixes

3) "site chgrp /Default.User": Eliminated "500 chgrp: Directory/File already
   exists" bogus error message.

4) Fixed a v0.5+ memory corruption issue with symbolic links that reference
   other symbolic links.  Example: /MP3/0101/XYZ is a real dir, /MP3/today ->
   /MP3/0101 and /Latest -> /MP3/today/XYZ.  If you try to access /Latest
   it will likely work but corrupt memory.

[Flow]

Thanks Yil !

[Yil]

Yet another fix to the lockup/crash bug. I think this should at least mask it. However, there is one final option, remove the offending system call as it's not entirely needed and we'll go that route if this isn't stable for people...

v6.4.3 Release Notes:

*** File Modifications:

1) File system\ioFTPD.exe changed. Version 6.4.3.0

2) File system\ioFTPD.pdb changed.


*** Bug Fixes

3) When an active mode (outgoing) connections fails to locally bind a socket
   it silently tries a few more times before reporting a failure.  However,
   it failed to clear the windows socket notifications and internal timer.
   The notificiation not being cleared is probably yet another cause of the
   lockup/crash bug besides the ones I already fixed.

4) I've now also special cased the lockup/crash bug so it might be "fixed".
   However, as an indication that there is still an underlying problem
   with the notification code it will output to the error log
   "AsyncSelectProc: Invalid socket error".  Please let me know if you
   see these cropping up.

[Yil]

Good and bad news: The good, no great, news is the server appears stable! The bad news is you will see the occasional new "AsyncSelectProc: Invalid socket error." messages in error.log on a busy server. I'm learning more about how/why we get this but remember this isn't an ERROR anymore, just something that I didn't think should happen that clearly is. It's being handled in a safe way now, just kinda annoying if you happen to see a bunch of those in the error log...

[exitwin98]

Thanks Yil !