ioFTPD v7.7.3 (STABLE) Released

Yil · 04-26-2011, 04:49 PM

UPDATE: v7.7.2 (it must be .2) appears really stable. So far nobody has reported any crashes and the few sites I can see all have nice uptimes. If you have had stability problems in the past try this version. The switch to OpenSSL might have solved the lockup bug as I hoped (a bizarre dll loader lock bug probably caused by the MS encryption library always triggering dll loads/unloads), but it wasn't until I noticed I screwed up and didn't register locking functions with OpenSSL that it became thread safe and the random crashes stopped.

Looks like I've uncovered a new serious bug in the directory caching code. Probably a good thing I decided to scrap all the existing code are re-write it huh? The previous serious bug I fixed in v7.6 appears to have helped a number of people with stability. I'm hoping this one will as well. This particular bug has to do with the directory locking code that in some cases with a lock holder and 2 waiters leaks handles and does other bad things. The good news has to be that I'm finding these bugs (in code that I didn't write!), the bad news is I didn't find them earlier when I looked before.

For this release I've decided to grab all the pure bug fixes and the "simpler" new features I added for v8 along and put them into this v7.7 release. I'm also considering taking the rest of the v8 changes like the new configuration module which affects a lot of code, user modifiable site flags and the things that depend on that, and TCL access to the ban lists and putting out a v7.8 release That would leave v8 with just the new directory caching stuff so it wouldn't seem like a big deal anymore but it might be the better way to go.

Highlights
* Matching list ('site user' plus the '--' and '-+' glob prefixes)
* Single-user/admin site close
* Login/logout logfile suppression via user flag
* Limited Banned user support
* Several user-friendly error messages added
* A heck of a lot of bug fixes.

NOTE: Both the TCL/OpenSSL libraries are upgraded and so is the /lib dir so I suggest making a backup of your current ioFTPD directory before upgrading.

**** NOTE: These links will be updated soon. Evidently my ISP has stopped supporting FTP/web pages for all it's users and the links are broken.

Latest Version:
Link: ioFTPD-v7.7.3.zip

STABLE Version:
Link: ioFTPD-v7.7.2.zip

SOURCE:
Link: ioFTPD-v7.7.3-src.zip

Yil · 04-26-2011, 04:51 PM

Code:

v7.7.0 Release Notes:

1) Files in \System:
   Changed : ioFTPD.[exe,pdb] - Version 7.7.0.0
   Changed : tcl85t.[dll,pdb] - Version 8.5.2.9
   Changed : libeay32.dll, ssleay32.dll - Version 1.0.0.4
     Added : libeay32.pdb, ssleay32.pdb - Version 1.0.0.4
   Changed : Help-SiteCmds.ini
   Changed : ioFTPD.ini - summary of changes by section...
     [FTP_Service] : OpenSSL_Options options and comments changed.
     [VFS]         : Default_Directory_Attributes    = 777 0:0
     [FTP_SITE_Permissions] : Rename "sectionnum" to "sectionums"
                              stat              =     *
     [Ftp]         : Single_Closed_Exempt_Name added
                     Who_Sort_Output added
                     Who_Hidden_Users comment changed
                     Banned_User_Flag added
                     Quiet_Login_Flag
     [Threads]     : Io_Threads              = 4
                     Remove Encryption_Threads option.
                     Section commands changed.

2) Directories in \lib:
     Replace entire reg1.2 directory.
     Replace entire dde1.3 directory.
     Replace entire tcl8 directory.
     The tcl8.5 directory is a bit trickier.  Probably the easiest thing to
       do is copy and overwrite all the files from the new directory into your
       existing dir.  For the record, here's some info on the subdirs:
         TCL: http1.0, msgs, opt0.4, tzdata (encoding not shipped w/ioFTPD)
         ioNinja-v0.8.9.6-2009-06-29: encoding, http, http2.5, twapi, zlib1.1
           *** nxHelper shipped with ioNinja is v2.3 but you need the v2.4
               one from nxTools else you get package require errors!
           *** opt0.4 is part of TCL, use the one shipped with ioFTPD
           *** reg1.1 is old, ioFTPD includes lib/reg1.2 which is better
     lib/nxHelper and lib/sqlite3 are part of nxTools

3) Files in \text\ftp:
   Changed : GroupList.[Header, Body]
   Changed : MyInfo.[Sections, Totals]
   Changed : UserInfo.[Sections, Totals]
   Changed : UserList.Footer

4) Files in \Doc:
   Changed : Cookies.txt
   Changed : Events.txt
   Changed : iTCL.txt
   Changed : Site-cmds.txt

5) Files in \Source
   Changed : Site-cmds.help


*** New Features:

6) New site command (site stat).  Displays all the information normally
   displayed after a file transfer including server close / shutdown
   notifications, any personal messages, and the TransferComplete file with
   the statline(s) which usually contains things like section, ratio, credits
   stats, freespace, etc for the current dir.  The speed will be from the last
   data channel transfer be it from a file transfer or a directory listing.

7) Modified site command (site users [ -errors ] [ -- | -+ ] [ <user-glob> ]).
   Previously this command just listed users matching the <user-glob> pattern
   or returned a list of all known users if no arguments were given.  That
   still works the same, however the list of matching users is now remembered
   and called the 'matching' list.  You can then refer to the users in the
   matching list via the '--' and '-+' syntax:
     '--' => start with the current matching list, but remove users from it
             who don't match the new <user-glob>.
     '-+' => add previously non-matching users to the list who match the new
             <user-glob>.

   Because all negative search terms in <user-glob> are processed first the
   ability to iteratively include / exclude users is much more powerful.
   Consider the command 'SITE USERS !V .1 -Bob' which will find all SiteOps
   and the user Bob provided they aren't also VFS Admins.  Now compare that
   to 'SITE USERS !V .1" followed by 'SITE USERS -+ Bob' which will match all
   SiteOps who aren't also VFS Admins and the user Bob.  You can see that if
   Bob happened to be a VFS Admin you would get different results.  In fact
   ONLY by using the matching list could you get the results you probably
   wanted.

   Now for the really cool bit.  The '--' and '-+' prefixes that refer to
   the matching list are now valid arguments to these other site commands:
   'SITE [S]WHO', 'SITE STATS', and 'SITE CHANGE'.

   This is really useful because it's sometimes tricky to craft a 'site
   change' command to modify just the users you wanted.  At the minimum it
   has always been a good idea to use 'SITE USERS <user-glob>' to see who
   would be effected, but with the new matching list feature you can view the
   list of users, make sure it's correct, and use 'site change --' to modify
   only the users you just verified.

   If there are any users records who couldn't be accessed the command will
   report a count of them if the caller has the Master flag.  You can then
   use the '-error' option to see what usernames had problems.  Errors should
   only be possible with the use of a 3rd party shared user module like
   nxMyDB.

8) Modified site change commands (site change <target> { homedir | vfsfile |
   tagline | opaque | groupvfsfile | groupdescription } <none>).  You can
   now specify "<none>" as the new value to any of the above change commands
   and the field will be cleared.  This is important because some commands
   like 'site change tagline' had no way to clear a user's tagline once set.
   Some commands like site change vfsfile and groupvfsfile still accept
   "default" to clear the field, and site change groupdescription will allow
   "none" to do the same thing, but "<none>" is now standardized across them
   all.

9) By default the server will no longer accept logins (except from a single
   exempt user defined in the .ini file) until all OnServerStart events have
   finished.  I'm not aware of any 3rd party scripts except NxMyDB who
   currently use this event and the new guarantee should be useful to it.

10) Modified site command (site close [ -single ] [ { -kick | -new } ]
    [<reason>]).  Site close now accepts the '-single' option (available only
    to 'M' flagged accounts) which puts the server into "single user" or
    "administrative" mode.  This mode doesn't exempt the users defined by
    the 'Close_Exempt' .ini option, so the only users allowed to stay logged
    in is the user issuing the command and the user defined by the new
    'Single_Closed_Exempt_Name' .ini option.  Ordinary users will see the
    normal site close message, users who would normally be exempt from site
    closure will now see this additional line when trying to login:
      530-ADMIN MODE CLOSURE BY '<username>'.
    This new option can be useful if you have a bunch of siteops and you want
    to close the site but they are all normally exempt.  It also provides a
    way to perform actions best done without any active transfers.

11) New ioFTPD.ini option (Single_Closed_Exempt_Name under [FTP]).  The single
    username defined here can always login to the server even if the server
    is in single-user / administrative mode or if the startup scripts are
    still running.
    WARNING: ioGUI uses the ioFTPD account and probably should NOT be running
    if single user mode is entered to do things like renumbering sections,
    mass modification of userfiles, etc because it won't pickup the changes
    so just disconnect and reconnect when done.

12) New ioFTPD.ini option (Banned_User_Flag under [FTP]).  This defines a
    single user flag character to indicate a user has been banned from logging
    in.  By default, the user will see the message
      'Account has been temporarily banned'
    if they try to login, however the actual text may be different if using
    an OnBannedLogin event.  Default is 'B'.  Leave undefined to disable.
    NOTE: The ability to set a ban time for users is not currently provided
          because it would require a change to the userfile or some tricks
          with overloading a field and that could cause issues with nxMyDB
          and/or 3rd party scripts.  However a very simple script and a new
          scheduler event should be able to add the ability while also showing
          the reason for the ban when the user tries to log in.  In the future,
          when there are enough changes to make it worth modifying the
          userfile, these features will be supported by ioFTPD itself.
    NOTE: Banning a user doesn't automatically kick them if already connected.

13) New event (OnBannedLogin).  This event is run whenever a user tries to
    login and has the Banned_User_Flag as one of their user flags.  Treated
    the same as OnExpiredLogin and OnDeletedLogin which means if the event
    returns an error code the generic 'Invalid password' error is displayed
    instead.

14) New ioFTPD.ini option (Quiet_Login_Flag under [FTP]).  This defines a
    single user flag character to indicate that normal LOGIN: and LOGOUT:
    log entries to ioFTPD.log should not be made.  This flag is very useful
    to prevent BNC log spam.  Default is 'Q'.  Leave undefined to disable.

15) New ioFTPD.ini option (Who_Sort_Output under [FTP]).  If set to True then
    the 'site who' output is case-insensitive sorted rather than shown in
    connection-id-slot (like site swho) since this is more useful to users.

16) Site Who for 1M flagged users no longer obscures the arguments to site
    commands, or hides paths.  It will, however, apply the Who_Hidden_Users
    attribute provided the hidden user isn't actively transferring a file.
    This still allows you to filter out sitebots, ioGUI, etc. from listings
    provided they aren't consuming bandwidth in which case you probably want
    to know about it as a SiteOp.

17) Site Who is a even more restrictive about hiding a user's actions if the
    user is determined to be in or transferring from a hidden path.

18) If the user has logged in with a '-' password prefix indicating they are
    a bot/script and wish for only single line command responses the server
    will no longer display the .ioFTPD.cwd and .ioFTPD.messages files when
    the bot CWD's into a directory and it will disable theme support.

19) The server will now return more descriptive login failure messages in a
    couple of cases if a valid username and password has been supplied.  In
    particular, if the VFS file specified by the groupfile, userfile, or the
    system default .vfs file is unable to be opened you will now see:
      530 Login failed: Error with VFS file.
    and if the VFS file was able to be loaded but the user's home directory
    or the '/' directory were unable to be resolved you will see:
      530 Login failed: Error with home directory.
    This should favorably resolve one of the most common issues new ioFTPD
    administrators have since lots of people seem to forget to setup the .vfs
    file and the 'Invalid password' error was misleading.

20) The server will now verify that the default TCL initialization script can
    be run successfully, or it will fail to start with an error message and a
    logfile entry.  This is particularly important because someone (including
    me!) will forget which version of TCL libraries is in the /lib directory
    and won't figure out why scripts are failing in weird ways.  The most
    common indication you had a TCL dll/lib mismatch is with the [clock]
    command not working.

21) Initialization errors with TCL will still report that the 'Event' module
    failed, but will now mention it was a TCL version issue.

22) ioFTPD error messages to the logfile during initialization will now record
    the textual error message if the log system has been initialized.

23) New super cookies (%[Max]).  Valid when processing UserList.Footer this is
    the total number of users found.  %[Pos] holds the total matching users.

24) %[Ratio(999)] now returns the LeechName setting from [Ftp] section in the
    .ini file, i.e. Leech, Unlimited, etc.

25) When processing [UserInfo,MyInfo].Totals the %[CreditSection] cookie will
    be set to -1 if per-section stats shown, else 0 and %[ShareSection] 
    will be the total number of sections defined.    

26) Added a "safety" word to all AllocateShared() memory allocations and to
    all FileInfo's that is checked prior to decrementing the reference count.
    If corruption is detected an entry is made to the Error log and details
    about the callstack are made to the Debug log.  If this does get triggered
    perhaps a pattern will emerge so we can find the problem.

27) Updated OpenSSL to v1.0.0d.


*** iTCL Changes

28) Updated TCL to v8.5.9.

29) New TCL command (ioServer { open | closedfor | close ... }).
    open      : Returns -1 if server already open, else the time the server
                was originally closed.
    closedfor : Returns -1 if the server is not closed, else a tuple like
                { time-server-closed close-msg-reason }
    close [ -single <username> ] [ -new ] [ <reason> ]
                Returns 0 if the server was just closed else the time it was
                previously closed.  The -single <username> option enables
                single user mode for the given username.  The -new option
                prevents logging non-exempt users off and just prohibits new
                users.  Unlike 'site close' this command doesn't offer a way
                to immediately kick users, just use [client kill] after the
                server has been closed.
    NOTE: The open/close commands generate ioFTPD.log events just like site
          open/close would.  The user/group names displayed in the log are
          based upon the currently active userfile, else "<unknown>" if not
          available.
    With programmatic access to the site open / close command and the new
    guarantee that all OnServerStart events will be executed before the server
    accepts connections you can now start a server up closed to logins and
    keep it that way while performing operations that are best done without
    anyone online.

30) New TCL command option ([user SingleLineResponse]).  Returns 1 if the '-'
    prefix was used on password during login indicating a bot/script, else 0.


*** Bug Fixes:

31) Fixed a bug with "Invalid symbolic path (pwd was <path>)." lines showing
    up in the error log improperly.  This created a lot of log spam when
    scripts like nxAutoNuke would run as the [resolve pwd] calls would
    trigger the bug.

32) Fixed a bug where the individual directory lock for cached dirs wasn't
    handling all cases correctly.  If the directory was locked and there were
    2+ other threads waiting on the event created to hold waiters and then the
    directory was release by the original owner and another waiter was added
    a new event would be created to hold waiters even though there were threads
    waiting on the original event still!  I'm not sure exactly what happens
    besides the event handle being leaked, but it's possible users could get
    stuck on the abandoned event, multiple users could "own" the lock, etc...

33) Fixed a bug where the RETR command was decreasing the active download
    count immediately as well as after the download finished.  This meant
    that the active download count goes increasing negative and rendered the
    max concurrent download setting for the user ineffective.

34) Fixed a bug where the STOR command could leave the active upload count
    incremented if the server had trouble creating / appending the file.

35) Fixed a bug with the STOR command where the data channel could be reset
    twice if the download failed in certain situations like not enough
    credits, etc.

36) Fixed a bug with AllocateBuffer() that always doubled the size of the
    buffer even if the minimum size requested was met.

37) Fixed a bug where invalid login attempts may leave the mountfile open.

38) Fixed a race condition with newly registered connections.

39) Fixed a bug where Login_Attempts = 1 under [FTP] would prevent users
    from logging in.  Default is 3.

40) Fixed a bug in determining if a path is a mountpoint or not.

41) Fixed a bug with message files not ending with \r\n generating an extra
    blank line.

42) Fixed a race condition on counting total logins for a user.

43) Fixed a race condition on the device share count.

44) Fixed a bug in PWD_Copy() that didn't handle pure virtual directories
    properly and could result in the server crashing in some cases.

45) Fixed a bug where the virtual event field associated with an item that
    resolved to a real path in a pure virtual directory wasn't being updated.

46) Fixed a bug if both sides of the %[IF] super cookie were zero length.

47) Fixed a bug in PWD_CWD2() which didn't close an open FileInfo which would
    lead to a memory leak if the directory was later updated or flushed from
    the cache.

48) Fixed a bug where the data channel socket might not be initialized before
    its first use if the first data transfer command failed.

49) Fixed a bug where it might be possible to close the wrong socket if the
    server ran out of passive/active local ports.

50) Fixed a memory leak when displaying a list of users for site re-add/purge.

51) Fixed two memory leaks with 'site grpdel'.

52) Fixed a memory leak in 'site help'.

53) Fixed a bug in DeleteLockObject() not marking the 2nd handle as invalid.

54) Fixed a memory reference count issue in 'site devices|services'.

55) Fixed a memory leak in 'site ioverify'.

56) Fixed a bug in 'site perms' which didn't close an open FileInfo, and fixed
    a pathname memory leak.

57) Fixed a bug with 'site perms -<yourself>' which would improperly update
    your userfile reference count.

58) Fixed a bug in 'site chattr' which wasn't closeing the FileInfo.

59) Worked around a bug in TCL during shutdown after TCL_FinalizeThread() had
    been called and returned in each thread with a TCL interpreter, but when
    TCL_Finalize was called on the main thread it would crash because not all
    the threads had actually cleaned up completely.  From the manual I fail to
    see how ioFTPD did anything wrong, but I've added code to mitigate the
    problem.

60) Fixed a bug in 'site who' which would call Free() on NULL if no excluded
    paths were defined.

61) Fixed a bug in 'site refresh' which wouldn't free all paths when dealing
    with merged directories.

62) Added a workaround for WinSock returning an odd error that resulted in:
      426 Connection closed: The semaphore timeout period has expired.
    and changed it to more accurately reflect what was going on:
      426 Connection closed: Data transfer timeout.

l.d.m · 04-27-2011, 03:23 AM

wow, a lot of changes and fixes bugs!
thanks for your hard work.. I try this release

Flow · 04-27-2011, 03:45 AM

Thanks Yil!

mr.babek · 04-27-2011, 04:39 AM

Much appreciated! gogogo!

Yil · 04-27-2011, 04:35 PM

Hmm, looks like I fixed the per-user download counter, but broke the per-user upload counter! LOL. Let me know what else I got wrong.

Yil · 04-28-2011, 02:25 AM

*SMACKS HEAD* I've been debugging a few crash dumps, and I've been up a long while, so it might not be true, but it looks like I completely missed initializing part of the threading system in the OpenSSL library. I mean, I compiled it with thread support, but evidently you need to call some function to register a bunch of locks for it to use at runtime, sort of like you need to register malloc/free.

This would only be an issue with v7.4+ obviously since that's when we started using OpenSSL, but it's clearly a really big deal and this is probably causing all the crash dumps I've seen recently.

I'll get a new release out tomorrow.

Flow · 04-28-2011, 03:05 AM

Can i smack too?

mr.babek · 04-28-2011, 03:34 AM

well, up till now its amazingly stable for me, better than previous versions

Yil · 04-28-2011, 06:50 PM

Code:

v7.7.1 Release Notes:

1) Files in \System:
   Changed : ioFTPD.[exe,pdb] - Version 7.7.1.0


*** Bug Fixes

2) A HUGE BUG fixed in how ioFTPD uses the OpenSSL library.  It appears that
   even though I compiled the OpenSSL libraries with thread support they
   provide no default locking implementation!  What kind of idiots are they?
   Oh sure, they provide a default implementation for getting the thread ID
   with all sorts of tests in the source to figure out how to do it on each
   platform, but then they ignore locking on a -DTHREAD compiled library?
   I mean, it isn't like they already distribute the code for the locks in
   the fricken source dir with all the platform tests to get it right, but
   they don't bother to enable it?!?  Did I mention they are idiots?  Heck,
   they even claim that the library is thread safe lots of places but fail
   to mention that isn't really true unless you setup some callbacks!  Anyway,
   ioFTPD internally protects each SSL object so it's in use by just one
   thread at a time, but shared state in the SSL context and cache wasn't being
   protected because it had no locking so sooner or later things would break
   if the server had lots of SSL connections starting/ending at the same time.

3) Fixed a bug with the per-user upload count for users.

4) Fixed a bug where 'site myinfo' wouldn't return an error if too many args
   passed to it, and in some cases wouldn't display anything.

Yil · 04-29-2011, 11:21 PM

Now that I've released the matchlist feature I'm going back to play with the new FlashFXP command set that I started a while ago. If you've written custom Flash command menus please let me get a copy so I can see other examples in case the way you've organized them makes more sense than how I did it.

One thing I've noticed is that the current help files with ioFTPD aren't quite right. I updated the current help file to have the new/changed commands from the v8.0 helpfile, but they take advantage of a feature or two that only exist in the new v8 help module. It only affects a couple of commands and isn't a big deal, but I expect I'll be updating them shortly with the Flash stuff just in case I find more things I want to change... Besides a cosmetic difference or two the "unworking" part is limit to normal users, SiteOps+ won't notice anything different/wrong.

m4r3z · 05-01-2011, 02:57 PM

Updated and running.
Lets see how it does for the next week

Flow · 05-02-2011, 01:09 PM

Hmm ... 550 msg display correctly?. File is transfered, file is thare, but 550 report No such file or directory.

SIZE 213-me_naked.mp3
550 213-me_naked.mp3: No such file or directory.

Yil · 05-02-2011, 04:05 PM

File uploaded to a regular dir and then SIZE doesn't find it? This the same login as uploaded the file? I wouldn't think that should be possible unless the file wasn't uploaded but extracted like a .nfo/.diz or something. On the other hand, if it was uploaded by a different user you'll have to pay attention to the timestamps carefully... Any chance it was it in the process of being uploaded in another login at the time?

On the other hand, it's possible I missed marking the directory as dirty somewhere with all the changes in v7.6 but I'd need more details, you can PM me logs if you think it shows the problem clearly. BTW, anyone can force ioFTPD to re-cache a directory using 'site refresh' to see if that makes the problem go away.

Does this happen for more than 1 file in a directory?

ArtX · 05-03-2011, 05:38 AM

Quote:

Originally Posted by Flow

Hmm ... 550 msg display correctly?. File is transfered, file is thare, but 550 report No such file or directory.

SIZE 213-me_naked.mp3
550 213-me_naked.mp3: No such file or directory.

I have got that message for a while, files are always there though, so i never really paid much notice to it

[R] SIZE a.little.test.file.rar
[R] 550 a.little.test.file.rar: No such file or directory.
[L] PRET RETR a.little.test.file.rar

yet in the directory D:\FTP\Test there is a file called - a.little.test.file.rar