ioFTPD v6.0 is HERE !

Yil · 03-31-2007, 11:30 PM

Here it is folks... I'm releasing ioFTPD v6.0

Latest Version:
ioFTPD-v6.0.4.zip

Highlights include FXP over SSL connections using SSCN protocol, recursive directory listings, moves files or directories seamlessly across physical disks, recursive chmod/chown with simple file globbing, plus several all new site commands! Also included are new configuration options as well as numerous bug fixes including getting tcl/php scripts and shared memory working again.

Post any bugs or weird behavior here (preferably with a test case or exact description of how to repeat the problem) and I'll see what I can do...

-Yil

Who knew you could only write a 10k message? Detailed changelog follows...

Yil · 03-31-2007, 11:32 PM

Code:

v6.0 Release Notes:

1) Implemented client SSL behavior in ioFTPD so it can initiate SSL
connections.

2) Implemented the FEAT command. This is used to report what additional
functionality (beyond RFC 959) that the FTP server supports.

3) Implemented the SSCN protocol to enable easy FXP SSL.

4) Changed the way LIST -l displays date/time in directory listings by default.
A file last modified 12:34PM on June 10, 2005 previously was displayed
incorrectly as
Jun 10 12:34
It now appears as it should (unix ls style)
Jun 10 2005

5) Implemented the LIST -T option (Raiden / Serv-U extension) so that
directory listings now include the full date and time. The example above
would now be:
Jun 10 12:34:56 2005

6) The new directory listing code now returns the directory in sorted order.
In the case of merged directories or submounts these are properly handled
so that size/modification/permission attributes are summed and/or computed
correctly.

7) Implemented the LIST -R option for recursive directory listings!

8) New ioFTPD.ini option (Allowed_Recursive). Set this permission flag to
enable the -R option to LIST. Users asking for a larger number of
directories to be listed can be time consuming and resource intensive.
For larger servers you may wish to enable this only for admins and/or a
script to generate a listing every few hours and drop it into the root
directory for people to download...

9) Changed the way file globbing works for directory listings. Previously
you could do something like LIST */* and it would return all the files
in the immediate subdirectories. Which sounds cool and is similiar to
what unix would do, except in this case since it doesn't include the
path you won't know which directory a file is from rendering it useless.
There were also problems with []'s in a name specified to LIST as it
would try to use it as a filename matching pattern. Thus
"LIST -al /test [old]/foo" wouldn't match a directory of that exact name.
I've, therefor, limited globbing to only the LAST component of the LIST
command so LIST *.txt, LIST xyz/*.txt, or even LIST -R *.txt (search all
subdirectories for files ending in .txt) will work and no ambiguity exists.
As a special case if an exact match for a filename including []'s exists
then no globbing is done at all and just the file/directory is returned.

10) New ioFTPD.ini option (Idle_Ignore) to support a configurable idle timer.
Instead of any successful command resetting the idle time you can choose
to ignore some. For example:
Idle_Ignore = NOOP CWD PWD LIST NLST STAT PASV
In this case if the user hasn't actually done something besides wander
around the filesystem looking at directories he/she is disconnected.
The idle time reported for the client is unaffected by this change, thus
ioGUI, SiteWho, etc will display the time the last user entered any
command even if it's ignored, however the time the user has before being
disconnected does not reset.

11) Implemented the MDTM command to retrieve and SET file modification times,
FTP clients noticing this extension can now make uploaded files show the
filetime of actual file and not the time it was uploaded.

12) Reverted the ONLINEDATA structure to keep compatibility with existing
tools/scripts such as ioGUI2 and SiteWho. Long live ioGUI!

13) Documented a number of things in the accompanying ioFTPD.ini and
Default.vfs file to help people understand some options.

14) &Service_Update is really called &ConfigUpdate! Go update your
ioFTPD.ini file... This should fix the Host=xyz.com issue since it
wasn't updating your IP. Useful for people behing a NAT router with
a dynamic external IP.

15) The Encryption_Protocol= line in ioFTPD.ini means something different
than indicated. It limits the encryption to a particular type. Thus
setting it to SSL2, SSl3, or the undocumented TLS setting would only
allow the indicated protocol. Not defining it implies all three are
avaliable with TLS->SSL3->SSl2 being the order tried.

16) Microsoft Sstream encryption is used for SSL/TLS and this has a few
issues. Windows XP / 2003 don't support AES and thus the cipher strength
is currently limited to 128 bit RC4/MD5 instead of 256 bit AES/SHA. This
isn't terrible but I believe it's slower and less secure, but the only
solution is to rewrite everything using openSSL or switch to Vista which
does support AES/SHA. The one side effect of this right now is don't try
to set Min_Cipher_Strength > 128 as it won't work unless you're on Vista
and even if you are if the other guy isn't using openSSL on XP he won't
be able to connect...

17) Fixed the shutdown logic. It turns out ioFTPD never actually shutdown
the active services and gracefully exited. I didn't notice until I
fixed it so it does shutdown services and then started getting errors.
This is the reason when you issue a site shutdown active users are never
shown as logging out in the logfile. I changed it so it closes services
which in turn closes all the connections and thus everyone is logged
out. If threads don't exit inside 10 seconds which might happen if a
TCL script or scheduler event or something takes a long time it just
continues on and exits which might still result in a crash, but it
would have done so in the old version anyway...

18) Changed the way user login works.
A) Corrected error messages that 5.8.6+ broke.
B) Require_Encrypted_Auth now controls the default SSL login behavior.
Previously if the user/group/flags matched then they would get an error
indicating an SSL connection was required immediately after entering
their username. HOWEVER, no such error was generated for invalid
usernames and thus it leaked user account information nearly as
bad as the previous problem of immediately denying access for invalid
usernames. I changed the permission code to accept fake tests so I can
now look for * in Require_Encrypted_Auth when I know the user is invalid
and return the SSL required error message anyway. Since the USER command
now looks the same in both cases it gives away nothing. Thus
Require_Encrypted_Auth = !-ioFTPD *
will not require the ioFTPD user to use SSL, but everybody else will
AND invalid usernames won't give anything away. Using !* or just not
specifying * will revert to current (I believe wrong) behavior.
C) Added a new text message file for logins requiring SSL. "SecureRequired"
in the text/ftp dir can be used to inform users how to turn on SSL.
The one I provided describes how to turn TLS/SSL on in FlashFXP for
those clueless individuals needing help...

Detailed login info:

5.8.5 Behavior
Invalid user [normal or requires SSL] (wrong - leaks usernames):
USER xyz
530 Login incorrect.

SSL required for valid user (right):
USER xyz
530 Your user class requires you to use secure connections.

User limit reached (right):
USER xyz
331 Password required for xyz.
PASS (hidden)
530 Login failed: Maximum concurrent connections for account reached, try again later.

5.8.6+ Behavior
Invalid user [normal or requires SSL] (right):
USER xyz
331 Password required for xyz.
PASS (hidden)
530 Login failed: Invalid password.

Valid user but SSL required (wrong - no info on why you can't login!):
USER xyz
331 Password required for xyz.
PASS (hidden)
530 Login failed: Invalid password.

User limit reached (wrong - no info on why you can't login!):
USER xyz
331 Password required for xyz.
PASS (hidden)
530 Login failed: Invalid password.
* The user limit reached breakage in 5.8.6+ actually affected a few more
cases which are all fixed, but it was the one I noticed first...

6.0 Behavior:
Invalid user with SSL NOT enforced by default (right):
USER xyz
331 Password required for xyz.
PASS (hidden)
530 Login failed: Invalid password.

Invalid user with SSL enforced by default (right):
USER xyz
530 Login failed: Your user class requires you to use secure connections.

Valid user but SSL required (right):
USER xyz
530 Login failed: Your user class requires you to use secure connections.

User limit reach (right):
USER xyz
331 Password required for xyz.
PASS (hidden)
530 Login failed: Maximum concurrent connections for account reached, try again later.

19) Upgraded tcl to 8.4.14 and included all the default .tcl files shipped
with tcl. In particular the HTTP package may be of some use...

20) Upgraded php to 4.4.5. I can upgrade to the latest v5 release but since
I don't know about compatibility with existing scripts it might be best
not to.

Yil · 03-31-2007, 11:33 PM

Code:

21) The executable file no longer needs to be named ioFTPD.exe which was
    required for the non-registered versions.  It has the original
    application icon again and now includes detailed build information.

22) The rename/move command on files or directories always works now.
    Doesn't matter if the move crosses physical drives.  To move directories
    you must have permission for every directory in the directory tree
    although access to every file is not required.  Permission/ownership
    for the moved directory is the same as the original.

23) New ioFTPD.ini option (DirectoryCache_Buckets).  You can now choose
    the number of buckets to use for the directory cache.  Previously you
    had 8 buckets and could set the max number of directories in each.
    Internally for each it kept a linked list of buckets in the order they
    were added and when space was needed it popped the oldest CREATED.  It
    also kept an array of pointers sorted by name for each bucket and would
    binary searched this array to locate entries.  The new defaults of 100
    buckets of 1000 depth instead of 8 and 1000 will obviously use more
    space if you have a very large server.  If memory is an issue for you,
    go back to 8/1000 or try 80/100, but I believe overall performance will
    be increased with the larger cache size.

24) The loopback address (127.0.0.1) is now immune from banning because of
    too many reconnections.  Hehe, I managed to ban myself during testing :)

25) The chown -R option was broken when dealing with merged directories or
    mounted subdirectories.  I rewrote the whole traversal logic to be
    generic and support multiple commands.

26) Added the -R option to chmod so you can recursively change permissions.

27) chown/chmod now accept a simplified globber ([]'s don't act as globs)
    when specifying files/directories to change.  Thus you can now use:
      chown -R newuser *.txt
    to change the owner of every file ending in *.txt to "newuser" or
    if you leave off the -R just the *.txt files in the current directory.
    "*" matches all directories and files, but as a special case "*/" will
    only match directories.  Thus you can do something like:
      chmod -R 644 *
      chmod -R 755 */
    The first changes everything to 644 and the 2nd changes only directories
    to 755.  Way simpler than changing everything one by one.

28) The main logfile now has 3 new events:
      START "pid=123" "<cmdline used to start ioFTPD>"
      STOP "pid=123"
      SHUTDOWN "<user-who-issued-command>"

29) New "site free" command which shows you the free disk space under the
    current directory.  I know there is a cookie which could show this but
    nice to have anyway.  In the future might show combined free space of
    merged directories across multiple drives or free space of all drives
    on system with associated mountpoints for admins/vfs admins.

30) New "site space" command which when used on a directory will show you
    the total number of files/subdirectories in the entire directory tree
    as well as the total disk space used.

31) Access checks have been changed to disallow viewing or access to NTFS
    hidden files/folders and important system files.  The one exception
    being drive letters which are actually flagged hidden & system.  As
    I documented in the default.vfs file it's still probably a bad idea
    to mount drive letters directly but this should prevent a number of
    potential security issues.

32) New ioFTPD.ini option (Hide_Xfer_Host).  You can now choose to hide
    the hostname/ip of transfers in the xferlog.

Flow · 04-01-2007, 01:08 AM

You Rock Yil!

Thanks again for your input on this projekt. You should be REWARDED !

EwarWoo · 04-01-2007, 01:17 AM

Man, thats a lotta changes.
Damn nice to see another release, fantastic work dude.
I'll have a full read and give it a try later

Once again, thanks for the great work mate

ArtX · 04-01-2007, 02:11 AM

just incase the link above starts getting overloaded or your having issues with your isp, you can try getting it from here

peep · 04-01-2007, 03:47 AM

Great stuff. U the man Yli, came out of nowhere and blasted out a new io release and which such an amazing changelog. Let the bugtesting begin, man it's been a long time since I've had the opportunity to bughunt a new io version.. getting nostalgic

Flow · 04-01-2007, 05:20 AM

Is it me or is it something elese ? Can anyone test ioGui and see if userlist and grouplist and activities windows shows twice of everything.

Thanks

Zer0Racer · 04-01-2007, 05:46 AM

Wonderful! I'll start testing it right away.

Yil, let me know if you need help with documenting stuff or just send me a couple of lines with suggestions and info. Then I can start updating/re-writing the general documentation (used on Inicom website today) that I wrote for 4.9.0 back in May 2003 and later updated for 5.8.x.

/ZR

Edit: I see you're doing a great job documenting yourself Anyway, let me know if there is anything I can do.

Zer0Racer · 04-01-2007, 07:23 AM

ioFTPD.ini

Are Device_Concurrency and Upload_PreAllocation back and working? If I remember correctly think they where taken out a few versions ago, before 5.8.4 or something.

Please clarify changes/rename made to Transfer_Buffer. Before there were [Network] Internal_Transfer_Buffer and [Ftp] Transfer_Buffer... I think. Or are they the same function? If they are/were different, how does/did it work compared to now?

Default VFS

For the default vfs I suggest putting an empty dir inside FTP-ROOT-DIR called "Incoming" (nice for the noobs hehe) since that's the example given in ioFTPD for special vfs permissions. Maybe even a .ioFTPD file (dir chmodded to 777). Also changing "C:\ioFTPD\FTP-ROOT-DIR" to "..\FTP-ROOT-DIR". Both things would make ioFTPD work a little easier "out-of-the-box" when for example putting it on another drive than C:\.

EDIT: That would of course mean an empty root dir inside the ioftpd dir structure. I suggest moving bin inside /ioFTPD and ioGui inside /ioFTPD/scripts (or at least in /ioFTPD for noobs). Not a good idea imho to use path C:\ioFTPD\ioFTPD or to put ioFTPD, bin, ioGui, FTP-ROOT-DIR and readme + lnk in root of any drive - makes it hard to keep track.

I even have a habit of putting my vfs-files in /vfs instead of /etc because vfs-files are (manaully) modified more frequently than the files in /etc. A seperate dir makes it easier to handle when you end up with alot more vfs-files, for different groups and/or users, and minimizes the risk for unintentional modifications/deletes of those files in /etc.

Suggestion (I should have posted this waaay earlier)

Please make a function for unsetting the value of the vfsfile string in user and group files. Suggestions (to follow current command structure):
site change <user> vfsfileunset
site change <group> groupvfsfileunset

The problem is if you have set a specific vfsfile/groupvfsfile for a user or group and just want to revert to default vfs, you can't. You would either have to manually edit the user or group file (usually a big no-no) or actually setting the vfsfile to ie. ..\etc\default.vfs which would make it harder to administrate if you at any point want to change the name or location of that default.vfs (you'd have to change those user/group files again).

Maybe there are other strings/options that need the ability to be unset? Like admingroups? Not sure how that work atm though.

Just my two cents for now

/ZR

Zer0Racer · 04-01-2007, 08:46 AM

Version: 6.0.0

With default config and no scripts, bigger vfs, ssl cert enabled I can crash my ioftpd when issuing LIST -R (or LIST -lR) command either as a raw command in ffxp or with custom list method so that ffxp uses LIST -R by default. LIST -alR and LIST -aR seems to work though.

Crash log:

Code:

Unhandled exception 3221225477 at address 0x00408a74 (0x00000000)
0x00000000 0x00000000

Code:

[15:37:12] [L] PROT P
[15:37:12] [L] 200 Protection set to: Private.
[15:37:12] [L] PASV
[15:37:12] [L] 227 Entering Passive Mode (127,0,0,1,21,67)
[15:37:12] [L] Opening data connection IP: 127.0.0.1 PORT: 5443
[15:37:12] [L] LIST -R
[15:37:12] [L] Connected. Negotiating TLSv1 session..
[15:37:20] [L] Connection lost: 127.0.0.1

The same happens without TLS/SSL.

Can anyone confirm this?

EDIT: Workaround: set Allowed_Recursive = !* for now

/ZR

peep · 04-01-2007, 10:04 AM

Same here

Code:

Unhandled exception 3221225477 at address 0x879111e0 (0x00000000)
0x00000000 0x879111e0 
Unhandled exception 3221225477 at address 0x00408a74 (0x00000000)
0x00000000 0x00000000

Code:

[R] PORT 192,168,1,2,15,59
[R] 200 PORT command successful.
[R] LIST -R
[R] Connected. Negotiating SSL session..
[R] Connection lost: ...

SSL, running as service, default config and scripts added (ioA and ioB)

Zer0Racer · 04-01-2007, 11:08 AM

Version: 6.0.0 (and earlier)

Event OnLogIn only seems to be triggered when named OnFtpLogIn in ioFTPD.ini. (old "bug")

/ZR

peep · 04-01-2007, 11:23 AM

This is not critical as it's a new feature, but apparently broken, or am I just using it in a wrong way? haven't seen any docs on the usage of the command anywhere.

Code:

[R] site space test_folder
[R] 500 SITE space: Command not understood.
[R] site space
[R] 500 SITE space: Command not understood.

tuff · 04-01-2007, 11:34 AM

admin.vfs seems to be missing from the package, is this intentional?

can we have a idle exempt added for the next version? ie.

Idle_Exempt = -tuff =SITEOP !*