ioFTPD v7.2.1 Released (BETA)

Yil · 03-12-2010, 10:23 AM

Highlights:
HELP, yea, I know... finally

More useful virtual directories, you can upload, etc to pure virtual dirs
SITE PERMS command for details on allowed user actions.
SITE SECTIONNUMS - describe sections and paths for each

Latest Version:
Link: ioFTPD-v7.2.1.zip

Source:
Link: ioFTPD-v7.2.0-src.zip

Yil · 03-12-2010, 10:24 AM

Code:

v7.2.0 Release Notes:

1) Files in \System:
   Changed : ioFTPD.[exe,pdb] - Version 7.2.0.0.
   Added   : Theme.ini, Help.ini, Help-SiteCmds.ini
   Optional: Help-nxTools.ini, Help-ioNiNJA.ini
   Changed : ioFTPD.ini - summary of changes by section...
     [Any]          : Feature_Suppression - Comments changed
     [VFS]          : TimeStamp, TimeStampOwn added
     [Virtual_Dirs] : Comment changed (New feature)
     [FTP_SITE_Permissions] : knock      =  !A *
                              perms      =  !A * (or GV1M if you want)
                              sectionnum =  !A *
                              uptime     =  !A *
     [Change_Permissions]   : DnSpeed    =    1M
                              NonLeech   (Removed)
                              UpSpeed    =    1M
     [Ftp]                  : Enable_TimeStamp_On_Last_Upload - added
     [Help]         : Comments and settings changed.
     [Themes]       : * Entire section completely removed *

2) Files in \etc:
   Changed : ioftpd.env

3) Files in \text\ftp:
   Changed : Color, MyInfo.Section, Services.Body, UserInfo.Section, Welcome
   Changed : GroupList.[Body, Footer]

4) Files in \doc:
   Added   : Helpfiles.txt, FTP-cmds.txt, Site-cmds.txt
   Optional: nxTools.txt, ioNiNJA.txt
   Changed : Cookies.txt, iTCL.txt

5) Files in \source:
   Added   : FTP-cmds.help, ioNiNJA.help, nxTools.help, Site-cmds.help
   Added   : INCLUDE-DIR-README.txt
   Deleted : \Include [dir]

6) Files in \scripts:
   Added   : FormatHelp.itcl


*** Incompatible changes:

7) Removed "SITE CHANGE <who> NON-LEECH" as you can do the same thing with
   "SITE CHANGE :=0 RATIO" and it's far more versatile.


*** New Features

8) New site command (site perms [ -<user> ] [ <item> ]).  Displays
   permission information for you, or another user, about the indicated
   item with the default being the current working directory if no argument
   supplied.  If processing a directory then also display the default rules
   for items in the directory.  This is actually a REALLY nice command!
   It shows why you can or can't upload, delete, rename, etc.  Try running
   it from '/' so you can see all the rules if you vary action permissions
   by path.  See 'SITE HELP PERMS' for more details, but make sure you do
   it from a VM flagged account as both the help information displayed and
   the command results vary by the flags of the calling user.

9) New site command (site sectionnums).  For each section number defined
   in the configuration file show its name and if it shares credits and/or
   ratio settings with another section.  Then display a list of paths and
   the section number to use for them.

10) New site command (site uptime).  Simply displays the server and FTP
    uptimes.

11) New site command (site knock).  If port knocking is enabled display the
    current port connection sequence to trigger a "knock" else let the user
    know the feature is disabled.

12) New list argument (LIST -U).  Replaces the group field with the average
    upload speed in KB/s or "-" if that information is unavailable as for
    all existing files prior to v7.2 and all non-file objects such as
    directories and symbolic links.

13) New site change command (SITE CHANGE <user-glob> DNSPEED <speed>). This
    sets the maximum download speed for matching users to <speed> much like
    the SPEEDLIMIT <speed> change command would.  It is provided for
    symmetry with UPSPEED and because it's easier to remember.

14) New site change command (SITE CHANGE <user-glob> UPSPEED <speed>). This
    sets the maximum upload speed for matching users to <speed>.  This is
    useful when modifying more than one user at a time because you don't
    need to also set the DNSPEED limit like you would if you used the
    CHANGE SPEEDLIMIT command.

15) New ioFTPD.ini options (TimeStamp and TimeStampOwn actions under [VFS]).
    You can now control who can modify timestamps of files on the server by
    path with these 2 new actions.  Previously if the file was writable by
    the user they could set the timestamp provided the MDTM command was
    accessible to them.  Since MTDM also allows you to read the timestamp
    disabling it was not a good option and it didn't allow for doing so
    based upon the path.

16) New ioFTPD.ini option (Enable_TimeStamp_On_Last_Upload under [Ftp]). The
    new TimeStamp and TimeStampOwn actions cover most cases, however you may
    wish to allow users to modify the timestamp of the last file uploaded
    even if those checks fail.  The main reason this is useful is when
    zipscripts remove +w access from an uploaded file that was verified and/or
    modify the owner of the file.  In both those cases the normal checks would
    fail.  It's also somewhat useful if the zipscript doesn't modify anything
    but you don't wish the user to change the timestamp later on, but don't
    mind them doing it right after it is uploaded.  There isn't any other way
    to express that.  The default is FALSE which disallows the exception.

17) Modified site command (SITE CHANGE <group> GROUPDESCRIPTION <what>). If
    <what> is "NONE" then the group will have an empty description.  Since
    all SITE CHANGE commands require an argument it previously wasn't
    possible to completely remove the description once set.

18) Modified site command (SITE GROUPS [ -ALL ]).  The site groups command
    has been changed to include the idea of "hidden" groups which should not
    be displayed in the listing by default.  This supports the practice of
    changing the group ownership of files and directories to convey
    information instead of ownership.  An example of this would be ioNiNJA's
    MP3 genre labeling via group owner.  A group is considered hidden if it
    has no users and a description of "-".  Use the -all argument to see
    hidden groups in the listing as well.

    This command is also changed for GroupAdmins.  They will only see GID #1
    ('NoGroup') and groups which they can admin.

19) The SITE WHO and SITE STAT (with LIMIT option) will now only accept
    user, username*?, and =group style arguments for non SiteOp/Masters.
    Even the =group option is limited to groups the caller is a member of.
    This is to prevent leaking group, ratio, and flag information via the
    powerful globbing mechanism.  Jury out on if GroupAdmins should be
    exempt here as well.

20) New user-glob feature (=.<group>).  The '.' means matches users whose
    primary group is <group>.

21) New user-glob feature (:<group> and :.<group>).  These match users who
    have group as an adminGroup or as their primary adminGroup respectively.

22) New user-glob feature ([] and [^]).  You can now use character group or
    range matching in addition to '*' and '?'.  The ^ prefix means to match
    any single character not specified.

23) Modified the behavior of 'SITE MYINFO' to no longer show section name
    and information if all paths in the section are hidden to the user.

24) New .ini feature for [Virtual_Dirs].  You can now make hidden virtual
    directories to some users.   The Virtual dir definition now looks like:
      <path> = [ "<perm>" ] TCL <script>
    The new optional <perm> argument is a flag-style permission string that
    acts like the hidden/private directory feature and controls who can see
    the dir.

25) The resolver flag to allow loading information from virtual directories
    has been added to a number of commands.  This should allow directories
    and files to be looked up from the last cached directory, and if that
    fails then it will call the script expecting a ||RESOLVED|| reply.  The
    ability to operate on completely "virtual" items that don't exist in the
    filesystem is being looked into for a future release.  

    The basic functions of downloading (RETR), uploading (STOR, APPE),
    renaming (RNFR/RNTO), deleting (DELE or RMD), and creating (MKD) files
    and directories will all now attempt to resolve via the virtual directory
    script.  In addition user specified home directories (other than "/")
    can be virtual and MDTM, SIZE, and XCRC should work.  The following
    site commands were also modified: chattr, chmod, chown, size, free, perms,
    and symlink.  However any feature that requires more than a single
    file, directory, or symlink being resolved will fail.  Thus the wildcard
    and -R features of chmod and chown will not function and the size command
    will not correctly handle merged/raided directories.

26) Modified ioFTPD.ini option (Feature_Suppression under Devices).  You
    should now use "LIST -" and "STAT -" instead of the full string since the
    actual arguments displayed after the "-" depends upon the user's flags.

27) New message cookie (%[$cwd]). Returns the user's current symbolic path
    or the users fully resolved path if symbolic paths aren't being kept.
    Also added this to ioftpd.env so it's exported to child processes run
    via the EXEC command.

28) New message cookie (%[$UpTime]). Returns the number of seconds since the
    FTP server started.  You can then feed this into the $[Duration] cookie
    for a human readable time.  New default text/ftp/Welcome uses this.

29) New message cookie (%[$SysUpTime]). Returns the number of seconds the
    computer has been running (doesn't count hibernation/sleep time).  You
    can use $[Duration] to get a human readable time.

30) Modified super cookie (%[Duration]).  The cookie displays elapsed time
    between now and a time in the past, or 2 arbitrary times, in human
    readable form (y, w, d, h, m, s).   It now takes 1 fixed and 8 optional
    arguments to control how it's output should look.
     *Arg1: Time1 (0 implies Time2 is duration)
      Arg2: Time2 (0 implies now)
      Arg3: Extra text to append
      Arg4: Precision, last field to show, default s
      Arg5: Precision, first field to show, default y
      Arg6: Style: 0="y", 1=" year", 2=" years" if >1, default 0
      Arg7: Field separator, default " "
      Arg8: 0 = hide zero fields after the first non-zero field displayed,    
            1 = show, default 0
      Arg9: min numeric field width, and adds an extra space if style 2
            for non-multiple entries, default 0


*** iTCL Changes:

31) The TCL virtual directory event now has 2 additional arguments and looks
    like: "<path>" "<glob>" "<old-glob>" <exists> <cmd>
    <exists> : Either 0 or 1 depending upon whether the resolver requires
               that the item exist in the filesystem.
    <cmd>    : The name of the FTP command that caused the resolver to be
               called or "-" if unknown.  Useful for deciding what to return.
    Please see the v7.0 release below for a description of the other args.

32) New iTCL command ([ioUptime]).  Returns the uptime for the OS and the
    FTP in seconds.

33) New iTCL command ([ioTransferStats]).  This command returns the size and
    duration (in milliseconds) of the last file transfer.

34) Modified iTCL command ([ioVirtual AddDir|AddFile]).  The <Link> argument
    is now optional to AddDir and AddFile.  There is now no difference between
    the long form of AddSubDir and AddDir when no <link> target is specified.

    AddDir    <Size> <ModTime> <AltTime> <User> <Group> <Mode> <Name> [<Link>]
    AddFile   <Size> <ModTime> <AltTime> <User> <Group> <Mode> <Name> [<Link>]

35) Modified TCL command ([resolve list <path>]).  The 'uptime' field has
    been added to the list of information returned about each entry.  Each
    element of the list now looks like:

    Each element of the list looks like:
      { name type uid user gid group size mode attributes
        win-last-time unix-last-time win-alt-time unix-alt-time
        subdir-count {realpath link ...}
        {chattr-0 chattr-1 chattr-2 chattr-3} uptime }
    'type' field is: 'd' (dir), 'f' (file), or 'l' (ioFTPD link)
    'win*' times are the actual FILETIMEs.
    'unix*' times are in time_t or unix style times (seconds)
    'realpath' and 'link' come in pairs where 'realpath' is the full path
       to the real directory and link {} if a regular directory or in the case
       of an NTFS junction the fully resolved target directory path.  This
       sublist will be {} for virtual dirs or mountpoints.
    The first 4 "reserved" chattr fields are returned in order as well:
       Private, ioFTPD Symlink, <reserved>, <reserved>.
    'uptime' is number of milliseconds file took to be uploaded.

36) Fixed a bug in [resolve list] where the 4 [chattr] settings returned
    were not actually the first 4 settings as intended.


*** Bug Fixes:

37) Fixed a bug with implicit SSL connections (Explicit_Encryption = False)
    that was causing the server to ignore all client commands after the SSL
    negotiation.

38) Fixed a bug where just the first 12 characters of the user and group
    were being printed in long directory listings.  This is obviously
    important when dealing with client macros that operate on the owner
    or group of a file or directory and extract this info from the directory
    listing.  This also allows for correctly displaying long fields via some
    list '-' options such as the private directory setting in the group field.

39) Fixed a bug when multiple custom commands in [FTP_Custom_Commands] were
    defined under the same name.  Previously it would print the response code
    after the each entry which would mess up FTP client response parsing.  It
    now stops and returns an error after the first failed entry, and prints
    success only when all entries returned successfully.

40) Fixed a bug with the tcl function [resolve] when using the 'target',
    'vfs', or 'symbolic' options that resulted in errors being generated to
    'Error.log' if you had disabled keeping symbolic links in the path via
    the .ini file and didn't specify a <cwd> argument when called.  In this
    situation the user's current symbolic path wasn't being updated by the
    server because it would always be equal to the non-symbolic path but the
    resolver was explicitly told it should operate on the symbolic path and
    thus would print an error before going ahead and performing the operation
    on the non-symbolic path.  Thus it would always function correctly, but
    since this is an allowable scenario no error should be generated.  The fix
    was not uniform however.  The 'target' and 'vfs' options no longer pass
    the flag to operate on the symbolic path to the resolver, while the
    'symbolic' option makes sure the symbolic path is set to the non-symbolic
    path (which is always true since the option for this was disabled in the
    .ini file) before calling the resolver to update it further.

41) Fixed a bug where the the 'G' flag for group admins was testing the
    target's account instead of the user's account before checking for
    groupadmin rights.

42) Fixed bugs in RNFR/RNTO, DELE, MKD, RMD, STOR, 'site symlink', and 'site
    chmod' where the parent directory to test for +w was computed from the
    real path instead of the virtual path which can differ when dealing with
    merged/raided directories.

43) Fixed a bug where resumed transfers were not being counted as uploads
    when testing user limits.

44) Fixed a bug where the MDTM would complain if the first of more than one
    argument doesn't look like a date, it now just assumes the whole line is
    a filename if it doesn't look like a date.  This handles spaces in
    filenames much better when client's don't enclose the filename to modify
    inside double quotes.

45) Fixed a bug where purged users where not increasing the group slot
    counter for the primary group of the deleted user.

46) Site change credits no longer defaults to the creditsection of the
    current path and instead defaults to section 0 all the time.


*** Internal non-visible changes:

47) Section information is now kept in a parsed form to speed lookups.

48) Exported function FindFirstUser changed.

Yil · 03-12-2010, 10:34 AM

Theme support moved out of ioFTPD.ini file and into a separate file. Also added "bright" versions of the existing 4 themes for use with newer FlashFXP v4 betas. In the future might go back and use the background color and underline features for even better displays...

Forgot this changelog entry:

XX) The raw color cookie %[C()] has changed. It no longer accepts single digit colors, but rather the new syntax documented in Theme.ini. I doubt anyone uses this as themes via the %[T()] cookie is the way to go though.

XX) If you created or modified a theme you'll have to convert to the new format which is easy to do and far easier to read now.

Yil · 03-12-2010, 11:14 AM

Oh... two more tidbits... The doc\Helpfiles.txt file describes the new helpfile format but I forgot to include a reference to read that in the Changelog for info on how things work... Like no mention of "(*BAD*)" to highlight commands referenced but not defined for Master users who view site help.

You should also be aware that site help is caller specific. It actually tests access to commands before displaying them. It's pretty cool and means a plain old user sees just the stuff they can use and a Master gets a huge list!

Any comments, feedback, suggestions for content, formatting, etc welcome.

Yil · 03-12-2010, 01:07 PM

It appears that the revamped theme code is outputting color control info even when disabled... Give me a few mins to fix that with a new release...

Flow · 03-12-2010, 01:20 PM

lol, right, this is big. I think safest redo a new reconf with this pakage. Thanks for this new lovely version.

Flow · 03-12-2010, 01:43 PM

Its alive

and looking really good

Right, first input. Thare should be a cookie for 'sitename' for help file header/footer. (Maybe i missed reading somewhare........). Um, i mean %sitename-cookie instead of displaying ioFTPD.

Code:

.-[ioFTPD Help]-----------------------------------------------------------.

Code:

'----------------------------------------------------------------[ioFTPD]-'

So itÂ´ll be easier handling for future usage.

Yil · 03-12-2010, 02:23 PM

Code:

v7.2.1 Release Notes:

1) Files in \System:
   Changed : ioFTPD.[exe,pdb] - Version 7.2.1.0.


*** Bug Fixes:

2) Fixed a bug introduced with the new v7.2 theme processing where the ANSI
   reset escape sequence was being output in command responses even when no
   theme was active.

3) Fixed a typo in specifying an octal number used for permission checking
   that was causing some checks to produce incorrect results and resulted in
   "Permission denied (directory permission)" errors to show up even when
   the user had permission to access the file.

4) Fixed a bug where Keep_Link_In_Paths wasn't applying to links in virtual
   directories.

Yil · 03-12-2010, 03:20 PM

Flow: There isn't a good way to handle that right now. Lines are passed through the message cookie parser so you can do some cool stuff, but there isn't a formatting cookie so drawing the bounding box with an arbitrary length site name pulled from the .ini file at runtime wouldn't work. Needless to say a new formatting cookie is on my TODO list somewhere to allow you to do this in help and in things like SITE WHO, etc.

You can, however, edit the .help file in /sources and create a new HELP/SITE HELP .ini file (see doc/Helpfiles.txt), or just edit the .ini file by hand. It will look ugly with all the color control cookies in there but it's not that bad for such a simple change...

SRH99 · 03-13-2010, 03:08 AM

Truely amazing work Yil, cant wait to test this one out.

thank you.

l.d.m · 03-13-2010, 12:38 PM

very very thanks for your great work Yil
I was waiting for bugfix of implicit SSL

Silly · 03-13-2010, 11:02 PM

nice...finally here comes a new release...thanks yil

isteana · 03-14-2010, 04:31 AM

great.....but to making space features still not yet?
it does not exist any available auto wipe feature for the present..except for old buggy warchive
give some solution with next version

for example way below .........it could be in ioftpd.ini

[maximum_dir_for_section] # it will be delete oldest dir when exceeds the standard value

set section game = <value>
set section util = <value>

[maximum_allowed_space] # it will be delete oldest dir during get the standard minum space

set section appz = <value>
set section movie = <value>

huh?

newguy · 03-22-2010, 05:09 AM

Maybe a stupid question but how do i use virtual directories?
Can anyone give an example for me on how to implement it?

Yil · 03-22-2010, 03:38 PM

newguy: Check out the Changelog (in particular v7.0 and the new 7.2.0) for information on Virtual_Dirs. An early example can be found if you look at source/nxSearch.itcl which is a simple test script I wrote to return search results from the nxTools dupedb. The ioVirtual call to fill in the directory listing is documented in the doc/iTCL.txt file.

Here's a newer example I was using to test with that just mirrors a VFS dir on the server as a pure virtual dir. I know, virtual this virtual that... I wasn't planning on releasing this yet so it's not documented and it contains whatever I was playing with at the time I was testing, but it may help you. In particular note that you can get the files/dirs to display in the listing from anywhere including a database like in the nxSearch example, but I just used another dir so I could compare them visually to see if things were working correctly...

Code:

# NOTE: <>'s are invalid in real filenames so useful to return them for purely
# fake subdirectories because the server won't try to resolve them and thus
# we save a lot of overhead.
set mDir "test2"
set vDir "vDir"

proc AddDummy {name {link ""}} {
    set t [clock seconds]
    if {[string length $link] != 0} {
        ioVirtual AddDir 0 $t $t "User" "Group" 0777 $name $link
    } else {
        ioVirtual AddDir 0 $t $t "User" "Group" 0777 $name {}
    }
}


proc MirrorDir {vPath oPath} {
    foreach vEntry [resolve list $vPath] {
#        putlog "entry: $vEntry"
        lassign $vEntry name type uid user gid group size mode attributes \
                winLastTime unixLastTime winAltTime unixAltTime subDirCount \
                pathList chattrList uptime
        # it's important the the mode passed to ioVirtual only contain the 0777
        # bits set else it complains and rejects it.
        set mode [expr {$mode & 0777}]
        if {$type == "d"} {
            if {$name == "." || $name == ".."} continue
            ioVirtual AddDir $size $unixLastTime $unixAltTime $user $group $mode $name "[file join $vPath $name]"
            #ioVirtual AddLink [file join $vPath $name] $name
        } elseif {$type == "f"} {
            ioVirtual AddFile $size $unixLastTime $unixAltTime $user $group $mode $name ":[file join $vPath $name]"
        } elseif {$type == "l"} {
#        putlog "type $type - [lindex $chattrList 1] -- $mode"
            ioVirtual AddDir $size $unixLastTime $unixAltTime $user $group $mode $name [lindex $chattrList 1]
        } else {
            return -code error "unknown <type>: $type"
        }
    }
    return 1
}


proc ProcessArgs {} {
    global ioArgs mDir vDir

    putlog "mirror: $ioArgs"

    # need to strip off quotes from arguments
    set fileName    [string range [lindex $ioArgs 0] 1 end-1]
    set globPattern [string range [lindex $ioArgs 1] 1 end-1]
    set oldPattern  [string range [lindex $ioArgs 2] 1 end-1]
    set exists      [lindex $ioArgs 3]
    set cmd         [lindex $ioArgs 4]
    
    if {$globPattern != ""} {
        # this can either be a real glob like *.txt, or it can be
        # the name of a file
    }
    set pathList [file split $fileName]
    
    if {![string equal -nocase [lindex $pathList 1] $vDir]} {
        putlog "Not in virtual dir!"
        return 0
    }
    
    set target [file join / $mDir {*}[lrange $pathList 2 end]]
    
    set real [resolve PWD $target]
    
    putlog "$target = $real"
    
    if {$real == ""} {
        putlog "failed to resolve"
        return 0D
    }
    
    if {[file isdirectory $real]} {
        return [MirrorDir $target $fileName]
    }
    if {[file isfile $real]} {
        putlog "resolved: $target -> $real"
        AddDummy "||RESOLVED||" $target
        return 1
    }
    if {$exists != "" && $exists} {
        # doesn't exist... just ignore
        return 0
    }
    putlog "non-exist resolved: $target -> $real"
    AddDummy "||RESOLVED||" $target
    return 1
}

return [ProcessArgs]