PDA

View Full Version : ioFTPD v6.5.0 BETA released


Yil
05-24-2008, 04:01 AM
This first v6.5 release includes a LOT of changes and is therefore definitely a BETA release, and it won't work with nxShareDB until it can be updated. When upgrading make sure you read the WHOLE ChangeLog as there are some potential script issues. Make sure you copy over all the new or changed text\ftp files and re-apply any customizations you may have done previously. Also make sure to replace the lib\ dir with the new one. And update your ioFTPD.ini file. I've included some hints about what exactly changed and which sections of the file you need to check at the top of the ChangeLog.

Highlights:

1) New/changed site commands: shutdown, open, close, deluser, readd, purge, bans, who (regular user who - hides priviledged information, users, paths, etc), and swho (siteop who).

2) New userfile fields and message cookies to access: CreatorUid, CreatorName, CreatedOn, LogonCount, LogonLast, LogonHost, MaxUpload, MaxDownloads, LimitPerIP, ExpiresAt, DeletedOn, DeletedBy, DeletedMsg, Opaque

3) New events: OnClosedKick, OnClosedLogin, OnDeletedKick, OnDeletedLogin, OnExpiredKick, OnExpiredLogin, OnUnknownLogin,

4) Now supports 25 sections instead of 10.

5) Conditional logic super cookie and a lot of changes to the message files.

6) The telnet service is no longer available.

7) TCL upgrade to TCL v8.5.2 which includes a lot of changes to the /lib directory from TCL 8.4.X releases from the previous years. This may cause problems for some scripts. Let me know what breaks.

Latest Version:
ioFTPD-v6.5.0.zip (http://home.comcast.net/~yil/ioFTPD-v6.5.0.zip)

Yil
05-24-2008, 04:02 AM
v6.5.0 Release Notes:

*** INCOMPATIBILITY WARNING ***

This version of ioFTPD introduces a large number of changes to the UserFile
structure in memory and the associated UserFile on disk. I have kept the
existing shared memory interfaces byte compatible with previous releases, but
added new function identifiers (DC_NEW_USERFILE_*) so newer shared memory
applications can access the new fields in the UserFiles. This should mean all
existing 3rd party compiled .exe files will still work although they will be
limited to using 10 sections. ioGUI2 using shared memory, for instance,
appears to be working fine, however I still believe it is better to use the
.itcl interface of ioGUI2 (and this is the default) since you can connect
remotely that way and you get accurate idle times, etc.

WARNING: nxShareDB as a user module manipulates UserFiles directly and as such
doesn't stand a chance of working and will likely corrupt your user database.
Wait for a new version before upgrading to v6.5 if you use this!

TCL based scripts usually use the iTCL commands to open/close/lock/etc
UserFiles and the ascii2bin and bin2ascii functions to parse them. Since
these functions have been updated I don't believe TCL based scripts should
have any issues, but I've been wrong before so report any problems.


*** SCRIPT CONFLICTS ***

There are a number of new built-in site commands. You can choose to use the
new ioFTPD versions by removing the your reference to the 3rd party script
under FTP_Custom_Commands, or you can use the new "^" prefix to disable the
ioFTPD internal version and continue using the external script. You should
pick one or the other as running both is probably a bad idea! See the
detailed ChangeLog below for details on the ^ prefix.

* site who is now a built-in command. This conflicts with the venerable
SiteWho.exe addon, nxTool's site who, ioB's site who, and a bunch of
similar scripts. See below for configuration options.

* site open/close are now built-in commands. This conflicts with nxTool's
open/close. See below for configuration options.

* site purge/readd are now built-in commands, and site deluser has changed.
This conflicts with ioDELUSER.


---------


*** File Modifications:

1) Files in \system:
Changed: ioFTPD.[exe,pdb] - Version 6.5.0.0
Changed: tcl85t.[dll,pdb] - Version 8.5.2.2
Changed: ioFTPD.ini - summary of changes by section...
[Devices] : *telnet reference removed*
[Services] : *telnet reference removed*
[Sections] : *comments changed*
[FTP_Custom_Commands] : *comments changed*
[FTP_SITE_Permissions] : open, close, purge, re-add, who, swho
: *alphabetical now*
[Change_Permissions] : Expires, LimitPerIp, MaxDownloads, MaxUploads,
Opaque, showjobs (deleted)
[FTP] : Close_Exempt, Who_Hidden_Users, Who_Hidden_Paths_#
[Events] : *comments changed* OnClosedLogin, OnClosedKick,
OnDeletedLogin, OnDeletedKick, OnExpiredLogin,
OnExpiredKick, OnUnknownLogin,
OnTelnetLogIn (deleted)
[Telnet_Service] : *deleted*
[Telnet] : *deleted*
[Telnet_Binaries] : *deleted*
[Telnet_Binary_Permissions] : *deleted*
[Telnet_Command_Permissions] : *deleted*

2) Replace the lib\ directory entirely with the new tcl v8.5 version which
should have 2 new directories tlc8 and tcl8.5.

3) Files in \text\ftp:
Added : ClientInfo.[Common, List]
Changed : ClientInfo.[Download, Idle, Login, Upload]
Changed : ClientList.[Download, Header, Idle, Login, Upload]
Added : ClientList.[Footer, List]
Added : DeletedKick
Added : ExpiredKick
Deleted : Free
Added : ServerClosed
Added : ServerStatus
Changed : UserInfo
Changed : UserList.[Body, Header]
Added : UserList.Footer
Added : Who.[Download, Footer, Header, Idle, List, Upload]

4) Files in \etc:
Changed : Hosts.Rules

5) Files in \doc:
Changed : itcl.txt
Changed : cookies.txt

6) Replace the entire source\include directory with the new one.


*** New Features:

7) The number of sections the server supports has been raised from 10 to 25.
The [Sections] documentation in ioFTPD.ini has been updated with much
more detailed examples.

8) New ioFTPD.ini [Events] feature. You can now use "!" to indicate that a
particular event should display a message file to the user much like you
can do inside FTP_Custom_Commands, etc. The [Events] documentation
has changed to reflect this feature.

9) New [FTP_Custom_Commands] feature ("^"). You can now prefix events with
"^" which turns on the override option which means that the server should
not try to execute any built-in site command of the same name as the
trigger. The [FTP_Custom_Commands] documentation changed.

10) New [FTP_Pre-Command_Events] and [FTP_Post-Command_Events] feature. You
can now define pre/post events on site commands. This includes built-in
commands! Just prefix the name of the site command with '@' in
FTP_Pre-Command_Events or FTP_Post-Command_Events. The order for site
command processing now looks like this.

1) Validate access to command via FTP_SITE_Permissions.
2) Process @cmd events in FTP_Pre-Command_Events and return failure on
the first error encountered.
3) Process matching events from FTP_Custom_Commands, until one returns
an error in which case the command is considered to have failed so
return failure.
4) Run the internal site command of the same name if one exists and no
event included the ^ prefix. If the internal site command is executed
and returns an error then return failure.
5) Process all @cmd events in FTP_Post-Command_Events ignoring failures.

11) Site rename, delete, addip, deleteip, userinfo, kick and all site change
commands have been modified to prevent pure group admins (G and not 1M
flagged users) from performing any actions on 1M flagged users who
happen to be in their group.

12) New type of cookie that's not only super, it's downright amazing! :)
The new amazing %[IF] super cookie supports simple conditional logic!

The first form: %[IF(cookie)(EQUAL)(string)(true-action)(false-action)]

Here <cookie> is any text, mangled cookie and/or mangled supercookie
whose textual output is compared to <string> and if a match then the
mangled <true-action> is evaluated else the mangled <false-action>, if
present, is evaluated. Mangled in this context means a simple character
replacement where the following changes are made to the string before
processing. {{'s are replace with {, }}'s with }, {}'s with []'s,
<< with <, >> with >, and <>'s with ()'s. This is necessary because the
cookie parser is pretty dumb and making it stateful isn't worth the
hassle it would take.

Example:
%[IF(%{$VFS})(EQUAL)(<default>)()(| VFS : %-60{$VFS} |)]
Read this as if %[$VFS] evaluates to <default> then don't print anything,
else pretty print a formatted VFS field.


The second form: %[IF(super-cookie)(THEN)(true-action)(false-action)]

Here <super-cookie> is a syntax mangled super cookie which evaluates
to true if no error in parsing takes place and at least one character
would have been printed, else it's false.

Example:
%[IF(%{sectionname<0>})(THEN)(| DAYUP: %7.0{dayup<bytes><mega><0>} |)]
Read this as if a name can be found for section 0 (which means it has
an entry in ioFTPD.ini) then print the number of megabytes the user
uploaded into section 0 today.

NOTE: with appropriate formatting you CAN enclose an %[IF] inside the
true/false clause of another %[IF] although it looks ugly... See the
UserInfo file for examples of this.

13) New message super cookie (%[RemoveBlankLine]). Because of the way the
message parser works a line that contains only a %[IF] super cookie that
evaluates to nothing will result in a prefixed (i.e. "200-") empty line.
If this isn't desired use this super cookie at the end of the line and
it will erase the prefix and return character thus eliminating the line.

14) Updated site command (site shutdown [cancel|now|<seconds>]). If you issue
"site shutdown now" or "site shutdown 0" the server responds as before and
immediately begins terminating all connections and exits. If, however,
you don't specify any arguments (assumes default of 300), or you specify
a number, the server will consider this the length, in seconds, of the
grace period to allow for the orderly logging out of connected users.
After the grace period expires the server will revert to terminating all
connections and then exiting. During the shutdown grace period non-M
flagged users will be prevented from logging in and will be informed
that the server is shutting down as the reason why. All currently logged
in non-M flagged users, except the user who initiated the shutdown command
if you allow non-M users access to the command, will be logged out as soon
as they try to enter a command after their current transfer, if any, is
completed. Once the user who initiated the shutdown is the only user left
online the server will exit even if the grace period hasn't expired. If
the server doesn't immediately exit because it is waiting for someone to
log off you can use the "site shutdown cancel" command to stop the
shutdown sequence.

15) New message super cookie (%[ShutdownGrace]). This returns the number of
seconds until the server shuts down or 0 if it isn't shutting down.

16) New site command (site close [-kick|-new] <msg>). This command is used
to prevent non-exempt users (see Close_Exempt below) from logging into
the server, and optionally to force already connected non-exempt users
to be logged out. If neither -kick or -new is specified then site close
will default to gracefully logging users off as soon as they try to enter
a command after their current transfer, if any, is completed. The
optional <msg> argument can be used to provide a reason for the closure.
Before logging a user off gracefully the server will call the
"OnClosedKick" event which, if used in the default configuration below,
will display the time since the server closed along with the reason if
one was specified. If the -new argument is supplied then the server will
only prevent logins from new non-exempt users and will not logoff existing
users. The -kick argument will immediately close the connections and
terminate any transfers of non-exempt users and is therefore not
considered graceful. Non-exempt users trying to login to a closed server
will see the results of the "OnClosedLogin" event, as well as having the
login denied with the following error: "Server is closed".

17) New site command (site open). This command returns to normal a previously
closed server.

18) New ioFTPD.ini option (Close_Exempt under [FTP]). This is a permission
style expression which should match users who are allowed to login to
a closed server. Default is 1M flagged users, although adding "-sitebot"
(if you use one) probably makes sense.

19) New ioFTPD.ini settings (close, open under [FTP_SITE_Permissions]).
Both close and open are set to "1M", but this setting should match whoever
is listed in Close_Exempt since anyone who closes a server who isn't
exempt will get kicked off and be unable to open it again.

20) New message cookie (%[$ClosedOn]). Returns the time() the server was
closed or 0 to indicate the server is open.

21) New message cookie (%[$ClosedMsg]). Returns the <msg> argument to site
close if one was provided. Only valid if $ClosedOn != 0.

22) New ioFTPD.ini event (OnClosedKick, OnClosedLogin under [Events]).
OnClosedKick is run when a non-exempt user is about to be logged off
gracefully because the server has been closed. OnClosedLogin is run when
a non-exempt user tries to login to a closed server.

The suggested settings are:
OnClosedKick = !..\text\ftp\ServerClosed
OnClosedLogin = !..\text\ftp\ServerClosed

The ServerClosed file just prints the time since the close command and
optionally the reason for the server being closed.
"Server closed <time since close> ago [for <reason>]

23) New server status updating feature (..\text\ftp\ServerStatus). Because
M flagged master accounts will never be kicked from a server or denied
the ability to login even if the server is shutting down they have no
way to know what is going on. Similarly, exempt users on a closed server
have no idea if they should be getting off. Therefore the server will
now attempt to display the ServerStatus message file whenever a user
changes directories, finishes transferring a file, or issues a NOOP
command IF the server status is closed or in shutdown mode. By default
the file is setup to display the appropriate line or lines based on the
server's status with an extra blank line before and after if something
was printed.

*** SERVER IS SHUTTING DOWN *** -- <grace-left> seconds remaining

*** Server CLOSED <time-since-close> ago -- Reason: <reason>

24) Updated site command (Site deluser <user> [<msg>]). Deluser now works
more like glftpd does in that site deluser must be followed by a site
purge to actually delete an account. Previously this command would delete
the UserFile as soon as possible, but it now simply marks the account as
deleted by recording the user who deleted it, the time it was deleted,
and, as a new feature, it also takes an optional <msg/comment> argument
will show show up when the user is viewed via site uinfo, or when the user
is shown in a list via site users, site purge, site readd, etc. Deleted
users are prevented from logging in and you can configure how the system
should respond to such events (see below). For example you can choose to
show a file (in which you can refer to the <msg> argument) or just return
an error indicating the account has been deleted, or hide the deleted
status altogether and just return the generic invalid password error reply
as would previously have happened. A currently logged in user who is
deleted will be forced off at the next command after any active transfer
completes. Group memberships are maintained until the account it purged
so group counters are not immediately updated. If you site deluser on an
account already marked for deletion only the <msg> field is updated.

25) New UserFile field (DeletedOn). This is the unix style time() when the
account was marked for deletion. 0 implies the account has not been
marked/deleted.

26) New message cookie (%[$DeletedOn]). Returns the setting in the UserFile.

27) New UserFile field (DeletedBy). This is the UID of the user who deleted
the account. Field not valid unless DeletedOn != -1.

28) New message cookie (%[$DeletedBy]). Returns the setting in the UserFile.

29) New UserFile field (DeletedMsg). This is the optional <msg> argument
provided to site deluser. Field not valid unless DeletedOn != -1.

30) New message cookie (%[$DeletedMsg]). Returns the setting in the UserFile.

31) New ioFTPD.ini event (OnDeletedKick under [Events]). This is shown to a
user when they are being logged off gracefully because their account has
been deleted. Default is !..\text\ftp\DeletedKick which just contains a
simple "Account has been deleted." message with some extra line padding.

32) New ioFTPD.ini event (OnDeletedLogin under [Events]). If a user tries to
login to a deleted account this event is called. If not defined the user
will see the error message "Account has been deleted". If defined but
returns an error it will print any text in the buffer, but the user will
see the error message "Invalid Password". Therefore a simple way to
achieve the old behavior of a deleted account returning the generic
invalid password error is to just use the new !messagefile syntax and
point to a non-existent file. The event is run in the context of the
deleted account and therefore you can refer to any UserFile cookies you
want including the new DeletedMsg cookie which holds the <msg> argument
passed to deluser so you can show it to the user if you want. Default is
a commented out undefined placeholder.

33) New site change command (site change <who> expires <arg>). ioFTPD now
supports temporary accounts. To set an account to expire at a particular
time use site change <who> expires with one of the following <args>:
yyyy-mm-dd : expires at the start of the identified day
yyyymmdd[hh[mm[ss]]] : expires at the start of the identified day
or optionally at a specific time on that day
+h* : expire h hours from now
never : account permanent and shouldn't expire

Expired accounts are not automatically deleted, but will show up under
site readd or site purge listings (see below) once they have expired and
will indicate that they are expired with the new text\ftp\UserList.Body
file installed. Expired accounts are prohibited from logging in, and
already logged in users will be forced off the same as if the account had
just been deleted.

34) New ioFTPD.ini settings (Expires under [Change_Permissions]).
Default set to "G1M".

35) New UserFile field (ExpiresAt). time() at which the account should
expire. 0 implies a permanent account.

36) New message cookie (%[$ExpiresAt]). Returns the time() at which the
account should expired. 0 implies a permanent account.

37) New message super cookie (%[Expired]). Prints 1 if the account has
expired, 0 if it hasn't expired but does have an expiration time, and
prints nothing if the account is permanent. Useful for %[IF] processing.

38) New ioFTPD event (OnExpiredKick under [Events]). This is shown to a user
when they are logged off because their account just expired. Default
is !..\text\ftp\ExpiredKick which just contains a simple "Account has
expired." message with some extra line padding.

39) New ioFTPD.ini event (OnExpiredLogin under [Events]). If a user tries to
login to an expired account this event is called. If not defined the
user will see the error message "Account has expired". If defined, but
the event returns an error, the user will see the error message "Invalid
Password". The event is run in the context of the expired account.
Default is a commented out undefined placeholder.

40) New site command (site purge [<user|*>]). Site purge without any
arguments lists the users who have been deleted and the accounts that have
expired that are available for purging by you. The "by you" is important
because there are now rules on who can purge who beyond having been
granted access to the command in the .ini file. Master (M flagged)
accounts can purge anyone, siteop (1 flagged) accounts can purge non-1M
accounts, and pure group admins (G but not 1M) can only purge regular
users in groups they admin. With this system in place a siteop can site
deluser another siteop, or a pure group admin another pure group admin in
their group, and this will prevent those users from logging in, but it
will take someone more superior to actually complete the account deletion.
Site purge <user> will purge a particular user, and site purge * will
purge all you have access to.

NOTE: Make sure you update the ..\text\ftp files because site purge/readd
only identify the reason the user matched (i.e. deleted or expired)
because the new UserList.Body file makes use of %[IF].

41) New site command (site readd [<user>]). Site readd without any arguments
lists the users who have been deleted and the accounts that have expired
that are available for re-adding by you. The rules for re-adding are
looser than those for purging. All 1M flagged users can re-add anyone,
and pure group admins can re-add members in their admin groups provided
the target isn't a 1M flagged user. Re-adding an expired account turns it
into a regular account without any expiration date.

42) New ioFTPD.ini settings (purge, readd under [FTP_SITE_Permissions]).
Both close and open are set to "G1M", but remember access to the command
does not guarantee you can execute it against particular users. See the
rules above.

43) New site command (site who [up|down|idle|bw|<users>]). Site who differs
from site swho in that it is designed to be used by non siteops and
therefore displays only non-sensitive data and can be configured to hide
users entirely, and/or obscure matching paths with the text <hidden>. It
also obscures the last command for each connection so that site commands
show up only as "site command" without any arguments, and the PORT and
PASV arguments are hidden as well. The PASS command is already obscured
everywhere even with site swho.

The following optional arguments are available:
up : just show uploaders
down : just show downloaders
idle : just show idlers
bw : just show header/footer for summary information
<users> : a "site users" style list (.FLAG, =group, name*, !, etc) to
limit display to just matching users

44) New ioFTPD.ini option (Who_Hidden_users under [FTP]). Users who match
this expression will be hidden from site who listings. Format is similar
to site users, i.e. name, name*, =group, .FLAG, !something, etc

45) New ioFTPD.ini option (Who_Hidden_Paths_# under [FTP]). Up to 20 path
entries may be provided and matching virtual paths for CWD or file being
transfered will be replaced with "<hidden>" it a match is found.

46) New ioFTPD.ini settings (who under [FTP_SITE_Permissions]). Default set
to "!A *" which is all non-anonymous users.

47) Updated site command (site swho [up|down|idle|bw|<users>|<cid>]).
Previously this command took no arguments for a list of users, or one
number which represented the connection-id to display more detail on.
It now accepts the same optional arguments as site who to limit the
list of users to display. The new text\ftp\ClientInfo.* files used with
"site swho <cid>" now display a lot of what "site uinfo" shows so you can
see details on the account along with connection details and the current
action. The new text\ftp\ClientList.* files also display more information
about what each user is doing and in a nicer format.

48) Modified ioFTPD.ini setting (swho under [FTP_SITE_Permissions]). Removed
the "G" flag from default permissions for site swho because site who
should be sufficient now.

49) Updated site command (site bans remove <IP|mask>). Site bans remove
now accepts a wildcard IP mask instead of only a fully qualified numeric
IP address. Thus "site bans remove *" will remove all active bans.

50) New ioFTPD.ini settings (NonLeech under [Change_Permissions]).
Default set to "1M". This was missing from previous releases.

51) New site change command (site change <who> MaxDownloads <arg>). ioFTPD
now allows you to control the number of simultaneous downloads a
particular user may have active at any one time across all their logins.
An <Arg> of -1 implies unlimited, 0 means the user can't download at all.
For existing accounts that predate this field the value is set to the
default of -1. This only affects the number of downloads an account can
have active at one time, and not the total number of allowed logins for
that user.

52) New site change command (site change <who> MaxUploads <arg>). ioFTPD
now allows you to control the number of simultaneous uploads a
particular user may have active at any one time across all their logins.
An <Arg> of -1 implies unlimited, 0 means the user can't upload at all.
For existing accounts that predate this field the value is set to the
default of -1. This only affects the number of uploads an account can
have active at one time, and not the total number of allowed logins for
that user.

53) New ioFTPD.ini settings (MaxDownloads, and MaxUploads under
[Change_Permissions]). Default set to "1M" for both.

54) New UserFile fields (MaxDownloads and MaxUploads).

55) New message cookies (%[$MaxDownloads], and %[$MaxUploads]). Returns the
setting in the UserFile.

56) File Hosts.Rules changed to allow everything by default. In particular
it is important to update the connections_per_ip to be -1 to disable
this feature because the previous limit of 4 connections per IP was just
too low if you wish to allow mass downloading, or mass uploading while
just allowing a single download per user.

57) New site change command (site change <who> LimitPerIp <arg>). ioFTPD
now allows you to control the number of simultaneous logins an account
may have active at any one time from a single IP address. This check is
performed at password verification time and is a per user setting. The
connections_per_ip setting in the Hosts.Rules file is across all users
from a particular IP address and thus is a stronger guarantee.

58) New ioFTPD.ini settings (LimitPerIp under [Change_Permissions]).
Default set to "1M" for both.

59) New UserFile field (LimitPerIp). Max number of connections from a single
IP address.

60) New message cookie (%[$LimitPerIp]). Returns the setting in the UserFile.

61) New site change command (site change <who> opaque <args>). ioFTPD
now allows you store up to 256 characters of extra data in a UserFile.
The server does nothing with this field except preserve its contents in
the UserFile in the hope that someone will find it useful. To encourage
compatibility the suggested format is TCL style quoted fields so it can
be easily parsed.

62) New ioFTPD.ini settings (opaque under [Change_Permissions]). Default set
to "1M" although it likely on scripts/sitebots will need access.

63) New message cookie (%[$Opaque]). Returns the setting in the UserFile.

64) New UserFile field (CreatorUid). This is the UID of the creator of the
account. For existing accounts that predate this field the UID is -1.

65) New UserFile field (CreatorName). This is the name of the creator of the
account at the time of creation. The reason this field exists is that
the user who created this account may be deleted at some point and thus
the server would be unable to lookup the creator's UID anymore. For
existing accounts that predate this field the name is left blank.

66) New message cookie (%[$Creator]). Returns the current name of the UID
who created the account (the CreatorUid field) if it can be resolved.
If it can't, but a name was cached in the CreatorName field then return
that. Finally return the string "<unknown>" which is the default
behavior for accounts created before this field was introduced.

67) New UserFile field (CreatedOn). This is the unix style time() when the
account was created. For existing accounts that predate this field the
time is set to 0.

68) New message cookie (%[$CreatedOn]). Returns the setting in the UserFile.

69) New UserFile field (LogonLast). This is the unix style time() when the
account last successfully logged into the server. For existing accounts
that predate this field the value is set to 0 until the first login where
it will be tracked normally from there on.

70) New message cookie (%[$LogonLast]). Returns the setting in the UserFile.

71) New UserFile field (LogonCount). This is a count of the total number of
times this account has successfully logged into the server. For existing
accounts that predate this field the value is set to 0 and will increment
normally from there on.

72) New message cookie (%[$LogonCount]). Returns the setting in the UserFile.

73) New UserFile field (LogonHost). This is the ident@hostname/IP of the
last successful login to this account after Obscure_Host/IP have done
their job. For existing accounts that predate this field, or for newly
created accounts, this value will be equal to "".

74) New message cookie (%[$LogonHost]). Returns the setting in the UserFile.

75) New message cookie ($VFS). This returns the user's VFS file setting. It
was missing from previous versions.

76) New message cookie ($FTPLogins). This returns the number of times the
user is currently logged into the FTP service under this account.

77) New message super cookie (%[SharedCredits(suffix)(section)]). This is
very similar to the %[Credits] super cookie. However if the section is
setup to share credits with another section instead of using it's own,
this will print "Shared (#N)" where N is the section whose credits are
being used.

78) New message super cookie (%[SectionName(section)]). This will print out
the name/<alias> defined in ioFTPD.ini for the first entry whose stats
section field matches the requested section number. This is useful to
print section names in server responses or UserInfo requests.

79) New message super cookie (%[HasFlag(flags)]). This returns "1" if the
UserFile contains at least one of the indicated flags else it prints
nothing. Only really useful for %[IF] processing.

80) New message super cookie (%[CallerHasFlag(flags)]). This returns "1" if
the user who called the command (rather than the current UserFile)
contains at least one of the indicated flags else it prints nothing.
Only really useful for %[IF] processing. Currently this is limited to
commands which load JUST a UserFile other than their own such as site
readd, purge, uinfo, groupinfo, and users. Site who/swho, etc load both
a UserFile and Online statistics and are not currently supported.

81) New message super cookie (%[Duration(time1)(time2)(extra)(lastfield]).
This command displays the time between time1 and time2 (defaults to 0
if not specified) in y, w, d, h, m, s form. The Extra field is additional
text to print as part of the result after the duration is printed and
can be "" if necessary. If lastfield is specified this will stop the
output after the indicated field has been printed. Thus "d" will only
print years, weeks, and days.

82) New message super cookie (%[WHO(VIRTUALDATAFILE)]). This cookie shows
just the filename (no path) of the currently active transfer for the user.

83) If a user is thrown off for exceeding the ioFTPD.ini setting for
Login_Attempts tries at logging in then an error message is generated to
Error.log:
"Host 'ident@obscured_ip' (obscured_hostname) had <Login_Attempts>
failed login attempts - last user was '<username>'."
Multiples of these with different usernames should indicate someone is up
to no good.

84) The following login failure error messages are now rate limited.

1) Host '%s@%s' (%s) did not match any of user '%s' allowed hosts.
2) User '%s' tried to login with improper rights to service '%s'.
3) User '%s' (%s@%s) tried to login with invalid password.
4) Warning: Group file for group '%s' does not exist, or is corrupted.
5) VFS file '%s' is invalid or missing.
6) Ini entry 'Default_VFS' under 'Locations' is missing.
7) User '%s' does not have access to root directory.
8) Host '%s@%s' (%s) had %d failed login attempts - last user was '%s'.

The rate limiting is similar to that used for connection rejections
however the timer is shared between all 8 error messages. It will,
however, always print an error message type (1-8) from each IP that is
different from the one last printed. Thus lots of say type 3 messages
until the user guesses his password might be suppressed and a long delay
before the next would be printed won't suppress the #7 bad root dir error.

I got tired of seeing lots of bad password, bad hostmask errors in my
logfiles as users tried to connect over and over...

85) New ioFTPD.ini option (Log_Suppression_Increment under [Network]). This
is the number of minutes to increase the delay between each suppressed
message up to Max_Log_Suppression. Default is 1 additional minute per.

86) New options to the iTCL change command (Expires, LimitPerIp, MaxDownloads,
MaxUploads, NonLeech, Opaque). All of these take the same arguments as
the similarly named site change command described above.

87) iTCL waitobject changed. The design of this just sucked. It's broken
for all but trivial uses of a simple fixed name lock that is never
deleted. I'll remind readers I had no part in this thing!!! If you
attempt to use a dynamically named lock and clean up after yourself
it breaks. In this example the number in parens is the internal
reference count, and because there is no way to tell if open created the
lock which gives it 2 references to start with, or just opened an
existing lock which adds 1 reference you can see what happens...

thread 1 thread 2
-------- --------
open "test"(2)
open "test"(3)
close "test"(2)
delete "test"(1)
close "test"(0)
delete "test"(DOH!)


Even if open returned an indication of a new lock verses an existing
lock or the reference count was always incremented twice the fact that
the delete operation always removes the name from the list of known
locks will cause problems as shown here:

thread 1 thread 2 thread 3
-------- -------- --------
open "test"
open "test"
close "test"
delete "test"
open "test" (DOH!)

Therefore I have removed the delete operation and made the close operation
do the work of both. It closes the local reference to the lock, and if
there are no other open references it deletes the lock as well. This
should solve the above problems.

88) New iTCL waitobject function (preserve). Because the cleanup code run
after a script exists automatically closes any local references to open
locks that you haven't explicitly closed already by the script it is
now impossible, with the above change, to have a lock that outlives the
current script invocation. To solve this the "preserve" operation simply
increases the internal reference count by 1 which means it won't be
deleted until you later call close twice on it.

89) New iTCL waitobject function (list). Useful really only as a debugging
tool this command will return a list of the names of all known TCL locks.

90) New ioFTPD.ini event (OnUnknownLogin under [Events]). If the USER command
entered by the user during login refers to an unknown user name then this
event will be called. If it returns true, which normally would indicate
an error, the user lookup is performed again. The addition of this event
should allow the dynamic creation of new user accounts which may be useful
if there are thousands of them stored in an Active Directory or other 3rd
party database and waiting to import them until they have been referenced
makes sense. Also useful in conjunction with a shared user module that
doesn't expect every account to use the server. Default is undefined.

91) The shared memory interface now supports the following new identifiers
DC_NEW_USERFILE_OPEN, DC_NEW_USERFILE_LOCK, DC_NEW_USERFILE_UNLOCK,
and DC_NEW_USERFILE_CLOSE which work exactly like the DC_USERFILE_*
identifiers do except they take the new version of the USERFILE structure.

92) The shared memory interface has changed the meanings of DC_USERFILE_OPEN,
DC_USERFILE_LOCK, DC_USERFILE_UNLOCK, and DC_USERFILE_CLOSE which now
have a context of USERFILE_OLD which is a structure that is byte
compatible with previous versions of USERFILEs so this change only
applies to users recompiling their code and not existing executables.

93) The text\ftp\UserInfo file, which is what site uinfo uses to display
information about a user, has been totally reworked to take advantage
of the new user fields and makes heavy user of the %[IF] super cookie
to format the output and hide unimportant information and/or missing
fields. It is only setup, by default, to display section based info
for up to 5 sections. It won't show info on undefined sections so you
should only see 1 if that is all you are using. To support more than
the default, just edit the file and either remove the %[IF] checking
since you should know how many sections you are using, or continue to
use the same style and copy/paste/edit the lines for more sections.

*** Functionality changes:

94) THE SERVER NO LONGER SUPPORTS THE TELNET SERVICE.
Delete the following sections from the .ini file:
;OnTelnetLogIn in [Events]
[Telnet_Service]
[Telnet]
[Telnet_Binaries]
[Telnet_Binary_Permissions]
[Telnet_Command_Permissions]
Reference to telnet under *Devices* and *Services* comments.

95) The %[include] message super cookie now processes cookies in the included
file.

96) Deleted ioFTPD.ini setting (showjobs under [Change_Permissions]). No
such command.

97) The idle time in site who/swho/etc is now the internal idle timer as
controlled by Idle_Ignore, etc and not the last time since a command was
entered.

98) ioFTPD will no longer display the .message file upon entering directories,
but will continue to display the .ioFTPD.message file which is hidden
automatically in directory listings, deleted automatically when a
directory is deleted, etc. The .ioFTPD.message file is already used by
most scripts.

99) ioFTPD will no longer process cookies in the .ioFTPD.message files which
will result in such files no longer having their parsed form cached in
the \cache directory. This should be a benefit since they never included
message cookies anyway.

100) Changed %[Ratio(section)] super cookie so that if the user has a ratio
of 0 it will print "leech" instead of "unlimited".

101) Changed %[time(format)(time)] cookie so that if time equals 0 it will
print "<never>" instead of "Jan 1, 1970...".

102) site rehash now invalidates existing TCL interpreters so that the next
TCL command will re-create the interpreter for the thread. The benefit
of this is you can now setup scripts to avoid reloading files already
included which offers a significant performance improvement. If you did
this previously you would be unable to edit a configuration file or
script and have the changes show up in existing TCL interpreters until
the server was restarted.


*** Bug Fixes

103) System error messages are now forced to use US English instead of the
user/system settings. This probably makes sense because all the ioFTPD
specific errors are in English already. Asian languages and other wide
character window's builds resulted in things that look like this when the
unicode conversation failed for window's system error messages:
426 Connection closed: xxx xxxx xxx x xx xxx x xxxx.

104) Minidumps now contain information open handle information to assist
in finding the lockup problem.

105) Transfer speed updates only occurred when new data was received and only
if at least 1 second had passed since the last update or start of the
transfer request. It will now update every 100ms which should allow
fast transfers taking under a second to have a shot at updating the
transfer speed instead of always reading 0. The event driven update
also failed to accurately reflect what was going on when a transfer
stalled so if no data has been received within 10 seconds a 0 byte
transfer of 1 ms length is faked so the stalled status can be seen.

106) Site chmod and site symlink would attempt to get information about the
parent directory of their target and this would fail if the parent
directory was an NTFS hidden directory that contained the target
as a mountpoint. This should now work.

107) The server was missing the ClientInfo.List and ClientList.List files
which meant that site swho would completely hide a user who was actively
transfering a directory listing. Fixed with the new text\ftp files.

108) The tracking of all valid IP address to the server would remove a users
entries and re-apply then each time a UserFile was closed. This is
a waste of time since most likely the UserFile was just updating
statistics or credits or something. The before and after hostmasks
are now compared and if no change is detected then the IP address
updating is not performed.

pointBreak
05-24-2008, 05:08 AM
Looks great, thanks Yil

JoC
05-24-2008, 06:11 AM
Impressive list of changes, great work Yil.

Mave
05-24-2008, 03:38 PM
Holy moly!
I guess when you came to earth you had a vision :p

Great work!!!

Yil
05-25-2008, 02:54 AM
I've been poking around scripts and noticed that nxTools can be setup to do some really cool things with the %[execute] cookie. I expect things like oneline messages, latest dirs, etc to go in the Welcome file, but others like listing the requests or section rules probably went into various .ioFTPD.message files. Unfortunately I just broke that because I've disallowed cookie processing in those files. I was only thinking of zipscripts, at the time, and the potential problems they pose writing to .ioFTPD.message. Because of those problems I won't be re-enabling that feature. I also thought the parsing and caching of hundreds or thousands of .diz/.nfo/sample/etc extracts that get stuffed into .ioFTPD.message files was stupid which happens because of the cookie parsing.

I expect I'll return the functionality by automatically showing a new filename like .ioFTPD.cwd which will have cookie parsing enabled and be shown whenever someone enters the directory. Right before the text of ioFTPD.message is displayed. If anyone knows of any situation where this won't be sufficient speak up now :)

As I mentioned in the v6.5 discussion thread, I'm looking at ways to eliminate or at least reduce configuration of the server and the duplication of data. For example I'd like to make site free support in sitebots not need to know about physical paths/sections. I'll probably add a new site command to provide all this dynamically since the server clearly knows where everything is. Once I figure out what disk spanning fits in hehe...


For those of you who use nxTools with a sitebot do you use Alcobot with the nxTools module for your sitebot? ioNinja looks like it has some nxTools supports and it calls and loads some Alcobot libraries but doesn't use the module. Just sort of taking a poll, what configuration are people using it in and why?

o_dog
05-25-2008, 12:36 PM
what do you mean by site bots knowing about physical paths and sections? i mean that part of the sitebot has nothing whatsoever to do with ioftpd.

changing cases in userfile on wkup etc might cause problems with some older scripts, and i really see no point to it. mistake?

Yil
05-25-2008, 04:20 PM
If the sitebot config file has stuff like section GAMES is d:\Games then it needs to know real paths. Clearly it needs to know this somehow so it can do things like site free, but what I'm looking to do is have the sitebot get this information from the server so if you change the VFS file you don't have to also change the sitebot... I expect a few new iTCL functions and perhaps a new site command should make all this really easy to do.

Sitebots will likely always have to know most section names since output is often redirected different places based on section, but I don't think there is any reason for it to know about real paths. On the other hand, the "default" configuration could just enumerate the section names as well and only if you want to do something like suppress messages or redirect them would you need to configure it manually.

Hmm, good point on the case changes in the userfile. It's simple enough to go back, but let's see what breaks. If it's TCL scripts parsing the output of bin2ascii they will be trivial to change, and if it's other scripts reading the userfiles directly then I want to discourage that anyway. The shared memory interface never sees any of this so it won't care. Things like nxShareDB won't have local userfiles and going to something like SQLite for the user db in the future would break direct userfile access anyway. Best to query the server for the info. It's simple enough to go back to all lowercase for the older fields though so if something important can't be fixed easily I'll get a new version out...

o_dog
05-25-2008, 11:35 PM
the sitebot does not get any information through the ftpd. It uses shared memory. The real paths for example could be used when determining where a release resides for nuking purpose from irc. It also converts paths to pwd and sections. I wouldn't waste my time on this if i were you.

The point beeing that not many of the scripts around are under active development, if it breaks stuff it wont get fixed.

jeza
05-27-2008, 12:19 AM
what do u mean with "sitebot get this information from the server ".
my scripts also use shared memory to get info from ftpd.
do u plan something with itcl, so bot can directly get info from ftpd (free, stats, delete user,...)?
in this case all .exe scripts (sitewho and other stuff) would not be needed anymore - and this is cool

pion
05-27-2008, 05:31 AM
I know I've asked about this before, but would it be possible to add the option for downoading a file in use? E.G getting uploaded. And keep sending the data stream until the upload of the file is complete.

Yil
05-27-2008, 04:35 PM
jeza: wrote a nice reply to you but the forum went down for maintenance and it got lost... The short version: Shared memory isn't going anywhere, although it's not a complete interface because getting stuff like configured sections, vfs info, etc isn't possible right now but if someone proposes an extension I'll happily add it. As far as sitebots go, everything can already be done via iTCL, the only question is does the iTCL directly provide the information or do scripts still have to parse ioFTPD config files. It's the latter I want to get rid of.

pion: don't think so. I know glftpd supports that, but it has to do some extra checks to make sure the upload is still making progress, etc. Seems like a lot of work. On the other hand, if the problem is you are fxping files that are still being sent and thus arriving incomplete, it would make sense to go the other direction and prohibit downloading of files being uploaded altogether. That would make far more sense especially since I can see zipscripts adding or subtracting .nfo files from a zip which could completely corrupt a file that had been partially transmitted before the change and the rest after...

jeza
05-27-2008, 11:32 PM
it would be the best if u write some .dll or .exe with ioFTPD, that we could use with sitebots.
it will be always updated, because u will develop it with ioFTPD.

if u plan to do it, it should give info about WHO, FREE, STATS/section, CREDITS/section, get settings from CONFIG and also posible to manage users (CHANGE, DELETE, ADDIP, DELIP,...)

maybe if u have some free time, u can write it :)

Flow
05-28-2008, 02:21 AM
*phew*, just upgraded. This one was big :p

I dont get howto disable buildin sitewho (since i use nxTools). Cant be be turn off ? Is listed double now.

^who = TCL ..\scripts\nxTools\nxUtilities.tcl WHO
^close = TCL ..\scripts\nxTools\nxClose.tcl CLOSE
^open = TCL ..\scripts\nxTools\nxClose.tcl OPEN

9) New [FTP_Custom_Commands] feature ("^"). You can now prefix events with
"^" which turns on the override option which means that the server should
not try to execute any built-in site command of the same name as the
trigger. The [FTP_Custom_Commands] documentation changed.

pion
05-28-2008, 06:37 AM
pion: don't think so. I know glftpd supports that, but it has to do some extra checks to make sure the upload is still making progress, etc. Seems like a lot of work. On the other hand, if the problem is you are fxping files that are still being sent and thus arriving incomplete, it would make sense to go the other direction and prohibit downloading of files being uploaded altogether. That would make far more sense especially since I can see zipscripts adding or subtracting .nfo files from a zip which could completely corrupt a file that had been partially transmitted before the change and the rest after...

glftpd supports this, yes. But VideoLan player also support reading a file beiing written to.. which is also open source.. dunno how easy it would be to use the logic from there?

The problem is that the file is unavailible when it's uploaded, any media file with a complete header can be viewed which is handy for multiple purposes.

Example; user 1 upload a cam recording to server 1, user 2 download cam recording at the same time, viewing it with vlc during the download process.

whocarez2k5
05-28-2008, 01:26 PM
Nice to see ioFTPD getting better and better.
only have 1 question for Yil, how to change values for site makecert? (NAME= / HOST=)
[1] 200-NAME="192.168.0.1" [FTP_Service Device (HOST=)]
[1] 200-Succeeded
[1] 200-Successfully loaded new cert!
[1] 200 makecert Command successful.

Yil
05-28-2008, 05:51 PM
Flow: You had "^who = TCL ..\scripts\nxTools\nxUtilities.tcl WHO" but what you want is "who = ^TCL ..\scripts\nxTools\nxUtilities.tcl WHO" :) Though I think the built-in version can be made to look the same if you adjust the Who.* files though and it's probably faster.

whocarez2k5: Actually makecert doesn't accept any arguments but instead uses the same algorithm the server uses to load a cert: (From the ChangeLog):
39) New site command (site MakeCert). First determine the preferred name for
the certificate by looking for a Certificate_Name= for the service, or
the Host= value for the device specified for the service provided it isn't
0.0.0.0, or finally the default value of ioFTPD. Then create and install
in the CurrentUser My/Personal store a new certificate if one doesn't
already exist. Requires the file system/makecert.exe and the M flag.


So just set Certificate_Name to whatever you want to use and site makecert will find that name.


pion: I can see the benefit of viewing an incomplete video file so locking it would prevent that, but it makes no sense to be viewing a file you have hit the end of and are waiting for more uploaded data from. Either the source is faster than the video frame rate in which case you won't have an issue today, or it isn't in which case the video will pause, jump, etc and it's pretty useless. I figure the bigger issue is with mirroring/racing files already in transit and for zip's the issue is one of zipcleaning, whereas .rars are usually fine. Perhaps there isn't a single solution and the answer is to support a list of file extensions to lock while transferring and the rest to wait for data from. Or visa-versa.

Yil
05-28-2008, 06:02 PM
Jeza: The shared memory interface already allows access to userfile data for both reading and updating so you can add/delip and look/change a user's stats, rename/add/delete users, and access client (who) data so of the list the only things not available are section name information and vfs/free data which is why I'm trying to solve those :)

I agree that a dll would be REALLY cool if it had a TCL wrapping so it could be loaded into windrops or standalone TCL scripts outside of the server... I'll have to think on that one because that does sound good. The only drawback would be you would have to host the windrop on the same machine, whereas a pure site based script solution wouldn't. It's also easier to get more specific error information from the site scripts because I don't think the shared memory interface offers much beyond success/fail indication which is usually sufficient but not always.

jeza
05-28-2008, 11:54 PM
yes i know u can already use shared memory for those things.
but it would be a good idea if u write a .dll with all this features and scripters can use it.
and if its "limited" to machine where ftpd is installed is not a big problem, because 99% of all servers use it this way.

o_dog
05-29-2008, 09:08 AM
there is already a dll for this purpose.
Neoxed wrote one for alcobot.
it's the same one i use in my bot. (with permission)

Yil
05-29-2008, 09:41 PM
It does look like AlcoExt06.dll uses the shared memory interface. There is no documentation or even google hits about how to use it but examining AlcoBot should be informative. On the other hand, it will need to be recompiled/updated to access the new userfile fields but should still work as is.

jeza
05-29-2008, 11:23 PM
i still think that you, Yill, should develop a .dll simultaniusly with ioFTPD.

whocarez2k5
05-31-2008, 07:38 AM
@Yil
Is there a complete command/options document for ioFTPD 6.5.0 some where in package?

Yil
05-31-2008, 11:13 PM
Not really. I totally redid the doc/Cookies.txt file so that should be up to date. The iTCL doc file leaves a little to be desired but is pretty good. The actual commands though? Try help.db and help.cmd perhaps. It's on the list of things to do at some point because I'd like to implement the HELP command :)

Mave
06-02-2008, 07:07 AM
Hey Yil

I was just wondering ... i did read the TELNET part has dissapeard which is fine to me but i still see in TEXT/TELNET those telnet text files... I assume we dont need those anymore or?

Also i checked the default userfile in USERS

RATIO 0 0 0 0 0 0 0 0 0 0

there are still only 10 sections counted ..... shouldnt it be 25 now?
Or wil they get auto added at all userfiles when the site gets more then 10 sections?
I assume its possible to set different ratio,s on all 25 sections.

Thx

whocarez2k5
06-02-2008, 08:36 AM
Hey Yil

I was just wondering ... i did read the TELNET part has dissapeard which is fine to me but i still see in TEXT/TELNET those telnet text files... I assume we dont need those anymore or?

Also i checked the default userfile in USERS

RATIO 0 0 0 0 0 0 0 0 0 0

there are still only 10 sections counted ..... shouldnt it be 25 now?
Or wil they get auto added at all userfiles when the site gets more then 10 sections?
I assume its possible to set different ratio,s on all 25 sections.

Thxabout that sections, is it possible to have sections in site uinfo <username> automaticly change depending on the amount of sections added (higher then 10)?
Now it's default 4, i changed it manual to the amount of sections i have (and isn't working btw, section 10 till 25 get data from first(default)section, and all sections after section10 aren't displayed) but would be great if it's dynamic.

@Mave:
It doesn't automaticly add them, i tested it with 15 sections, changed ratio on section 12 and it wasn't added or changed.
So i added extra 0's to it, changed ratio again, zero's was brought back to 10 again in user file.
Bug maybe?

Yil
06-02-2008, 12:58 PM
Don't edit user/Default.user manually. Use site change /Default.User and site uinfo /Default.User to set or view the settings.

DOH! I switched from 25 to 10 sections at some point when I was debugging something in the shared memory interface and forgot to switch it back... So, uh, I'll fix that! Soon!

whocarez2k5: Yea, I really should have found a better solution to supporting all the sections for uinfo but I figured so many people just use the default that parsing cookies for that many undefined sections was just stupid. I've gone ahead and split the UserInfo file into 3 parts with the section piece called once for each defined section. Much cleaner now...

I was going to get a new version out today with a cool new feature or two, but the new UserInfo took long enough that I ran out of time. Expect a new version you are going to LOVE in a day or two.

whocarez2k5
06-02-2008, 01:27 PM
Thanks Yil

1 thing before i start waiting for the new version.
I played around with some scripts today and suddenly ioftpd says:
[2] 426 Connection closed: Your user class requires you to use secure connections.

I removed all scripts and settings disabled SSL, enabled it again but it keeps giving me that message?

Didn't changed io specific settings. I guess i need to reinstall io again or do you have a solution?

Yil
06-03-2008, 02:00 AM
Require_Encrypted_Data is the setting that controls forcing SSL over data connections. If you do force it then it's up to you to make sure your FTP client is set to use SSL for transfers. Check your settings for the site. Particularly tricky is FXP SSL since you must enable SSL for BOTH sites else one side will be trying to negotiate an SSL connection and the other will just be expecting data and they won't be happy...