ioFTPD v6.7.0 *experimental*

[Yil]

Here it is folks. A version of ioFTPD and TCL that should work for users using code pages that define characters with the high bit set. In particular this should allow Asian users to use TCL scripts.

I've labeled this as an experimental release since early testing shows this to be working, but I had to change a lot of functions so there's a number of things I could have broken. I wrote my own UTF-8 converters since we aren't converting between code pages so much as just properly encoding them and it looks OK but clearly will need some testing on systems with other languages.

I've included my current work on TCL extensions and some other bug fixes, but rather than wait for everything to be finished completely I figured I should get the UTF-8 / TCL changes out the door to help those having problems now.

NOTE: Scripts that maintain state such as nxTools may need to rebuild their dupe databases since entries added prior to this version with Asian characters probably had incorrect entries loaded. So if you are having problems with searching, dupechecking, etc. consider a rescan/rebuild/whatever.

Let me know how it works.

Oh, and check out the new stat features. It's pretty cool to keep stats per section now to see who up/downs where.

Latest Version:
ioFTPD-v6.7.1.zip

[Yil]

v6.7.0 Release Notes:

1) Files in \system
   Changed : ioFTPD.[exe,pdb] - Version 6.7.0.0

2) Files in \text\ftp
   Changed: [AllDn, AllUp, WkDn, WkUp, MonthDn, MonthUp, AllDn, AllUp].Header

3) Files in \doc
   Changed: itcl.txt, Cookies.txt


*** New Features:

4) Modified site command (site stats [dir]).  Previously "site stats" would
   default to showing information for the current section.  Thus the results
   would depend upon where you were in the VFS.  You could, however, specify
   a particular section to display but there was no way to see stats
   across multiple sections.  Now site stats defaults to showing the TOTAL
   stats across all sections which should make it much less painful to keep
   individual statistics for each section since you can get the totals now.
   If, however, you wish the old behavior of seeing just statistics for the
   current section based upon your location in the VFS you can specify the
   "dir" option, or for a single specific section use "section #".  If you
   specify a section of -1 it means you want the total for all sections.

5) New cookie usage (%[pos] in text/[AllDn, AllUp, MonthDn, MonthUp, WkDn,
   WkUp, DayDn, DayUp].Header).  In any of the stat header files you can
   now use %[pos] to retrieve the section of the stat information being
   displayed.  The default stat header files now print the section number and
   the section name so it's easy to see what info is being displayed.

6) New cookie feature (%[SectionName(-1)]).  The SectionName supercookie now
   returns the string "[TOTAL]" if you specify the previously invalid section
   number of -1.

7) New cookie feature (%[AllDn, AllUp, MonthDn, MonthUp, WkDn, WkUp, DayDn,
   DayUp]).  The stat cookies now accepts a section number of -1 which means
   the total across all sections.

8) TCL UTF-8 conversion added.  The ioFTPD interface to TCL now properly
   converts all parameters passed back and forth.  The new UTF-8 encoder
   encodes 8bit characters with the high bit set to two byte UTF-8 sequences.
   The new UTF-8 decoder converts all unicode code points below 256 to their
   8bit equivalent, everything above that to '?', and in the case of
   non-properly formatted input does nothing and simply returns the string
   TCL provided.

   * NOTE:  Special thanks to isteana for a bunch of suggestions, and for
   answering a lot of questions that helped solve the TCL issue with foreign
   character sets.

9) New TCL iputs option (iputs -raw).  If the -raw option is specified the
   UTF-8 converter is NOT called on the string to be printed.  This should
   make printing Ascii art in .NFO files safe to do since it's possible that
   some strings could be unlucky and turn out to be valid UTF-8 sequences and
   converted by mistake.

10) New TCL putlog option (putlog [-sysop|-error|-system|-general] string).
    You can now specify which ioFTPD logfile the string should be logged to.
    Previously everything went to the -general (ioFTPD.log) file and this is
    still the default if no option is specified.

11) New TCL config option (config version).  Return the ioFTPD version string
    as a list in the form "major minor patch".

12) New TCL config option (config counter).  Return the number of site rehash
    commands executed.  This can be used by scripts to determine if the user
    indicated that the settings may have changed.

13) New TCL command (sections).  This command will return information about
    configured sections from the ioFTPD.ini file.  The format is one long
    list with 5 elements for each section in the file:
        <section-name> <credit-section> <stat-section> <share-section> <path>

14) New TCL command (mountpoints [file]).  This command will return a parsed
    form of the currently open mountfile or the specified file.  The format
    is a list of 2 elements of the form:  
        <vfs-path> {<real-path> ...}
    Merged or raided directories are reported together for the indicated
    vfs-path and \'s are pre-converted to / to make manipulation easier.

15) New TCL variable (uniqueid).  The existing io_id variable was too bulky
    to use.  "uniqueid" is constant for each TCL interpreter and indicates
    the count of previous interpreters created + 1.  This is therefore unique
    and small enough to use in logfiles.

    
*** Functionality changes:

16) The %[sharedcredits()] supercookie now looks up the requested STATS
    section and display the credits for that section if stats=credit=shared
    instead of just credit=shared.  If that test fails then credit sharing or
    separate statistics are being kept and it displays the "shared #"
    message instead.  This should now mean the proper information is shown
    in "site uinfo" calls when you use section 0 for all credits, or
    essentially global credits, but want to keep separate statistics for all
    sections of the FTP so you can see who up/downloads where.

17) When adding a user with "site gadduser" the indicated group now becomes
    the user's primary/default group.  Previously if Default.User indicated
    groups to automatically add the user to, the first listed group there
    would become the primary group.  Now the specified group will be used
    as the primary group and any other groups listed in Default.User will
    be added after.


*** Bug Fixes:

18) The user callable "site who" command now properly displays the sanitized
    arguments to hide information from other users.  Previously this was
    computed correctly, but the new field wasn't used.

19) If a FTP_Pre-Command_Events event failed but didn't print an error message
    this would cause problems because no reply to the command was ever sent
    to the user.  Now a generic command failed error message is returned which
    should make the FTP client happy.  This means you can now disable a 
    command by specifying a FTP_Pre-Command_Event that points to a missing
    file.

20) The itcl.txt file didn't document the already existing [group list]
    command.

21) When creating a user via "site adduser" or "site gadduser" the addip
    message lines were showing up before the created user message.  This
    has now been fixed.

22) "Site adduser" would indicate it added new users to the "NoGroup" group
    even if they were really members of real groups because of automatic
    group assignment via Default.User.  Site adduser now displays the actual
    default/primary group for the user.

[Yil]

Think I found a bug in my encoder, so wait until 6.7.1 to see if I fixed it.

[Yil]

Fixed:

ioFTPD-v6.7.1.zip

v6.7.1 Release Notes:

1) Files in \system
   Changed : ioFTPD.[exe,pdb] - Version 6.7.0.0

*** Bug Fixes:

2) Fixed a char vs unsigned char comparison bug in my UTF8 encoder.  doh!

[Yil]

Early feedback from isteana suggests that nxTools likes the changes and some things appear working that previously didn't. ioNinja not so much. Hopefully we can get that fixed as well.

[jim1568]

i tested the new version as soon as i saw this update
the pure ftpd work nothing different with old version about what i had tested, such as created or upload folder with chinese name

sorry my mistake, i forgot to update the tcl
more test will be posted after my carefully testing
thanks for your work

[o_dog]

running into problems with unicode. Especially simplified chinese. Most fxp clients doesn't even seem to support it.

However it seems that some chars can be used without problem now. such as åäö for the swedish people.

In all honesty I don't know how much time i'm willing to spend on this - since i have absolutely no personal interest in this feature in ioNiNJA.

If it's just the crccheck you're after i would suggest you use one of the older zipscripts instead.

[jim1568]

o_dog, thanks for your attention first
maybe you hasnt found the client can support chinese simplified...flashfxp, ftprush, filezilla..all of them can work with chinese characters
i really hope that you can make ioninja work with chinese because i think ioninja is the perfert zs just as i need
but anyway, no matter whether it can support or not, thanks very much for your great job~~

[jim1568]

a test report:
1. pure ioftpd worked the same as before, nothing different or any bug was found till now
2. with scripts
#nxtools
nxtools cannot check the dupe folder with chinese name before, but now can.
nxools cannot nuke(wipe) the directory in the folder named in chinese, that problem still exist

#ioninja
depend the VFS
if the VFS as this:
"D:\ioFTPD\FTP-ROOT-DIR" /
"D:\divx" /电影(chinese name)
"F:\" /divx
ioninja can do crccheck and mediainfo check in all the directory(include 电影 directory) with the rls named in english, but when i created the directory in chinese in root directory, then ioninja didnt do anything...
change VFS as below:
"D:\ioFTPD\FTP-ROOT-DIR" /
"D:\divx" /xvid
"F:\" /divx
if there was any new folder created in chinese under xvid/divx, ioninja didnot work...

anymore test needed please let me know

[Yil]

nxTools working with dupe checking (and searching I presume) means that characters are being properly transferred back and forth between ioFTPD and TCL. If you rebuild the nxTools DB (on a test setup) and search for things does that work? The reason I ask is the name given for newly created directories may just be stuck into the DB unchanged and untested and thus searching for things you just entered into the DB should work.

What's interesting is that once you need to find a real directory on disk things break. I tested some of this locally in my TCL code and it appears to be working OK. I can find the dir with korean characters in it but I needed to process the directory names very carefully because if I ever treated them as a list via any list operation the [], {}, $, and spaces would start causing problems. That just means the script needs to be careful and wrap string in[list] and/or use lindex to retrieve the data and never lrange. This is true of any non-list string with special characters though. On the other hand with directory names that are only a couple of simple (but non-ascii) characters this shouldn't be a problem. Can you double check even the simplest of directory names is having issues?

It just came to me, but it's possible that TCL is doing character conversion based on the computer's OS code page which is exactly the right thing for it to be doing most of the time, but not something we want happening. That may explain why it works for me, but not you. Could you try setting up a test server in vmware or something but using a default english install? If this turns out to be an issue, I'll see about configuring TCL to think it's always running on a simple ascii OS install which means it won't try to convert characters back and forth from the OS calls.

[o_dog]

jim1568: i can't get any of them to support it using simplified chinese.

And all of this is starting to sound like a mistake judging from what YiL just wrote. You have to be way to careful when writing a tcl script if that is the only way it can work with unicode. That Kind of take away any of the positive things using tcl. If it is as YiL says, unicode support will not be added to ioNiNJA atleast. I doubt nxtools support will be added by neoxed either since he seems to have left. I'm just not interested in spending all the time needed to add support for this since i see no reason to why people should be using it in a ftp enviroment and it just gets annoying coding that way. The problem isn't just in the list things it might be in the regexp matching too aswell as some other commands.

YiL: Wouldn't making TCL think it runs on a simple ASCii OS kill some os dependent features within TCL itself?

[jim1568]

Yil:
just like you said, the dupe check and search can work now;
and i have tested the simplest directory with only one chinese word, the answer is the same as before;
the third, ioftpd installed on the english version windows has been test too, but the problem still exist

and i have another question:
why project-zs, php or iob(exe) can support unicode?
sorry, im a laical, but thats confusing...so can you make a brief explanation
thanks~~

[jim1568]

O_dog
it doesnt matter. any way, thanks a lot for your reply
wish ioninja become better and better~~~

[Yil]

I'll take another look and see if I can figure out what's going on. That won't be for another week or so though...