PDA

View Full Version : ioFTPD v6.7.0 *experimental*


Yil
08-18-2008, 07:13 PM
Here it is folks. A version of ioFTPD and TCL that should work for users using code pages that define characters with the high bit set. In particular this should allow Asian users to use TCL scripts.

I've labeled this as an experimental release since early testing shows this to be working, but I had to change a lot of functions so there's a number of things I could have broken. I wrote my own UTF-8 converters since we aren't converting between code pages so much as just properly encoding them and it looks OK but clearly will need some testing on systems with other languages.

I've included my current work on TCL extensions and some other bug fixes, but rather than wait for everything to be finished completely I figured I should get the UTF-8 / TCL changes out the door to help those having problems now.

NOTE: Scripts that maintain state such as nxTools may need to rebuild their dupe databases since entries added prior to this version with Asian characters probably had incorrect entries loaded. So if you are having problems with searching, dupechecking, etc. consider a rescan/rebuild/whatever.

Let me know how it works.

Oh, and check out the new stat features. It's pretty cool to keep stats per section now to see who up/downs where.

Latest Version:
ioFTPD-v6.7.1.zip (http://home.comcast.net/~yil/ioFTPD-v6.7.1.zip)

Yil
08-18-2008, 07:14 PM
v6.7.0 Release Notes:

1) Files in \system
Changed : ioFTPD.[exe,pdb] - Version 6.7.0.0

2) Files in \text\ftp
Changed: [AllDn, AllUp, WkDn, WkUp, MonthDn, MonthUp, AllDn, AllUp].Header

3) Files in \doc
Changed: itcl.txt, Cookies.txt


*** New Features:

4) Modified site command (site stats [dir]). Previously "site stats" would
default to showing information for the current section. Thus the results
would depend upon where you were in the VFS. You could, however, specify
a particular section to display but there was no way to see stats
across multiple sections. Now site stats defaults to showing the TOTAL
stats across all sections which should make it much less painful to keep
individual statistics for each section since you can get the totals now.
If, however, you wish the old behavior of seeing just statistics for the
current section based upon your location in the VFS you can specify the
"dir" option, or for a single specific section use "section #". If you
specify a section of -1 it means you want the total for all sections.

5) New cookie usage (%[pos] in text/[AllDn, AllUp, MonthDn, MonthUp, WkDn,
WkUp, DayDn, DayUp].Header). In any of the stat header files you can
now use %[pos] to retrieve the section of the stat information being
displayed. The default stat header files now print the section number and
the section name so it's easy to see what info is being displayed.

6) New cookie feature (%[SectionName(-1)]). The SectionName supercookie now
returns the string "[TOTAL]" if you specify the previously invalid section
number of -1.

7) New cookie feature (%[AllDn, AllUp, MonthDn, MonthUp, WkDn, WkUp, DayDn,
DayUp]). The stat cookies now accepts a section number of -1 which means
the total across all sections.

8) TCL UTF-8 conversion added. The ioFTPD interface to TCL now properly
converts all parameters passed back and forth. The new UTF-8 encoder
encodes 8bit characters with the high bit set to two byte UTF-8 sequences.
The new UTF-8 decoder converts all unicode code points below 256 to their
8bit equivalent, everything above that to '?', and in the case of
non-properly formatted input does nothing and simply returns the string
TCL provided.

* NOTE: Special thanks to isteana for a bunch of suggestions, and for
answering a lot of questions that helped solve the TCL issue with foreign
character sets.

9) New TCL iputs option (iputs -raw). If the -raw option is specified the
UTF-8 converter is NOT called on the string to be printed. This should
make printing Ascii art in .NFO files safe to do since it's possible that
some strings could be unlucky and turn out to be valid UTF-8 sequences and
converted by mistake.

10) New TCL putlog option (putlog [-sysop|-error|-system|-general] string).
You can now specify which ioFTPD logfile the string should be logged to.
Previously everything went to the -general (ioFTPD.log) file and this is
still the default if no option is specified.

11) New TCL config option (config version). Return the ioFTPD version string
as a list in the form "major minor patch".

12) New TCL config option (config counter). Return the number of site rehash
commands executed. This can be used by scripts to determine if the user
indicated that the settings may have changed.

13) New TCL command (sections). This command will return information about
configured sections from the ioFTPD.ini file. The format is one long
list with 5 elements for each section in the file:
<section-name> <credit-section> <stat-section> <share-section> <path>

14) New TCL command (mountpoints [file]). This command will return a parsed
form of the currently open mountfile or the specified file. The format
is a list of 2 elements of the form:
<vfs-path> {<real-path> ...}
Merged or raided directories are reported together for the indicated
vfs-path and \'s are pre-converted to / to make manipulation easier.

15) New TCL variable (uniqueid). The existing io_id variable was too bulky
to use. "uniqueid" is constant for each TCL interpreter and indicates
the count of previous interpreters created + 1. This is therefore unique
and small enough to use in logfiles.


*** Functionality changes:

16) The %[sharedcredits()] supercookie now looks up the requested STATS
section and display the credits for that section if stats=credit=shared
instead of just credit=shared. If that test fails then credit sharing or
separate statistics are being kept and it displays the "shared #"
message instead. This should now mean the proper information is shown
in "site uinfo" calls when you use section 0 for all credits, or
essentially global credits, but want to keep separate statistics for all
sections of the FTP so you can see who up/downloads where.

17) When adding a user with "site gadduser" the indicated group now becomes
the user's primary/default group. Previously if Default.User indicated
groups to automatically add the user to, the first listed group there
would become the primary group. Now the specified group will be used
as the primary group and any other groups listed in Default.User will
be added after.


*** Bug Fixes:

18) The user callable "site who" command now properly displays the sanitized
arguments to hide information from other users. Previously this was
computed correctly, but the new field wasn't used.

19) If a FTP_Pre-Command_Events event failed but didn't print an error message
this would cause problems because no reply to the command was ever sent
to the user. Now a generic command failed error message is returned which
should make the FTP client happy. This means you can now disable a
command by specifying a FTP_Pre-Command_Event that points to a missing
file.

20) The itcl.txt file didn't document the already existing [group list]
command.

21) When creating a user via "site adduser" or "site gadduser" the addip
message lines were showing up before the created user message. This
has now been fixed.

22) "Site adduser" would indicate it added new users to the "NoGroup" group
even if they were really members of real groups because of automatic
group assignment via Default.User. Site adduser now displays the actual
default/primary group for the user.

Yil
08-18-2008, 09:43 PM
Think I found a bug in my encoder, so wait until 6.7.1 to see if I fixed it.

Yil
08-18-2008, 11:13 PM
Fixed:

ioFTPD-v6.7.1.zip (http://home.comcast.net/~yil/ioFTPD-v6.7.1.zip)


v6.7.1 Release Notes:

1) Files in \system
Changed : ioFTPD.[exe,pdb] - Version 6.7.0.0

*** Bug Fixes:

2) Fixed a char vs unsigned char comparison bug in my UTF8 encoder. doh!

Yil
08-18-2008, 11:16 PM
Early feedback from isteana suggests that nxTools likes the changes and some things appear working that previously didn't. ioNinja not so much. Hopefully we can get that fixed as well.

jim1568
08-19-2008, 08:43 AM
i tested the new version as soon as i saw this update
the pure ftpd work nothing different with old version about what i had tested, such as created or upload folder with chinese name

sorry my mistake, i forgot to update the tcl
more test will be posted after my carefully testing
thanks for your work

o_dog
08-19-2008, 02:25 PM
running into problems with unicode. Especially simplified chinese. Most fxp clients doesn't even seem to support it.

However it seems that some chars can be used without problem now. such as åäö for the swedish people.

In all honesty I don't know how much time i'm willing to spend on this - since i have absolutely no personal interest in this feature in ioNiNJA.

If it's just the crccheck you're after i would suggest you use one of the older zipscripts instead.

jim1568
08-19-2008, 11:51 PM
o_dog, thanks for your attention first
maybe you hasnt found the client can support chinese simplified...flashfxp, ftprush, filezilla..all of them can work with chinese characters
i really hope that you can make ioninja work with chinese because i think ioninja is the perfert zs just as i need
but anyway, no matter whether it can support or not, thanks very much for your great job~~

jim1568
08-20-2008, 02:26 AM
a test report:
1. pure ioftpd worked the same as before, nothing different or any bug was found till now
2. with scripts
#nxtools
nxtools cannot check the dupe folder with chinese name before, but now can.
nxools cannot nuke(wipe) the directory in the folder named in chinese, that problem still exist

#ioninja
depend the VFS
if the VFS as this:
"D:\ioFTPD\FTP-ROOT-DIR" /
"D:\divx" /电影(chinese name)
"F:\" /divx
ioninja can do crccheck and mediainfo check in all the directory(include 电影 directory) with the rls named in english, but when i created the directory in chinese in root directory, then ioninja didnt do anything...
change VFS as below:
"D:\ioFTPD\FTP-ROOT-DIR" /
"D:\divx" /xvid
"F:\" /divx
if there was any new folder created in chinese under xvid/divx, ioninja didnot work...

anymore test needed please let me know

Yil
08-20-2008, 08:19 AM
nxTools working with dupe checking (and searching I presume) means that characters are being properly transferred back and forth between ioFTPD and TCL. If you rebuild the nxTools DB (on a test setup) and search for things does that work? The reason I ask is the name given for newly created directories may just be stuck into the DB unchanged and untested and thus searching for things you just entered into the DB should work.

What's interesting is that once you need to find a real directory on disk things break. I tested some of this locally in my TCL code and it appears to be working OK. I can find the dir with korean characters in it but I needed to process the directory names very carefully because if I ever treated them as a list via any list operation the [], {}, $, and spaces would start causing problems. That just means the script needs to be careful and wrap string in [list] and/or use lindex to retrieve the data and never lrange. This is true of any non-list string with special characters though. On the other hand with directory names that are only a couple of simple (but non-ascii) characters this shouldn't be a problem. Can you double check even the simplest of directory names is having issues?

It just came to me, but it's possible that TCL is doing character conversion based on the computer's OS code page which is exactly the right thing for it to be doing most of the time, but not something we want happening. That may explain why it works for me, but not you. Could you try setting up a test server in vmware or something but using a default english install? If this turns out to be an issue, I'll see about configuring TCL to think it's always running on a simple ascii OS install which means it won't try to convert characters back and forth from the OS calls.

o_dog
08-20-2008, 09:47 AM
jim1568: i can't get any of them to support it using simplified chinese.

And all of this is starting to sound like a mistake judging from what YiL just wrote. You have to be way to careful when writing a tcl script if that is the only way it can work with unicode. That Kind of take away any of the positive things using tcl. If it is as YiL says, unicode support will not be added to ioNiNJA atleast. I doubt nxtools support will be added by neoxed either since he seems to have left. I'm just not interested in spending all the time needed to add support for this since i see no reason to why people should be using it in a ftp enviroment and it just gets annoying coding that way. The problem isn't just in the list things it might be in the regexp matching too aswell as some other commands.

YiL: Wouldn't making TCL think it runs on a simple ASCii OS kill some os dependent features within TCL itself?

jim1568
08-21-2008, 11:22 PM
Yil:
just like you said, the dupe check and search can work now;
and i have tested the simplest directory with only one chinese word, the answer is the same as before;
the third, ioftpd installed on the english version windows has been test too, but the problem still exist

and i have another question:
why project-zs, php or iob(exe) can support unicode?
sorry, im a laical, but thats confusing...so can you make a brief explanation
thanks~~

jim1568
08-21-2008, 11:30 PM
O_dog
it doesnt matter. any way, thanks a lot for your reply
wish ioninja become better and better~~~

Yil
08-22-2008, 08:21 AM
I'll take another look and see if I can figure out what's going on. That won't be for another week or so though...