Yil,can you make ioftpd support chinese

jim1568 · 08-12-2008, 09:18 PM

maybe the title is not correct...
can you read this thread:
https://oss.azurewebsites.net/forum/showthread.php?p=72155
im not sure whether ioftpd cannot support or ioninja, but i hope that this problem canbe solved
thanks for your great job~~

Yil · 08-13-2008, 03:54 AM

The short answer is.... no, but it does partially work.

I've said in the past that enabling real Unicode support in ioFTPD would be REALLY hard to do. I'd almost be better off writing a new server kind of hard. On the other hand, FTP clients use UTF-8 encoded filenames which ioFTPD doesn't understand and thus leaves untouched so things just work as far as the FTP client is concerned.

Here's some tests that you could perform for me... Start up your favorite FTP client and look at a local directory with Chinese characters in it and make sure it shows the names correctly. Now, start up a plain ioFTPD (no scripts at all) and transfer some files, then look at the directory. Do the files look the same when viewed in the FTP directory listing? They probably should sort the same way in the FTP client as well, but can't be positive of that under all cases.

I suspect from what I can tell that the above test will actually work. Here's where it gets interesting though. If you view the directory in windows where the FTP uploaded the files you will NOT see recognizable directory names. The names will look really odd because they are actually the UTF-8 encoded names. When those names are fed back to the FTP client and it decodes them you get back the original correct Chinese name.

So, from a plain ioFTPD perspective the FTP will support the UTF-8 naming conventions, but it isn't Unicode aware so the local directory names are not converted to Unicode so you can see them properly. This also means that any real Unicode directory name you copy over to an exported directory in the VFS is essentially inaccessible from the server because it's not smart enough to handle it, but if you upload it locally instead of copying it will be visible.

Assuming I'm correct in how things are working so far, we now run into TCL script problems. TCL actually uses UTF-8 internally, and thus is Unicode aware but that's actually a liability at the moment since ioFTPD doesn't. If it were to assume UTF-8 names were being sent to it, but treat them as plain characters I think it should work. I don't really have a way to test stuff though, and my early tries showed some problems so I don't know what's going on...

Here's another test you can do for me though. Create a .sfv file for a bunch of Chinese named files and then upload the .sfv and the files to an ioFTPD without any scripts installed. Now look at the directory where the files were uploaded, does the .sfv file internally encode the names the same as you see in the directory? I think .sfv files should use UTF-8 to encode the names but I can't be sure. If it does, then in theory scripts should be able to manage...

So please run those tests and let me know what you find out.

jim1568 · 08-13-2008, 09:06 AM

thanks for your voluminous reply~~

the first question
the pure ioftpd can handle folder and file name in chinese correctly. no matter the folder create in server or client or upload by ftp client, all of these can be performed well
and at present only project-zs, Php and iob(exe) can work with chinese characters correctly
all of the scripts written in TCL cannot work well, such as ioninja, iosfv, iob(TCL)
the last question
ioftpd worked well too...both on server and at the client can see the correct name....
anymore test needed just tell me~~ill let it done

Yil · 08-14-2008, 12:14 PM

TCL quick update:

First off, I thought FTP clients were using UTF-8 encoded names since that is provided for in the standard (FEAT UTF8). ioFTPD doesn't support this, but I figured clients were sending it anyway because they didn't have any other means to encode the characters...

What I didn't realize is that clients weren't doing anything with the name at all, but that the clients were running windows using different code pages so they could see different characters...

So now that I understood what was going on, things started to make more sense.

The ioFTPD problem with TCL is that it is passing back and forth strings that are supposed to valid UTF-8 encoded strings and they aren't. This of course confuses it...

There are a bunch of functions to convert strings in the TCL library and I'm not sure what the best way to handle it is yet, but it should be fixable.

Alternatively, TCL might be able to use the [encoding] feature to re-interpret the string. This is what my plan was earlier but I kept getting errors because I thought it was valid UTF-8 filenames that the FTP client was sending...

So, the good news is, I think I understand what is going on which means there should be a work around/fix provided you are using code pages.

jim1568 · 08-14-2008, 09:45 PM

yeath, totally right about the code pages
in my memory there was one time that my classmate saw the mess code with his flashfxp connected to my ioftpd server, but when he updated the language package the problem was fixed.
another test about VFS
a pure ioftpd using chinese characters in VFS can work correctly, whereas using iosfv the log always said" no such file or directory"and blablabla...