ioFTPD v7.0.3 Released

[dr.owned]

Yil, could you please check out this script i wrote?

PHP Code:

putlog -general "Script called"

set remount [string range [lindex $ioArgs 0] 1 end-1]
if {[string index $remount end] == "/"} {set remount [string range $remount 0 end-1]}
if {[string index $remount 0] == "/"} {set remount [string range $remount 1 end]}
set remount "/_$remount"

set path [resolve pwd $remount]

if {[file isfile $path] == 1} {
    ioVirtual AddFile 0 1 1 "anonymous" "nobody" 0755 "||RESOLVED||" "$remount/[file tail $path]"
    return 1
}

set dirs [glob -nocomplain -type d "$path/*"]
if { [llength $dirs] > 0 } {
    foreach d $dirs {
    putlog -general $d
    ioVirtual AddSubDir [file tail $d]
    }
}

set files [glob -nocomplain -type f "$path/*"]
if { [llength $files] > 0 } {
    foreach f $files {
    ioVirtual AddFile [file size $f] 1 1 "anonymous" "nobody" 0755 [file tail $f] ""
    }
}

return 1 


here i'm just trying to popullate virtual directory with contents of another mounted directory ($remount). as you suggested, i use empty string "" as a link and file listing in web browser looks great ^) but when i try to retr a file ioftpd says "modifying virtual directories is not allowed" and does not call my script to ||resolve|| the file (there is no "Script called" line in the log).

and why stor to virtual dir is not allowed? following the same idea of ||resolve||-ing filenames, a script should just point out a vfs destination for server and the rest is just like a normal upload..

[Yil]

"modifying virtual directories is not allowed" - I'll have to look into that. If you CWD into the directory can you download the virtual files by using a simple name?

Modifying virtual directories just seemed like a bad idea. I assumed that people would sort, etc real directories and return links to them and thus everything would pretty much work normally from that point on. I can look into letting the virtual resolver attempt to get a path for pure virtual files and see if the script returns a RESOLVED style link and then use that...

[dr.owned]

hmm.. not sure i understood "by simple name" right..

here

PHP Code:

ioVirtual AddFile [file size $f] 1 1 "anonymous" "nobody" 0755 [file tail $f] "" 


if i put "$remount/[file tail $f]" instead of empty string "" for the last parameter, files are downloaded ok, but web-listing is broken. and if i put an empty string there i get "modifying virtual directories is not allowed" whether i CWD into directory and say RETR file.ext or just say RETR /virtual/dir/file.ext.

hope you get this figured out soon because there is an idea in my mind that depends heavily on ioftpds virtual dirs

as of actually "modifying" virtual dirs - i'd like to be able just to upload files at the moment, and if you think it's not worth the effort then nevermind

[paja1]

Yil: About 'caching' problem with browsing Virtual Folders.

Ok, i founded out what is happening...
It seems like the code is ok, it just that in first call is the cookie/variable $pwd outdated... means from last call.

CWD /INDEX/
250 CWD command successful.
PWD
257 "/INDEX" is current directory.
NLST *.*

$pwd = "/INDEX/"

CWD /INDEX/Foo
250 CWD command successful.
PWD
257 "/INDEX/Foo" is current directory.
NLST *.*

$pwd = "/INDEX/"

NLST *.*

$pwd = "/INDEX/Foo" (correct)

Are you updating $pwd AFTER calling iTCL script or what?

I'm doing that wrong or there is something wrong in the code?
I'm convinced this is the only problem so far!

Btw: in first call (first virtual folder) it's working well.


THANKS!!!!!!
Pavel


******* UPDATE *********************************

After replacing $pwd with $ioArgs[0] everything works quite well!

[dr.owned]

to be more specific, here is the detailed log of different browsers talking to ioftpd (script included).
http://d.padfly.com/ioftpd?download

the following link is given - ftp://127.0.0.1/public/10.jpg
/public - virtual directory
/_public - dir mount whose contents are duped in /public
as a result, opera succeeds and iexplore fails

hope this'll help

[dr.owned]

oh man.. ignore my last post i found an error in my script.
so file retrieval using full path works ok.
and the only problem remaining is weblinks...

also during debugging of my script i noticed the following two things.

- it would be nice to know which command caused server to call the script, to be able to respond accordingly. for now i imitate such feature by having specific tcl scripts bound to all possible pre- and post- handlers where i save and clear the last command respectively. it works, but still..

- and since scripts will be able to find out the command they were called by, why not make the same script handle the processing of all commands which somehow involve its virtual directory. on the one hand it eases VDs implementation by leaving all handling to programmer, but on the other it's a performance loss. so maybe there is a point in making it configurable (like ForceVirtualDirectoryScriptCalls = TRUE/FALSE) so those who don't require speed could benefit from extended functionality.

[dr.owned]

i'm sorry for this onslaught of posts

1. i patched ioftpd.exe not to return symlinks and voila, weblinks look great and thay work

2. i also experimented with uploads. and i actually managed to upload files to virtual directory! ) as well as all other operations like deleting and renaming. i don't know if you planned it from the beginning, Yil, but these are certainly great news thanks for being around and supporting ioftpd

[paja1]

Originally Posted by Yil

*** Bug Fixes:

2) Fixed a bug in the EXEC module that improperly manipulated the input buffer
and would crash the server when the undocumented redirect commands such as
!vfs:add were used by scripts such as ioZS. Redirect commands allow EXEC
scripts to ask the server to perform operations. These command lines are
stripped from the output before it's sent to the user. For the record the
full list of redirect commands are: !putlog, !vfs:add, !vfs:chattr,
!change, !unlock, !newlines, !prefix, !buffer, and !detach.

Hi Yil,
unfortunately.. this is still not working.. please check my crashdumps on your ftp.
Thanks!

[Yil]

dr.owned: Can you give me some examples of why knowing the command being run makes a difference as to what the script should return? It's possible to record the command being executed and pass that to the script whenever the virtual directory script is being executed and perhaps even when a pure virtual directory is being accessed from the cache. It would be difficult to entirely replace virtual directories with scripts because all commands would have to resolve their targets normally first (to support things like list -al /search/foo from /) and then handle things differently depending on it being virtual or not. That's a lot of commands to have to modify unless I found some cool way to do it I can't think of right now.

My initial implementation of virtual directories caches the last directory listing for each user per virtual mountpoint. I commented somewhere that once I could get some usage data that a single directory might be too small if people were using trees instead of say a simple /latest dir. I'm open to say caching 3-5 dirs per mountpoint. Let me know if you think that would dramatically change the number of times the script would be called in your implementations and if that would be desirable.

I also liked the ":" link prefix idea I mentioned before. That would allow you to symbolically link a directory but to hide the link in listings. That should dramatically reduce the script load I assume because the server would internally resolve things instead of having you return RESOLVED links from the script. If I can find/solve paja's problem I'll perhaps slip in this "new" feature into the next bugfix release sometime soon.

[paja1]

Yil,

I've found potential error... but maybe I'm just wrong.

Event OnUploadComplete returns invalid CRC32 value... i'm trying to get rid of ioZS as it's making ioFTPD 7.0.2 crashing... so i started with this value.. but as I already mentioned, this value is wrong

Compared to iTLC crc32 or TotalCommander, etc...

Should be this value correct crc32 of currently upoloaded file or something else?

Thanks!

Pavel

[paja1]

Yil,

do you think that durring VFS preloading the server can respond with message that server is initializing or starting up...? I have quite a lot of virtual mounts oven on network drives.. and it takes about 40 sec to startup/preload server.

Just idea... not requested at all.... but will be better to know what happening.

Or allow to (M) masters to login anyways.

What do you think?

[Yil]

paja1: I'm considering a requested feature of allowing the server to startup as if the site was closed. A new option to the site close command would make it carry over until it was removed even if restarted. Probably just via a simple file being created in the system dir with the reason. Assuming I do this I don't see why I can't use similar login logic and let the server startup normally and respond with a "server coming up" message as the closed message (provided you haven't set a real one) while preloading finishes. Perhaps even applying a different CloseExempt parameter. I'm also looking at ways to prevent LIST/MLST from timing out during 2+ minute listings by just spitting out 200- keep alive messages although it's a bit tricky to do. The real problem is STAT/NLST is very specific about what you can return so I'm looking into faking out the "." entry or something if needed. Preloading makes logins appear snappier when connecting to a newly started server but were really designed to get around 2+ minute initial directory listings which clients timeout...

Event OnUploadComplete returns invalid CRC32 value? Provided you haven't disabled computing it in the .ini file I think it should be working fine. I remember ioSFV used it for a while and I'm pretty sure ioNinja uses it now. I added support somewhere like v6.3 or so that it would compute the CRC32 value over the whole file and not just the uploaded section as it did before that which made a difference for resumed files. The builtin XCRC (XCRC32?) FTP command is special cased though. It will return the CRC32 value for the uploaded file the first time it is called immediately after uploading (so clients can check if the file transferred OK), however this might not be the same as the file on disk if the zipscript modified (i.e. added/stripped .nfo files) so you can call it again and it will give you the disk based result on all further calls. If the OnUploadComplete CRC32 value when called is different than that called by the [crc32] command then that indicates a problem. Is it reproducible? I can't believe ioNinja would work without it being correct at least most of the time... Nothing weird like attempting to upload a single file via multiple connections going on?

[paja1]

Originally Posted by Yil

paja1: I'm considering a requested feature of allowing the server to startup as if the site was closed. A new option to the site close command would make it carry over until it was removed even if restarted. Probably just via a simple file being created in the system dir with the reason. Assuming I do this I don't see why I can't use similar login logic and let the server startup normally and respond with a "server coming up" message as the closed message (provided you haven't set a real one) while preloading finishes. Perhaps even applying a different CloseExempt parameter. I'm also looking at ways to prevent LIST/MLST from timing out during 2+ minute listings by just spitting out 200- keep alive messages although it's a bit tricky to do. The real problem is STAT/NLST is very specific about what you can return so I'm looking into faking out the "." entry or something if needed. Preloading makes logins appear snappier when connecting to a newly started server but were really designed to get around 2+ minute initial directory listings which clients timeout...

Thanks.. sounds like a good solution... even i can understand that implementation of 'progress' messages can be quite tricky.. and i think (maybe) not necesary.

Originally Posted by Yil

Event OnUploadComplete returns invalid CRC32 value? Provided you haven't disabled computing it in the .ini file I think it should be working fine. I remember ioSFV used it for a while and I'm pretty sure ioNinja uses it now. I added support somewhere like v6.3 or so that it would compute the CRC32 value over the whole file and not just the uploaded section as it did before that which made a difference for resumed files. The builtin XCRC (XCRC32?) FTP command is special cased though. It will return the CRC32 value for the uploaded file the first time it is called immediately after uploading (so clients can check if the file transferred OK), however this might not be the same as the file on disk if the zipscript modified (i.e. added/stripped .nfo files) so you can call it again and it will give you the disk based result on all further calls. If the OnUploadComplete CRC32 value when called is different than that called by the [crc32] command then that indicates a problem. Is it reproducible? I can't believe ioNinja would work without it being correct at least most of the time... Nothing weird like attempting to upload a single file via multiple connections going on?

Unfortunately I'm not able to reproduce it any more... and seems like it works well atm actually.
I'd experienced this on text file (*.bat) but works fine with this one as well.. dunno why it returns wrong values at that point... but i restarted ioFTPD serveral time between, so anything could happen.

Thanks for you quite good answer and explanation.

Pavel

[dr.owned]

Originally Posted by Yil

dr.owned: Can you give me some examples of why knowing the command being run makes a difference as to what the script should return?

that's really easy. say i'm renaming or deleting a subdir in a virtual directory. the sciprt is only aware of that he should process "/virtual/subdir", so it looks at the specified path, determines it's a directory and returns its listing. but instead it should have returned ||resolve||-d link to it. there a no means by which the script should know what to do (only with a workaround i mentioned before).

or another example - to fake responses to commands like MDTM, RETR... or reversively, to make responses more ftp-client-understandable. because "modifying virtual directories is not allowed" isn't exactly what ftp clients expect when they commit something they can't.

and when RETR-ing, it would be nice to know whether it starts from byte 0 or a REST 123123 command was issued earlier and the requested file is resumed. same goes to pre-retr event.

Originally Posted by Yil

Let me know if you think that would dramatically change the number of times the script would be called in your implementations and if that would be desirable.

in my case the whole point is that contents of virtual directory is about never the same, it is changed constantly so caching isn't helping me at all. so it seems like a waste to enable caching (=> memory consuming) functions for directory that doesn't require it.

Originally Posted by Yil

I also liked the ":" link prefix idea I mentioned before.

it seems like a nice idea. but to me actually NOT adding ":" should return usual name, and adding it return a symlink

p.s. also check the situation when a declared virtual directory path already exist and it is a non-empty mountpoint. ioftpd crashes while only warning should have been issued.

[Yil]

dr.owned: Interesting. First comment I'd make is that most of the problems you appear to be having are caused by trying to fake out pure virtual files/dirs instead of letting the server know they are links to real items. MTDM, RETR w/RESTORE, etc all would work correctly if you weren't trying to hide the link from the server because it's outputting in a way that some web clients don't like. If I fix that with the ":" option for you all that should become cleaner while still outputting as you like. For the moment try using links and see how much of a difference that makes and let me know what still doesn't work right. Also, see if there is a difference between dir and file handling in practice.

Having said that you bring up an interesting point with renaming. If the original directory is a real directory then once you realize that the script should return a resolved link to it. I'm not sure what the target should do however. That's a pretty special case but just returning an error probably works since the item shouldn't exist anyway. It gets even weirder if you try to rename purely virtual items since even if I pass the current command in as an option that wouldn't give you enough data to completely understand what is going on. I can however pass as an extra argument the stored source name if I have one. Renaming is already handled as one of a very few special case multiple command sequences with the other being downloading/uploading. I'm not convinced yet that with using resolved links and by letting the server know about existing items that normal file up/down won't work correctly without all that extra stuff.

The resolver when running does have a couple of flags that might be interesting to pass to the script. The most important are the EXIST and DIRECTORY flags which indicate the item must actually exist (i.e. downloading needs to actually find a file, or the dir must be already created, etc) and/or that the target must be a directory and not a file. I can certainly arrange to have those passed along if it would make a difference but right now it just looks at the results of the script and applies the flags to that.