Distributed PASV: PASV with PRE Transfer Command

[mog]

Hi.

I'm the author of drftpd, which needs a minor protocol extension to be able to use of PASV.

I've written a comprehensive specification on the command here, i hope you find all your questions answered in the spec
http://drftpd.mog.se/wiki.phtml?title=Distributed_PASV

Currently drftpd has very few users, but i'm in the process of setting up a larget cluster. Of course, PASV isn't required but is certainly nice for the FlashFXP users behind NAT/FXPing to another PASV-only site.

The specification is intended for easy implementation and the only thing needed is to send a "PRET <transfer-command>" command before PASV. Any response from the command can safely be ignored. You might want to disable sending PRET if the response code is 202, 500 or 502 though.

----
mog
DrFTPD author

[darkone]

You should also create extension for port command (As per drafts, daemon is supposed to connect() to remote as soon as it receives PORT command)

However, instead of adding new commands to already bloated ftp-protocol, you could try to persuade client authors to modify code that stores/retrieves files. From:

PASV
STOR <directory>/<filename>
PASV
STOR <directory>/<filename2>


to:

CWD <directory>
PASV
STOR <filename>
PASV
STOR <filename2>

This lessens burden from ftp-server (no need to parse path), reduces amount of obsolete data transfered & is easier to implement than the extension you suggested. If client sends directory in transfer request, reject it with suitable errorcode.

[mog]

This specification is an easy fix (or hack) to support distributed transfers.

Rewriting all transfer commands is another option but it places much more burden on ftp client authors wanting to implement it. Possibly preventing them from implementing it too soon.

I wrote another specification for this but another client author suggested it was too much work to get everyone to implement it and this would be simpler.

http://drftpd.mog.se/wiki.phtml?titl...nsfer_commands
It has IPv6 support, and better NAT support for PORT transfer mode.

[Shark]

I think we leave this is the hands of Darkone, he is the jedi master of FTPD

[mog]

You can already use both absolute or relative urls, i don't see the problem dark0n3 is describing.

And i have implemented path-parsing in a very nice way (currentDirectory.lookupFile(path) aka LinkedRemoteFile.lookupFile(path)) so it's not a burden at all for me.


I haven't seen any problem all with not connecting until the actual transfer command is sent so i don't see the need for an extension for it. If any clients have problems with it they should change it so they don't accept the connection until after they have sent the transfer command (LIST/RETR/STOR)

[darkone]

What I meant is that client should only use filename when issuing stor/retr/appe/size/list/... :

It might be easier to explain this with examples:

"STOR file.tgz" would be allowed, while "STOR /directory/file.tgz" wouldn't be.

Advantages:

1) This does not require new commands, and some clients are already compatible with this.
2) Modification to client is minimal (no new settings etc)
3) Server may free its' resources immeaditely from ill-behaving client
4) Server can operate with clients, that are using incompatible transfer commands, by doing:

CLNT> STOR /Directory/File.tgz
SRVR> 550 '/Directory/File.tgz': Filename contains path, please re-upload file to begin transfer.
^^ Server caches working directory
CLNT> PASV
SRVR> ....
^^ Server uses cached working directory to determinate which ip to return
CLNT> STOR /Directory/File.tgz
SRVR> Opening BINARY mode data connection for File.tgz.

[mog]

Ah, i think i see what you mean. You're assuming i use only one slave per directoroy. That is not the case however.

With DrFTPD, LIST transfers always go to the master.

And all files are striped on different slaves(/nodes). I put a flowchart on http://drftpd.mog.se that shows this. One slave per file.

[darkone]

Ahh, then it isn't possible - you could still use the method i described, for those clients that do not have direct support for you extension (every second transfer command would succeed)

Btw try posting suggestion to: http://www.ietf.org/html.charters/ftpext-charter.html.

[MxxCon]

why post suggestion?
all mog is asking for is support for optional "PRET" command.
it's part of FEAT specs, so as long as ftpd and client understand them there is no problem.

[darkone]

Posting suggestion on ietf mailing lists ensures, that noone else does/has done similar implementation using different arguments.

It seems that PRET command is already defined in:
http://www.globus.org/research/paper...ftpSpec02.doc, and differs from what suggested here (I'm not sure, if it has been actually implemented to daemon)

[MxxCon]

that document talk about PRET which is implemented almost exactly like mog suggestion.
for specific implementation ftp client could use OPTS command.

Introduce a new command called the Pre Transfer (PRET) command (see further description below in the section on new commands under consideration). This command would allow arbitrary state information to be set for a single transfer before any other command were issued. This option has the advantage that state information other than the filename could be provided. File size is one option that comes to mind. However, this system allows information not anticipated today to be made available.
...
PRET PRETransfer Command
FTP has no way of specifying critical information (such as filename) before the start of a transfer. The introduction of PRET (pre-transfer) is designed to fix this inadequacy.

The PRET commands sets state for the *next* transfer only. It does not set global state, and is tightly coupled to a single STOR/RETR command.


PRET <RETR/STOR> [attribute list]

arguments:
----------
<RETR/STOR>:
The first argument MUST be the string "RETR" or the string "STOR". In order to make key decisions the server must know what type of transfer this command is in regard to.

[attribute list]:
The attribute list contians comma seperated attribute value pairs. The attribute is defined by the specific server implementation. This allows it to be modified as new requirements come up.

[mog]

Oh, cool, i didn't know anyone else had done the same thing and with the same name even. When writing the spec i googled for FTP and PRET but all i found was some french language.

I have read GridFTP specs, although i'm not sure i think they have switched to

OPTS <command> <attr>

Their latest GridFTP draft mentions nothing of PRET http://www.globus.org/datagrid/gridftp.html

Anyway if this is of concern the syntax

OPTS PASV filename=<filename>

could be used.


I am not sure how likely it is for my spec to be accepted as an IETF standard, i consider it a quick and dirty protocol hack to support distributed PASV. Making it as easy as possible for client writers to add support for it. A clean implementation would involve using the EPSV EPRT commands and integrating them into any transfer command, i wrote a spec for that before writing the much simpler PRET spec http://drftpd.mog.se/wiki.phtml?titl...nsfer_commands

It would involve much more work for client to support both this and the older command model, whereas with PRET it's trivial fix. There are already 4 FTP clients supporting PRET: lftp, pftp, SmartFTP and Staff-FTP, see http://drftpd.mog.se for an up to date list.

/mog

[mog]

Also, isn't GridFTP a new and improved protocol inspired by FTP? Would normal clients work on GridFTP? Afaik, they don't. Making it a non-issue. However i agree that it's not good with different specifications to definine the same command in different ways.