Which setting to save bandwidth / overhead

[marc3]

Hi,

I'm transfering files between two servers (fxp).

The main reason is because my internet connection has a traffic limitation. Now I found out that my transfer tooks 2.500 kbit/s (1 GB per hour) because of the many ftp commands.

I had this error, too:
https://oss.azurewebsites.net/forum/flashf...html#post82128

Because of unchecking "Request file size/date prior to transfer" I save one command per transfer. Now I ask myself if there are other commands that can be switched off?

regards,
Marc

[bigstar]

From the help file.

Request size/date prior to transfer
When checked, FlashFXP will send additional commands to the server to determine the current size and/or date of the file prior to transferring the file.

This option ensures that the correct file transfer rule is used during evaluation, Just in case the file was modified since the last time it was seen by FlashFXP.

Un-checking this option can decrease the accuracy of the file transfer rules, but can also help speed up file transfers significantly -- As the command doesn't need to be sent for each file.

In addition to the info from the help file.. These commands are sent to accurately apply the correct file transfer rule, without it FlashFXP can unknowingly overwrite a file that was uploaded by another FTP user or skip a file that no longer exists because it was deleted by another user after the directory listing was retrieved.

Now to answer your question, Well maybe....

It depends on the other settings on the transfer rules dialog.

If not already un-checked you could uncheck the Preserve File Date options.. Again.. this decreases the accuracy of the file transfer rules..

[MxxCon]

Originally Posted by marc3

Hi,

I'm transfering files between two servers (fxp).

The main reason is because my internet connection has a traffic limitation. Now I found out that my transfer tooks 2.500 kbit/s (1 GB per hour) because of the many ftp commands

Your math is way wrong.
2.5kilobit/sec * 1 hour = 1.13megabyte.

1.13 megabyte is an insignificant amount. You loading this page, registering, posting and reading used up more data.

[marc3]

Sorry. In Germany we use a dot as thousands separator

It is 2500 kbit/s

2500 kbit * 60 * 60 = 9000000 kbit / 8 = 1125000 KB / 1024 = 1099 MB / 1024 = 1.07 GB (english notation )

[marc3]

Originally Posted by bigstar

From the help file.
In addition to the info from the help file.. These commands are sent to accurately apply the correct file transfer rule, without it FlashFXP can unknowingly overwrite a file that was uploaded by another FTP user or skip a file that no longer exists because it was deleted by another user after the directory listing was retrieved.

Now to answer your question, Well maybe....

It depends on the other settings on the transfer rules dialog.

If not already un-checked you could uncheck the Preserve File Date options.. Again.. this decreases the accuracy of the file transfer rules..

I don't have other rules set. So it is default except of the one. Thank you for the hint but at this time I don't need the date because the other server is empty

Its a pity as I don't really save bandwith and fxp is not very fast as well. It would be faster to download the complete ftp files and upload them afterwards.

[bigstar]

I don't think the commands being sent generate that much bandwidth, the commands are either SIZE <filename> or MLST <filename>, this is a few bytes perhaps an average of 30 bytes per command + reply, for 100 files that's only a few kilobytes.

Sending the additional command could slow things down when there is extremely high latency between you and the FTP server, for instance if it takes 500ms for the server to receive the command and send a reply then over the course of sending say 100 files there is a total delay of 25 seconds.

[marc3]

Look on the traffic statistic in the right bottom corner:


It copies very much small files (vbulletin attachments).

You can see the logs as well. Doesn't really look for two commands only. I counted 13 log entries per transfer.

[MxxCon]

Marc3, you are concentrating on a wrong thing.
Commands themselves to not take up any bandwidth. It is just plain text that you see in that status log.
What could be happening is some device(maybe your router) is messing with your FXP transfer and is actually sending it through your connection instead of directly server to server. If you can, try to enable encryption on your FTP server. That should make it impossible for any device to monitor FTP commands and consequently mess with your FTP commands and ensure that FXP transfers actually go server to server.

[marc3]

Hmm. Ok. Because of this thread I thought this can't be possible:
https://oss.azurewebsites.net/forum/flashf...sfers-fxp.html

If the server is not configured to allow site to site transfers the FTP server might return an error message (the error message can vary from server to server) or the server might disconnect you.

Because of that I thought fxp works or won't and nothing between.

What means activating encryption? You mean FTPS or SFTP?!

My new Server supports both. The old server supports at the moment FTP only but I can activate FTPS.

I testet with SFTP at the new server and without encryption at the old one. Now I can see in the status line for every file "receiving" and "sending".

If I choose at both servers FTP it shows "transfering". The same is if the new server is connected through FTPS.

The attached screenshot shows the SFTP Test with "receiving" and "sending". It has more peaks but the average used bandwidth looks the same. So I think you are correct.

[bigstar]

I have to agree with MxxCon, I suspect that your router or internet provider is causing your site to site transfers to be redirected through your local internet and then back out.

When transferring with SFTP the file is first downloaded to your local machine and then uploaded to the destination server.

Only FTP allows direct site to site transfers and only if none of the routers/firewalls in-between interfere with it.

If you have SSH access on the servers you can create a tar.gz archive of all the files on the original and wget it on the new server.

[marc3]

I don't have SSH access on both. That is the reason why I want to use FXP (and the reason why I bought this software).

As I'm not able to use DynDNS, VPN or similar services I think my internet provider causes the problem.

But there is a little bit hope. I will make a test tommorrow. If I use 3G instead of 4G I'm able to change the APN to get a public ip. Maybe this solves the problem.

Any other ideas (regarding bigstars explanation I don't think the encryption idea of MxxCon will work)?

[MxxCon]

SFTP will not work, but FTPS should work.
I suspect that it might be your fritz!box that could be causing these problems...Unless you have some other device connected as well.

[marc3]

Yes the fritz!box is behind a second router. Now I've tested the first one that builds the internet connection. There does not start anything:

Serverfehler, Abgebrochen
[R] Erneutes verbinden....
[R] Verbindung herstellen zu army-outdoor-shop.de -> DNS=dedi7.your-server.de IP=78.47.8.109 PORT=21 (Versuch # 1)
[R] Verbunden mit army-outdoor-shop.de
[R] 220 FTP Server 1.0
[R] USER armysc
[R] 331 Password required for armysc
[R] PASS (hidden)
[R] 230 User armysc logged in
[R] SYST
[R] 215 UNIX Type: L8
[R] FEAT
[R] 211-Features:
[R] LANG en-US.UTF-8*;en-US
[R] MDTM
[R] MFMT
[R] TVFS
[R] AUTH TLS
[R] UTF8
[R] MFF modify;UNIX.group;UNIX.mode;
[R] MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX .mode*;UNIX.owner*;
[R] PBSZ
[R] PROT
[R] REST STREAM
[R] SIZE
[R] 211 End
[R] OPTS UTF8 ON
[R] 200 UTF8 set to on
[R] PWD
[R] 257 "/" is the current directory
[R] CWD /public_html/admin/fpdf/doc
[R] 250 CWD command successful
[R] PWD
[R] 257 "/public_html/admin/fpdf/doc" is the current directory
[R] Auf PORT: 60204 warten, Verbindung erwarten.
[R] PORT 192,168,1,60,235,44
[R] 200 PORT command successful
[R] MLSD
[R] 150 Opening ASCII mode data connection for MLSD
[R] 226 Transfer complete
[R] Verzeichnis geladen: 114 bytes in 0,23 Sekunde (0,1 KB/s)
[R] TYPE A
[R] 200 Type set to A
[L] PASV
[L] 227 Entering Passive Mode (5,9,64,77,166,212).
[R] PORT 5,9,64,77,166,212
[R] Ãœbertragung fehlgeschlagen: addfont.htm
[R] Verbindung verloren: army-outdoor-shop.de (Dauer: 1 Minute 0 Sekunde / Leerlauf: 1 Minute 0 Sekunde)
Serverfehler, Abgebrochen

FXP not supported? Or still the proxy thing of my internet provider?

EDIT: Next test with FTPS on both sides: Server-to-Server does not work error, trying alternative mode without success (same problem as above). This is the same on both routers.

[MxxCon]

Did you edit this session log? It doesn't seem to be showing all the commands..

Post session log from FTPS connections.