Go Back   FlashFXP Forums > >

Project: FlashFXP Bug Reports Ticket Tools
ID: 999 Category: Server Compatibility Issue
Title: sftp uploads fail when HPN enabled Status: Closed (Fixed / Implemented)
Severity: Medium Version: 5.0.0

Junior Member
gregm
12-04-2014, 11:32 AM
sftp uploads fail when HPN enabled

FlashFXP version: 5.0.0 (build 3791)
Operating System: Windows 7 x64 Service Pack 1
Installation type: All User - Common data folder

FreeBSD 10.0-RELEASE amd64 GENERIC running inside VirtualBox 4.3.18 r96516

Reminiscent of bugid 720

If sshd's "HPN Tuning Improvements" are enabled, uploads larger than a few hundred KB will fail/resume repeatedly and (with sshd LogLevel DEBUG3) auth.log is filled with:

Code:
sftp-server[pid]: error: bad message from 192.168.56.1 local user user
sshd[pid]: channel 0: rcvd too much data 7461, win: 420
sftp-server[pid]: error: bad message from 192.168.56.1 local user user
sshd[pid]: channel 0: rcvd too much data 5760, win: 5406
sftp-server[pid]: error: bad message from 192.168.56.1 local user user
sshd[pid]: channel 0: rcvd too much data 6376, win: 1697
MTUs on the FreeBSD and Windows network interfaces (including VirtualBox Host-Only Network) are all set to 1500, which was the only clever idea I had.

Code:
[22:44:19] [R] Directory Created "/usr/local/www/piwigo/galleries/California_09-2010"
[22:44:19] [R] Changed Directory to: /usr/local/www/piwigo/galleries/California_09-2010/
[22:44:19] [R] Retrieving file list...
[22:44:19] [R] List Complete: 179 bytes in 0.01 seconds (0.2 KB/s)
[22:44:19] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0271.jpg
[22:44:19] Upload: IMG_0271.jpg 363 KB in 0.07 seconds (363.2 KB/s)
[22:44:19] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0272.jpg
[22:44:19] Upload: IMG_0272.jpg 426 KB in 0.05 seconds (426.4 KB/s)
[22:44:19] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0273.jpg
[22:44:19] Upload: IMG_0273.jpg 498 KB in 0.05 seconds (498.7 KB/s)
[22:44:19] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0274.jpg
[22:44:19] Upload: IMG_0274.jpg 487 KB in 0.08 seconds (487.3 KB/s)
[22:44:19] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0275.jpg
[22:44:19] Upload: IMG_0275.jpg 629 KB in 0.11 seconds (629.3 KB/s)
[22:44:19] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0276.jpg
[22:44:20] [R] SFTP Connection closed (No such file)
[22:44:20] [R] SSH Connection closed
[22:44:20] [R] Transfer Failed: IMG_0276.jpg
[22:44:20] [R] Connection lost: 192.168.56.80 (Duration: 7 minutes 58 seconds)
[22:44:20] Transferred 5 Files (2.35 MB) in 0.52 seconds (2.35 MB/s)
[22:44:22] [R] Attempting to Reconnect.
[22:44:22] [R] Connecting to 192.168.56.80 -> IP=192.168.56.80 PORT=sekrit (attempt # 1)
[22:44:22] [R] Connected to 192.168.56.80
[22:44:22] [R] Host key algorithm ssh-RSA, size 2048 bits.
[22:44:22] [R] Fingerprint (MD5): :::sekrit:::
[22:44:22] [R] Key exchange: diffie-hellman-group14-sha1. Session encryption: aes256-ctr, MAC: hmac-sha1, compression: none.
[22:44:22] [R] Auth Type: Keyboard-interactive
[22:44:22] [R] Keyboard Authentication Requested. Prompt (1) "Password for user@freebsd10r:"
[22:44:22] [R] Keyboard Authentication, Password Sent.
[22:44:22] [R] Authentication succeeded
[22:44:22] [R] SSH Connection open
[22:44:22] [R] Connection established with OpenSSH_6.4_hpn13v11 FreeBSD-20131111 (SFTP v3)
[22:44:22] [R] SFTP Connection Ready
[22:44:22] [R] Changed Directory to: /usr/local/www/piwigo/galleries/California_09-2010/
[22:44:22] [R] Retrieving file list...
[22:44:22] [R] List Complete: 803 bytes in 0.01 seconds (0.8 KB/s)
[22:44:22] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0276.jpg
[22:44:22] Upload (Resumed): IMG_0276.jpg 1.40 MB in 0.15 seconds (1.40 MB/s)
[22:44:22] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0277.jpg
[22:44:22] [R] SFTP Connection closed (No such file)
[22:44:22] [R] SSH Connection closed
[22:44:22] [R] Transfer Failed: IMG_0277.jpg
[22:44:22] [R] Connection lost: 192.168.56.80 (Duration: 0.84 seconds)
[22:44:22] Transferred 6 Files (3.75 MB) in 3 seconds (1.11 MB/s)
[22:44:24] [R] Attempting to Reconnect.
[22:44:24] [R] Connecting to 192.168.56.80 -> IP=192.168.56.80 PORT=sekrit (attempt # 1)
[22:44:24] [R] Connected to 192.168.56.80
[22:44:25] [R] Host key algorithm ssh-RSA, size 2048 bits.
[22:44:25] [R] Fingerprint (MD5): :::sekrit:::
[22:44:25] [R] Key exchange: diffie-hellman-group14-sha1. Session encryption: aes256-ctr, MAC: hmac-sha1, compression: none.
[22:44:25] [R] Auth Type: Keyboard-interactive
[22:44:25] [R] Keyboard Authentication Requested. Prompt (1) "Password for user@freebsd10r:"
[22:44:25] [R] Keyboard Authentication, Password Sent.
[22:44:25] [R] Authentication succeeded
[22:44:25] [R] SSH Connection open
[22:44:25] [R] Connection established with OpenSSH_6.4_hpn13v11 FreeBSD-20131111 (SFTP v3)
[22:44:25] [R] SFTP Connection Ready
[22:44:25] [R] Changed Directory to: /usr/local/www/piwigo/galleries/California_09-2010/
[22:44:25] [R] Retrieving file list...
[22:44:25] [R] List Complete: 908 bytes in 0.01 seconds (0.9 KB/s)
[22:44:25] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0277.jpg
[22:44:25] [R] SFTP Connection closed (Success)
[22:44:25] [R] SSH Connection closed
[22:44:25] [R] Transfer Failed: IMG_0277.jpg
[22:44:25] [R] Connection lost: 192.168.56.80 (Duration: 0.68 seconds)
[22:44:25] Transferred 6 Files (3.75 MB) in 6 seconds (629.8 KB/s)
[22:44:27] [R] Attempting to Reconnect.
[22:44:27] [R] Connecting to 192.168.56.80 -> IP=192.168.56.80 PORT=sekrit (attempt # 1)
[22:44:27] [R] Connected to 192.168.56.80
[22:44:27] [R] Host key algorithm ssh-RSA, size 2048 bits.
[22:44:27] [R] Fingerprint (MD5): :::sekrit:::
[22:44:27] [R] Key exchange: diffie-hellman-group14-sha1. Session encryption: aes256-ctr, MAC: hmac-sha1, compression: none.
[22:44:28] [R] Auth Type: Keyboard-interactive
[22:44:28] [R] Keyboard Authentication Requested. Prompt (1) "Password for user@freebsd10r:"
[22:44:28] [R] Keyboard Authentication, Password Sent.
[22:44:28] [R] Authentication succeeded
[22:44:28] [R] SSH Connection open
[22:44:28] [R] Connection established with OpenSSH_6.4_hpn13v11 FreeBSD-20131111 (SFTP v3)
[22:44:28] [R] SFTP Connection Ready
[22:44:28] [R] Changed Directory to: /usr/local/www/piwigo/galleries/California_09-2010/
[22:44:28] [R] Retrieving file list...
[22:44:28] [R] List Complete: 908 bytes in 0.01 seconds (0.9 KB/s)
[22:44:28] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0277.jpg
[22:44:28] [R] SFTP Connection closed (Success)
[22:44:28] [R] SSH Connection closed
[22:44:28] [R] Transfer Failed: IMG_0277.jpg
[22:44:28] [R] Connection lost: 192.168.56.80 (Duration: 0.60 seconds)
[22:44:28] Transferred 6 Files (3.75 MB) in 9 seconds (439.4 KB/s)
[22:44:30] [R] Attempting to Reconnect.
[22:44:30] [R] Connecting to 192.168.56.80 -> IP=192.168.56.80 PORT=sekrit (attempt # 1)
[22:44:30] [R] Connected to 192.168.56.80
[22:44:30] [R] Host key algorithm ssh-RSA, size 2048 bits.
[22:44:30] [R] Fingerprint (MD5): :::sekrit:::
[22:44:30] [R] Key exchange: diffie-hellman-group14-sha1. Session encryption: aes256-ctr, MAC: hmac-sha1, compression: none.
[22:44:30] [R] Auth Type: Keyboard-interactive
[22:44:30] [R] Keyboard Authentication Requested. Prompt (1) "Password for user@freebsd10r:"
[22:44:30] [R] Keyboard Authentication, Password Sent.
[22:44:30] [R] Authentication succeeded
[22:44:30] [R] SSH Connection open
[22:44:30] [R] Connection established with OpenSSH_6.4_hpn13v11 FreeBSD-20131111 (SFTP v3)
[22:44:30] [R] SFTP Connection Ready
[22:44:30] [R] Changed Directory to: /usr/local/www/piwigo/galleries/California_09-2010/
[22:44:30] [R] Retrieving file list...
[22:44:30] [R] List Complete: 908 bytes in 0.01 seconds (0.9 KB/s)
[22:44:30] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0277.jpg
[22:44:31] Upload (Resumed): IMG_0277.jpg 873 KB in 0.11 seconds (873.7 KB/s)
[22:44:31] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0278.jpg
[22:44:31] [R] SFTP Connection closed (No such file)
[22:44:31] [R] SSH Connection closed
[22:44:31] [R] Transfer Failed: IMG_0278.jpg
[22:44:31] [R] Connection lost: 192.168.56.80 (Duration: 0.86 seconds)
[22:44:31] Transferred 7 Files (4.60 MB) in 12 seconds (405.0 KB/s)
[22:44:33] [R] Attempting to Reconnect.
[22:44:33] [R] Connecting to 192.168.56.80 -> IP=192.168.56.80 PORT=sekrit (attempt # 1)
[22:44:33] [R] Connected to 192.168.56.80
[22:44:33] [R] Host key algorithm ssh-RSA, size 2048 bits.
[22:44:33] [R] Fingerprint (MD5): :::sekrit:::
[22:44:33] [R] Key exchange: diffie-hellman-group14-sha1. Session encryption: aes256-ctr, MAC: hmac-sha1, compression: none.
[22:44:33] [R] Auth Type: Keyboard-interactive
[22:44:33] [R] Keyboard Authentication Requested. Prompt (1) "Password for user@freebsd10r:"
[22:44:33] [R] Keyboard Authentication, Password Sent.
[22:44:33] [R] Authentication succeeded
[22:44:33] [R] SSH Connection open
[22:44:33] [R] Connection established with OpenSSH_6.4_hpn13v11 FreeBSD-20131111 (SFTP v3)
[22:44:33] [R] SFTP Connection Ready
[22:44:33] [R] Changed Directory to: /usr/local/www/piwigo/galleries/California_09-2010/
[22:44:33] [R] Retrieving file list...
[22:44:33] [R] List Complete: 1,013 bytes in 0.01 seconds (1.0 KB/s)
[22:44:33] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0278.jpg
[22:44:33] [R] SFTP Connection closed (Success)
[22:44:33] Upload (Resumed): IMG_0278.jpg 992 KB in 0.14 seconds (992.0 KB/s)
[22:44:33] [R] Connection lost: 192.168.56.80 (Duration: 0.76 seconds)
[22:44:33] Transferred 8 Files (5.57 MB) in 14 seconds (395.4 KB/s)
[22:44:33] [R] SSH Connection closed
[22:44:33] [R] Transfer Failed: IMG_0279.jpg
[22:44:35] [R] Attempting to Reconnect.
[22:44:35] [R] Connecting to 192.168.56.80 -> IP=192.168.56.80 PORT=sekrit (attempt # 1)
[22:44:35] [R] Connected to 192.168.56.80
[22:44:36] [R] Host key algorithm ssh-RSA, size 2048 bits.
[22:44:36] [R] Fingerprint (MD5): :::sekrit:::
[22:44:36] [R] Key exchange: diffie-hellman-group14-sha1. Session encryption: aes256-ctr, MAC: hmac-sha1, compression: none.
[22:44:36] [R] Auth Type: Keyboard-interactive
[22:44:36] [R] Keyboard Authentication Requested. Prompt (1) "Password for user@freebsd10r:"
[22:44:36] [R] Keyboard Authentication, Password Sent.
[22:44:36] [R] Authentication succeeded
[22:44:36] [R] SSH Connection open
[22:44:36] [R] Connection established with OpenSSH_6.4_hpn13v11 FreeBSD-20131111 (SFTP v3)
[22:44:36] [R] SFTP Connection Ready
[22:44:36] [R] Changed Directory to: /usr/local/www/piwigo/galleries/California_09-2010/
[22:44:36] [R] Retrieving file list...
[22:44:36] [R] List Complete: 1,014 bytes in 0.01 seconds (1.0 KB/s)
[22:44:36] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0279.jpg
[22:44:36] [R] SFTP Connection closed (No such file)
[22:44:36] [R] SSH Connection closed
[22:44:36] [R] Transfer Failed: IMG_0279.jpg
[22:44:36] [R] Connection lost: 192.168.56.80 (Duration: 0.83 seconds)
[22:44:36] Transferred 8 Files (5.57 MB) in 17 seconds (329.7 KB/s)
[22:44:37] [R] Retry attempt Aborted
With "HPNDisabled yes" in /etc/ssh/sshd_config and sshd restarted:

Code:
[22:52:07] [R] Connecting to 192.168.56.80 -> IP=192.168.56.80 PORT=sekrit
[22:52:07] [R] Connected to 192.168.56.80
[22:52:07] [R] Host key algorithm ssh-RSA, size 2048 bits.
[22:52:07] [R] Fingerprint (MD5): :::sekrit:::
[22:52:07] [R] Key exchange: diffie-hellman-group14-sha1. Session encryption: aes256-ctr, MAC: hmac-sha1, compression: none.
[22:52:08] [R] Auth Type: Keyboard-interactive
[22:52:08] [R] Keyboard Authentication Requested. Prompt (1) "Password for user@freebsd10r:"
[22:52:08] [R] Keyboard Authentication, Password Sent.
[22:52:08] [R] Authentication succeeded
[22:52:08] [R] SSH Connection open
[22:52:08] [R] Connection established with OpenSSH_6.4 FreeBSD-20131111 (SFTP v3)
[22:52:08] [R] SFTP Connection Ready
[22:52:08] [R] Retrieving file list...
[22:52:08] [R] List Complete: 955 bytes in 0.01 seconds (0.9 KB/s)
[22:52:08] [R] Directory Created "/usr/local/www/piwigo/galleries/California_09-2010"
[22:52:08] [R] Changed Directory to: /usr/local/www/piwigo/galleries/California_09-2010/
[22:52:08] [R] Retrieving file list...
[22:52:08] [R] List Complete: 179 bytes in 0.01 seconds (0.2 KB/s)
[22:52:08] [R] Uploading: /usr/local/www/piwigo/galleries/California_09-2010/IMG_0271.jpg
[22:52:08] Upload: IMG_0271.jpg 363 KB in 0.04 seconds (363.2 KB/s)
...no failures
[22:52:24] [R] Changed Directory to: /usr/local/www/piwigo/galleries/
[22:52:24] [R] Retrieving file list...
[22:52:24] [R] List Complete: 879 bytes in 0.02 seconds (0.9 KB/s)
[22:52:24] Transfer queue completed
[22:52:24] Transferred 77 Files (160.01 MB) in 16 seconds (9.78 MB/s)

The same transfer using FileZilla (with HPN enabled on the server) completed without error:

Code:
Status:	Connecting to 192.168.56.80:sekrit...
Response:	fzSftp started
Command:	open "user@192.168.56.80" sekrit
Trace:	Looking up host "192.168.56.80"
Trace:	Connecting to 192.168.56.80 port sekrit
Trace:	Server version: SSH-2.0-OpenSSH_6.4_hpn13v11 FreeBSD-20131111
Trace:	Using SSH protocol version 2
Trace:	We claim version: SSH-2.0-PuTTY_Local:_Oct_16_2014_11:15:32
Trace:	Doing Diffie-Hellman group exchange
Trace:	Doing Diffie-Hellman key exchange with hash SHA-256
Trace:	Host key fingerprint is:
Trace:	ssh-rsa 2048 :::sekrit:::
Trace:	Initialised AES-256 SDCTR client->server encryption
Trace:	Initialised HMAC-SHA-256 client->server MAC algorithm
Trace:	Initialised AES-256 SDCTR server->client encryption
Trace:	Initialised HMAC-SHA-256 server->client MAC algorithm
Trace:	Using keyboard-interactive authentication. inst_len: 0, num_prompts: 1
Command:	Pass: **********
Trace:	Access granted
Trace:	Opened channel for session
Trace:	Started a shell/command
Status:	Connected to 192.168.56.80
...
Status:	Starting upload of sekrit\California_09-2010\IMG_0311.jpg
Command:	put "sekrit\California_09-2010\IMG_0311.jpg" "IMG_0311.jpg"
Status:	local:sekrit\California_09-2010\IMG_0311.jpg => remote:/usr/local/www/piwigo/galleries/California_09-2010/IMG_0311.jpg
Status:	File transfer successful, transferred 1,687,552 bytes in 1 second
...
Junior Member
gregm
12-04-2014, 11:42 AM
Re: sftp uploads fail when HPN enabled

With sshd's HPN enabled during FlashFXP upload, Wireshark says:

Code:
Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)
Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)

(repeats several times, and eventually...) 

Expert Info (Warn/Sequence): TCP transmission window is now completely full
FlashFXP Developer
bigstar
12-04-2014, 12:30 PM
Re: sftp uploads fail when HPN enabled

Thank you for your bug report.

I am in the process of setting up a FreeBSD VM for testing this issue.

I will let you know what I find.
FlashFXP Developer
bigstar
12-05-2014, 04:33 PM
Re: sftp uploads fail when HPN enabled

I setup FreeBSD 10.1 and I was able to reproduce this behavior using the following
  • OpenSSH_6.6.1p1-hpn14v2 FreeBSD-openssh-portable66-6.6.p1_4,1
  • OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420

I was not able to reproduce the problem on a few other *nix flavors (Ubuntu, Fedora, Solaris) so I started checking a few system parameters and I discovered something rather interesting.

Code:
sysctl net.inet.tcp.recvspace
net.inet.tcp.recvspace: 65536
sysctl net.inet.tcp.recvbuf_max
net.inet.tcp.recvbuf_max: 2097152
When using OpenSSH+HPN you want large buffer space to get the maximum performance. It seems to me that when you use HPN these values should probably be increased from the default to much reasonable values.

Procedure for raising network limits under FreeBSD
http://www.psc.edu/index.php/networking/641-tcp-tune

So why does OpenSSH+HPN have this problem with FlashFXP and not other sftp clients?

I suspect that this is due to the fact that FlashFXP is sending using large packets and the other clients use many small packets.

I will continue to investigate this issue so that I can determine exactly what changes need to be made in FlashFXP to avoid this type of compatibility issue.

At this moment the solution from the client side is not 100% clear.
FlashFXP Developer
bigstar
12-09-2014, 05:02 PM
Re: sftp uploads fail when HPN enabled

Please try this updated build

ffxp5.0.0.3798.zip

Download and then unzip the flashfxp.exe into your FlashFXP program folder.
Ticket Tools
Subscribe to this Ticket


Posting Rules
You may not post new tickets

Smilies are On
[IMG] code is On
HTML code is Off


All times are GMT -5. The time now is 05:59 PM.

Parts of this site powered by vBulletin Mods & Addons from DragonByte Technologies Ltd. (Details)