some serious unreliability

okay i need to file some weird behavior.
im FXPing site to site. source is a glftpd2, so that cant be a problem, its been rock solid always and the target is a raidenftpd server on my PC. source site is using TLS1.2
now what happens is this:
I queued appr. 1000 dirs and hit transfer. when I came back there is a message waiting asking to overwrite. okay its my fault, didnt set the transfer rules. so I go and quickly set them. of course by this time both servers have been logged off due to inactivity, cause I was away for long.
and here is the thing. once the rules were set I hit transfer to resume the work, it reconnects to the glftpd2 and my raidenftpd and then wham: error 435 (dont have the exact message, cause I accidentally flushed the log, but the message involved TLS) on the glftpd's side, when the first file wants to transfer. only thing I could do to resolve this is to exit the FlashFXP client and run it again. now this I never saw happening with 4.0. hitting the disconnect button and reconnecting to the sites does not flush the problem. I have to restart the FLASHFXP client each time to get it working again, so im pretty sure its the client.
second phenomenon: the job was finished. some files got transferred, some got caught in dupechecker and as normal marked with a red X, remained in the queue. okay Im gonna just make sure everything went like it supposed to, so I requeue some randomly selected files with CTRL+R 'reset selected' and hit transfer again. SURPRISE!!!! some of those files did transfer second time around.
now this could have been the dupechecker or FLASHFXP, the logs werent long enough, couldnt track back what files and how they went wrong the first time, but again, this is something I never experienced with 4.0. I asked the siteop and he said the FXP connection was up and running all the time, so this didnt happen due to my internet connection breaking. this will be harder to reproduce, but I'll try to.
running 5.0 portable build 3784 on Windows 7 x64

It's quite difficult to say whats going on without the ftp session log, but I can only speculate that the 435 error might of been some type of TLS handshake failure.

When FlashFXP encounters a TLS error during FXP it will attempt to switch from the current FXP mode to the alternative or normal depending on the original setting. This switches which server handles the PASV/PORT commands. Now after switching if this new mode also fails the files are marked as failed since neither mode was successful.

However I am not so sure if this is exactly what happened because after one successful transfer the FXP mode will not automatically switch on failure, it's only if the first FXP attempt resulting in a failure that the switch will occur.

It is possible that the code in v5.0 is not 100% reliable or most likely an unusual compatibility issue under these conditions but its impossible to know without a ftp session log that includes enough information to see exactly what's going on.

In v5.0 most of the FXP code was rewritten to better handle SSL/TLS version incompatibilities and errors.

I personally have tested many different servers and combinations with the new code and never encountered such a problem but that doesn't mean one doesn't exist.

I would recommend turning on disk based logging, this would give us the best chance of having a complete session log if this should happen again.

this i caught today
[10:47:44] [L] 435 Failed TLS negotiation on data channel (SSL_accept(): (1) error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher), disconnected
and it skipped the file, marked it as failed
strangely, it was able to resume normal operation on the next file (it transferred okay)

that help any?

Can you please check the following setting

Start FlashFXP > from the main menu Options > Preferences

Then on the Preferences Dialog click Transfer section

Under Retry Failed transfers

Site to site: [value] Times

The default value is 0 and this could explain why the transfer is marked as failed instead of being retried, Increase this value to 3 and see if it helps prevent the files from being marked as failed.

435 Failed TLS negotiation on data channel (SSL_accept(): (1) error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher), disconnected

Does this occur on the first first file you attempt to transfer?

okay, i increased the value to 3, will see how it behaves now.
I will also try to narrow down when this error occurs, but yes, today it was the first file that didnt get caught in the dupecheck.
will keep you posted.

Just some general observations that may apply here, from my own experiences and testing when everybody starting switching to EC (elliptical curve) ciphers.

This type of error usually indicates incompatible certificates (cipher suits etc). However something I noticed early on what that this was dependent on which server initiated the negotiation. If one server is using EC ciphers and the other is not then transfer success will depend on who initiates the negotiation. If the negotiation is initiated by the server using EC ciphers then it will select an EC cipher by default. When the non-ec server attempts to match the cipher it will fail. If the negotiation is initiated by the non-ec server then it will usually succeed. This is because the server with ec ciphers still includes all the usual ciphers... so it can match if the other server is responsible for selecting the cipher.

This can be forced by tinkering with the retr-port/stor-pasv options under 'site to site method' in the site manager. This can of course cause other issues for servers with tricky networking/routing/etc. The best solution of course is to update the certificates where possible.

I had many discussions at the time with people resisting updating certificates to include ec ciphers. The argument was always concern about breaking compatibility, but as I said to them the burden of compatibility lies on those that refuse to update. It was a legitimate concern but as I found out almost nobody was aware of the 'hack' fix as mentioned above and they were all compromising long term security and performance for short term concerns.

In regards to the second issue.... there are a million reasons a transfer could fail. This includes a server issue (though I have not seen it for a long time) where a server has very limited port allocations and/or is not releasing them correctly or in a timely manner. This causes port exhaustion and if you monitor a transfer queue you can often observe several files in a row fail then all of a sudden one succeeds (as a port finally becomes available).

As bigstar mentioned you will need to look more closely at the logs to investigate this properly, but I would start by taking a closer look at the certificates being used.

the problem somehow got resolved accidentally, as i had to switch my RaidenFTPD over to TLSv1 in order for some of my buddies to be able to log on to it (dont ask, dont know why TLS worked where SSL failed, but RaidenFTPD is an obsolete software, I only keep it around for its convenient 'built-in' dupechecker). now this resulted that i could no longer transfer anything from the glftpd due to ssl connection error so i went into the options and selected protocol independent transfer for FXP and eversince i havent been receiving this error.

edit:
I take it all back. I queued about 1000 folders. When the process was done there were ofc a lot of failed transfers, which I was hoping was due to them being caught by the dupechecker. Just make sure I queued up all the failed transfers and surprise, there were a lot that transferred alright second time around. At the end I again requeued all failed and third time around again there were still some files that transferred.
[R] Data Socket Error: Failed TLSv1.2 negotiation, disconnected
checked the log and this was the only message I could find.
again this was FXP site to site with protocol independent transfer.
it just aint normal that some files randomly do not transfer the first time...

edit nr2:
something is definitely wrong, cause I see files transferring (can see the speed bar) that are supposed be caught by the dupechecker, but when the transfer is done, the files are not at their destination like they are not supposed to be, so that is at least normal. However I see some inconsistent fluctuations in credits meaning I lost some credits, but not as many megabytes as I can see transferring.
im totally lost now, cant follow this anymore....

Can I suggest you turn on disk/file logging in flashfxp and post some of it so the process can be examined more closely. In order to really narrow this down one would have to look closely at several examples of failed transfers and successful ones and compare what each is doing. This could very well point out the problem.

Also, since this started happening, have you tried with flashfxp 4?