Skiplist not working with transfers from 2 simultaneous sources

Bit of a hard one to explain this!

I have a FTP set up on a PC here and if I have 2 flash fxp windows open which are transferring the same material but from 2 different sources then the files from each server don't check if the file has already been uploaded and just writes the file anyway.

By way of a demonstration - Imagine that A is my home server and B and C are external servers that have exactly the same files on in exactly the same directory structure.

Flash FXP window 1 is transferring files from B to A
Flash FXP window 2 is transferring files from C to A

All the files have identical names are going into exactly the same directory.

I have skiplist activated for both manual and fxp transfers and it should automatically skip files of an identical size

However, flash 4.1.8 (1708) doesn't seem to check whether a file is there from another process before opening the file for writing, which ends up with files which have already been uploaded in their entirety being written twice in some instances.

For example if file 207.mp3 has already been uploaded from B to A, the second process (C to A) doesn't invoke the skiplist and just overwrites 207.mp3.

It is not a settings issue on the local server as flash fxp 3.6.0 (1240) doesn't have this problem and just skips equal files or tries to resume files that are smaller as this is how my skiplist is set up. It is also not limited to a particular file type as i have observed this behaviour with a variety of different file extensions.

Even if I export the settings from 3.6.0 (1240) and import them into 4.1.8 (1780) the problem still exists, so it looks like a bug in the flash 4 code somewhere.

This does not rely on the skip list but the file transfer rules.

Options > File Transfer Rules
Make sure you have a rule configured to do the following
If destination file size is equal then skip

On the same dialog make sure this option is checked
[X] Request file size/date prior to transfer

Finally click the OK button.

The file transfer rules were completely redone for v4, most likely you are just missing these settings.

BTW you can also configure this per-site so that these rules only apply to Site A via the Site Manager.

If you are already using per-site transfer rules for Site A then you will need to modify them instead of the global rules, as they override the global rules.

Hi, that is exactly how I have got it set up, but it is not working as it should. Apologies for not referring to it correctly in the thread title, but this is certainly broken.

Based on the two session logs you showed me I can clearly see that in the second log that FlashFXP checks to see if the file exists and the FTP server reports that it does not.

[L] MLST 07-obduktion-semi_humans.flac
[L] 550 File or directory not found.

If the second log occurred after the first then this is an issue with the FTP server not reporting that the file exists to FlashFXP.

Just tested again aand got exactly the same things happening.

Some times the files are correctly reported as being there and are not transferred, but some times they are over written and the MLST command does not report if the file is there.

As part of the testing I have gone back and done exactly the same transfers with flash fxp 3 and ftprush and both of those do not transfer files that are already uploaded.

Therefore, it would seem fairly clear that the issue lies in flashfxp4 somewhere becasue as soon as different programs are used there is no problem.

FlashFXP v3.x doesn't use MLST, does ftprush?

Please post the complete logs from the other clients that you are testing so we have a thorough comparison.

Also open the flashfxp.ini in notepad and make sure that this entry does not exist under [main], if it does delete the line.

DGFS=1

This ini only setting instructs FlashFXP not to get or check the file size on the target when performing a site to site transfer.

DGFS=1 isn't present at all in the .ini (there is no entry for dgfs at all)


Possible thoughts on this;
The MLST feature on flash 4 has a bug in it somewhere - However, turning MLST off in flash 4 produces the same error

With turning MLST off and it still not working, it sure looks to me like the error is in flash 4 as flash 3 and ftprush both do exactly what they are supposed to do

Each site has exactly the same settings - is flash not sending the command, or does it know that the file is there and simply not care and over-writes it anyway?

Originally Posted by spudgun

Why isn't flash sending a MLST command in the second process like this?
[L] MLST 03-willis_dempo-where_did_you_get_that_(body)_(instrubodymental).f lac
[L] 550 File or directory not found.

This is what I am trying to find out; it should be sending the MLST command as far as I can tell from your session log, but since your log is incomplete it's extremely difficult to know 100% for sure.

I conducted several tests and was not able to reproduce the issue, I will set more time aside tomorrow to investigate this issue in more detail.

I've done a lot of testing today and I think I have narrowed this down to a particular FTP site.

What is happening is that this particular site, as indicated above, only sends 1 MLST command per transfer, this doesn't seem to matter if you are running 1 process or multiple processes

okay - I think we have a breakthrough with this

Have done loads more testing and if options/filters/skiplist/enable skiplist is ticked then the problem happens

If that option is unticked, then everything is good and it works as it should do.

Only problem with this, however, is that an old problem that I posted here https://oss.azurewebsites.net/forum/79529-post14.html

and the solution was given here https://oss.azurewebsites.net/forum/79536-post19.html

now remains, as the only way to fix it was to have that folder type skipped via the skip list

Changing things via Preferences Dialogs > Transfer > Retry Failed Transfers has no impact and it just gets stuck in the loop I identify in the first link posted above

Your results are quite unexpected.

The skip list *should* not have any impact on the file transfer rules.

When the skip list is enabled any matching items are skipped immediately and will never make it into the transfer queue.

This applies to queuing files/folders directly or any matching items within a queued folder.

When FlashFXP processes a queued folder for transfer it it skips any items matching the skiplist and then adds the remaining items to the transfer queue.

In addition the old 'bug report' regarding the failed folders and end-less loop, This should no longer be an issue. Even with the transfer retry limit and reset and retry failed transfers, it will only retry a specific amount of times.

I have to wonder if this is more or less a really unusual compatibility issue with the only this specific FTP server software and configuration.

I have tested at least a dozen different FTP servers and I have not been able to reproduce this issue. Is there any way for me to get a test account on this server?

Thanks for getting back to me, setting up a temporary account isn't a possibility I'm afraid - so perhaps best that we focus on fixing the skiplist issue as that is still very much broken and it just gets stuck in a loop.

If you tick retry failed transfers for site - site and ask it to retry them twice you still get exactly the same and the only way to avoid it is to put an entry in the skiplist as you highlighted in your earlier fix

Status update on the end-less looping on directory change failure..

It appears that this is caused by having the option "Remove failed transfers from queue" unchecked. When this option is unchecked the result is an endless loop.

When "Remove failed transfers from queue" is unchecked the "Retry failed transfers" settings do not apply, as the retry failed transfers feature requires failed transfers to not be removed from the queue.

I'm not sure if there is a good reason to have the "Remove failed transfers from queue" feature.

One thing is for sure, I will need to test all of the code that uses this setting unchecked to make sure the result is as intended. Right now lots of the code will cause an endless loop on failure.

Thanks for the update.

The exact same problem happens if the box is ticked for 'retry failed transfers' and you set a limit - it still gets caught in the endless loop of it trying to enter this directory (or any directory created by a .sfv check.

I think the retry failed transfers needs to be unchecked for situations where multiple people are uploading the same batch of files - so this feature/option does need to remain

Best of luck in finding a solution for this

I sent you a private message with a dev build for testing, I look forward to hearing from you.

In case anyone else is following this thread - the development build fixed the infinite loop caused by the directories and following lengthy discussions I think bigstar has figured out what is causing the problems with 2 simultaneous transfers

This issue is now resolved