File Date Problem

/~Uploads/Sea7/Tosec UNK/
Red Zone (US) (Disk 2) (1992) (Psygnosis Ltd) (MFM + Longtrack 80).zip
reisende_im_wind_a.zip


Posting again, as my post earlier yesterday doesnt seem to have saved? Same FTP, I can send again if you need details.

Path : /~Uploads/Sea7/Tosec UNK/

Red Zone (US) (Disk 2) (1992) (Psygnosis Ltd) (MFM + Longtrack 80).zip
reisende_im_wind_a.zip

FTP date has these set at 23/07/2013

However, when downloaded the first file gets set to 22/07/2013 and the second gets set at 23/07/2013.
I have other examples I can provide.

There is a file called : /filelist.txt which highlighted the problem to me -- was trying to write a script to keep track of my local paths/dates/size and to compare it to this filelist.txt to find out any new additions. His filelist.txt matches the dates shown in flashfxp (his txt file is in MM/dd/yy)... I cant find out why some file dates are correct when I download them, and others are out by a day? I suspected something around no time stamps on FTP side showing ... but doesnt give me any reliable results / conclusions.

I do have File Transfer Rules->Preserve File Date->Upload / Download / Site-to-site ticked.

If this is fixable, do you have any ideas how I might fix the file dates locally / re-sync the files / folders recursively without having to re-download to fix?

In the File Transfer Rules dialog do you also have the following setting checked?

[x] Request file size/date prior to transfer

When this option is checked FlashFXP will send additional commands to the server to retrieve the exact size & date of the file prior to transfer, when the transfer is completed FlashFXP sets the local file date to match the date provided.

The FTP server is only returning a partial date (date without time) in the file listing.
And on this FTP server it appears that the partial date is rounded up to the next whole day.
(Depending on the FTP server software some round up while others round down)
So 07/22/2013 06:32 PM is rounded up to 07/23/2013

Here we can see in my session log the following:

FlashFXP requests the current size of the file and the server returns the result
[07:28:27] [R] SIZE OkimateDemo.zip
[07:28:27] [R] 213 347096

FlashFXP requests the current date/time of the file and the server returns the result
[07:28:27] [R] MDTM OkimateDemo.zip
[07:28:27] [R] 213 20130722225916

FlashFXP sets up the data channel using the PASV command
[07:28:27] [R] PASV
[07:28:27] [R] 227 Entering Passive Mode ()
[07:28:27] [R] Opening data connection IP: XX PORT: XX

FlashFXP requests the file to be sent over the data channel
[07:28:27] [R] RETR OkimateDemo.zip
[07:28:28] [R] 150 Opening BINARY mode data connection for OkimateDemo.zip (347096 bytes).
[07:28:30] [R] 226 Transfer complete.
[07:28:30] Download: OkimateDemo.zip 338 KB in 2 seconds (151.1 KB/s)

Now if you want to use FlashFXP to mirror the remote content to your local PC without having to worry about downloading the same files over and over (and only download missing/modified files) then the solution is rather simple.

Add a new (per-site) file transfer rule to skip on download when the date/size are equal.

Hi,

Just checked and "Request file size/date prior to transfer" is enabled

From the session I see this :

[14:26:00] [R] MDTM Red Zone (US) (Disk 2) (1992) (Psygnosis Ltd) (MFM + Longtrack 80).zip
[14:26:00] [R] 213 20130722231501
[14:26:04] [R] MDTM reisende_im_wind_a.zip
[14:26:04] [R] 213 20130723000545

So I guess from what I understand the FTP server side is doing some rounding down (even though time is 1501).
MDTM queries more detailed information on the correct date / time.

I would hazard his FTP software is probably running some script dump the files / dates output to the filelist.txt -- and doesn't report the true date/time correctly.

I cant think of any reason why I would want to change my dates to what the FTP server thinks they should be. Makes sense to me to stick with what is returned from MDTM.

Ill update my script check to assume +/- 1 day on the date field.

Would there be a way I could trawl the FTP for every file, and retrieve the folderpath / filename / date / size to a text file?
I'd like to periodical check for difference in content, but do this in an offline manner ... as my data is stored on another drive, until I get around to getting a NAS unit.

Just in case, do you know of any fast / memory efficient tools for doing a difference compare of the content. I tried powershells compare, but its memory hungry on large files (2*84MB files). Windiff early tests seems to work... just need to test it some more... but open to better tools you might suggest.

Again, many thanks for the explanation. I will close the ticket in the next 24 hours.

I did notice, Ill just point it out as I dont think it really matters...but worth mentioning?

the time on the files are (according to MDTM info)

23:15:01
00:05:45

in Windows 10 / Properties it shows as 23:15:00 and 00:05:44.

This is a limitation caused by using the Windows API DosDateTimeToFileTime()

The DOS DateTime format has resolution of 2 for seconds.

https://msdn.microsoft.com/en-us/library/windows/desktop/ms724247(v=vs.85).aspx

I am going to revisit the code and try to find a better solution to eliminate this step.

I came up with an alternative solution that does not require the use of the DosDateTimeToFileTime() API and so far everything appears to be working great, I included this simple change in the new release that I posted today, Version 5.4.0.3956.

I think I may have linked you a pre-release of this build, and if so you'll need to manually download the new version from the website to get the update since I neglected to increase the build number.

Awesome.

Ill be sure to take a look tomorrow.
Many thanks for looking into this.

Looks like I only have the URL for : ffxp5.4.0.3956.zip ... but reports a 404 warning about the file not being created yet.

The primary download on the website was updated with the new release

https://oss.azurewebsites.net/download

Brilliant,

Just re-downloaded and installed over the top. Quick few tests seems to indicate everything is working as discussed.
Thanks for taking the time... sorry it was a minor query about the second field after a really useful discussion how the date / time fields work. Any quick ideas how I might be able to recursively trawl the FTP / retrieve just the details so I can re-sync the file/date time fields where the path / size matches locally (without having to re-download... I have 1/2 TB of synced small and large files)? I have seen a FTP powershell script, which I can probably adapt ... is there any way I can plugin / control FlashFXP from powershell? I'm not a real programmer, but I do know some basic scripting.

Other than that, many thanks for the support again.
Happy for me to close this ticket?

Originally Posted by sTe

Any quick ideas how I might be able to recursively trawl the FTP / retrieve just the details so I can re-sync the file/date time fields where the path / size matches locally (without having to re-download... I have 1/2 TB of synced small and large files)? I have seen a FTP powershell script, which I can probably adapt ... is there any way I can plugin / control FlashFXP from powershell? I'm not a real programmer, but I do know some basic scripting.

I am unaware of any existing solution that can do this.

I looked at the possibility of using custom command macros in FlashFXP to do it, but a few steps are missing to make it work.

FlashFXP would need to be able to issue the MDTM command on each object in the folder to retrieve an accurate modified timestamp, Adding this would be rather trivial (which I did for testing purposes), but then we face some additional issues, such as getting info from one browser pane and then applying it to the other and navigating both browser panes to update the timestamp recursively. I have come up with a partial working script (after some code changes in flashfxp) but the script must be executed manually for each folder, which is not very practical.

Also since MDTM command needs to be issued for each file in the file listing, it adds quite a bit of time to each folder being browsed, I think trying to correct the ~1 second error on the timestamp on all your existing files would probably take several days and quite a bit of work.

Using FlashFXP this 1 second difference would not have any major impact on the file transfer rules since we allow for up to a minute difference on new files and up to a day on files over a year old (where only the DD/MM/YYYY is provided in the listing) on files returned by the legacy UNIX:L8 file format.

If you ever think of a solution would you let me know?

I had an idea... setting a rule to if size is equal and date match ... then resume, and force a 1KB file roll back. As I thought maybe the date/time overwrites at the end of the transfer. However, it looks like I cant force a "resume" on existing files when its already the same size... Ill have to double check my settings to verify I haven't missed something.

Leaving this for days isnt a problem for me, so happy to have anything run in the background to correct the times. As long as it can recursively drill through the files and folders.