Archiver - Page 2

jeza · 10-08-2011, 04:01 AM

Yil
As you mentioned the correct way for the archive script is to change directory attributes and kick users from it, before taking any archive action.
Right now the script will fail when someone is browsing in source and after moving the folders all info about who uploaded it is gone.

Trough c# wrapper one should be able to have access to VFS permissions, managing user/groups, see who is online, kick users, get user/group stats...

Yil · 10-10-2011, 08:55 PM

Jeza: When you say the info about uploaded files is lost are you talking about the file/group/timestamp/perms of the files themselves or are you talking about the zipscript info such as ioNinja shows you when entering a directory?

The later is chattr data stored in the .ioFTPD file itself for the directory. If you create the target dir then immediately copy the .ioFTPD file into the dir like I suggested then this data probably makes it over. I say probably because without telling ioFTPD to mark the dir as stale and forcing it to re-read the .ioFTPD file it might not pick up the change and write out a blank entry with no chattr data in it if someone access the dir between creation and the copying of the .ioFTPD file. There's a number of other rare race conditions but that's the most likely because you're doing this all behind ioFTPD's back...

On the other hand, if you're talking about normal file perm stuff the problems go from really rare to annoyingly often. You're obviously aware that internally ioFTPD caches every directory before it's displayed/used by the server. That's necessary because that is when the permission information from the .ioFTPD file is applied to the directory itself and any files in the directory. To make sure the cache isn't stale because of external manipulation of the directory (i.e. files added/deleted in windows explorer, zipscripts extracting files, etc) the directory timestamp is compared against the cached entry and if they differ the cache is updated. Currently this means the directory is rescanned and the .ioFTPD reparsed, however if the .ioFTPD file notices that it has entries for non-existing files (and I think missing entries for files) it re-writes itself when done so it reflects the current directory.

What that actually means is problems. You see, if you move a file from the src dir to the dest dir and a user accesses the src dir again the .ioFTPD file gets re-written to reflect the deleted file. If you grabbed the .ioFTPD at the start, like I mentioned above, nothing about existing files is lost. The problem for you is much trickier at the dest dir though if you copied the .ioFTPD only at the start and someone enters the dest dir it will clear all the entries for the missing files. ioFTPD doesn't have this problem because it locks the dest and src dir so nobody can enter/touch them at all while things are being moved...

Now you see why I mentioned it would be really useful to just give you a ioFTPD copy dir function or expose dir locking facilities. But that would of only helped you if you were doing this in TCL, but in C# even the chattr get/set and vfs permission functions can't be used by you right now.

On the other hand you do have a 99% good enough solution I think. Grab the .ioFTPD file at the start, make a list of all files to copy by examining the dir, then make the dest dir, copy the .ioFTPD file into the target dir and make a backup of it there or just stash it in memory as it isn't large, then copy all the files from the list you made (you can deleted after each file), then re-write the destination .ioFTPD from the backup/memory copy at end. That should cover almost everything except permission changes while the copy is taking place and any new files added to the src/dest which in an archiver shouldn't really be happening... In TCL I'd suggest tricks like make the dest a hidden private dir, read-only, etc but just copying the .ioFTPD file at the start then re-writing the same info (not the current .ioFTPD file at src) at end should work for you.

jeza · 10-10-2011, 11:13 PM

OK. Tnx for your information. I will do it as you say.

Quote:

Originally Posted by Yil

Hmm. Jeza if you get a chance, let me know what "features", or even propose an interface, you would like to see out of ioFTPD to help support whatever scripts your working on.

I created a interface of methods that could be useful i think...

Sc0tTy · 10-12-2011, 02:28 AM

Thanks!

I'll install it today and wi'll let you know

Sc0tTy · 10-12-2011, 02:41 AM

It works

(date older move)

I'll start configuring regexp moving and let you know.

PS: does it check if something is complete or not?
If not it would be a great option, since I don't want stuff moved to my archive when its incomplete.

Also when i unpacked the zip it had Archive and Archivescript folder , i used the latter.

jeza · 10-12-2011, 02:59 AM

great

I have no clue how to check if some release is complete or not.

Maybe before move, the script could check if there is some COMPLETE tag inside the folder that should be moved.

If you are using the script on some ftp site where daily new releases are being uploaded, then some sort of AUTO NUKE release script checks for that.

If you are using the script just for auto moving releases when downloaded from torrent for example, than i have no idea how to find out if a release was completed.

Always use the ArchiveScript folder

Sc0tTy · 10-12-2011, 03:23 AM

Found a bug, when not configuring the action node ( wich i want because i want it to always run ). i get the following error:

[2011-10-12 10:16:23] Starting with task: ['ArchiveType: Move, ArchiveStatus: Enabled, Source: '*****', Destination: '*****', Action: ']
[2011-10-12 10:16:23] System.NullReferenceException: Object reference not set to an instance of an object.
at jeza.ioFTPD.Archive.ConsoleApp.Execute()
at jeza.ioFTPD.Archive.Program.Main(String[] args)

I did a quick work around:
<action id="TotalFolderCount"> :: Action will execute if total folder count in source is greather than 33 andwill remove 33-22 oldest folders
<value>0</value>
</action>

Yeh i'm using ioNinja zip script ( v1 alpha ) so i have a COMPLETE file ( or in complete :P ) no torrent crap here.

jeza · 10-12-2011, 04:29 AM

If a task should not be executed set its status to Disabled.

Besides only regExpressionInclude and regExpressionExclude are not necessary in configuration, all other nodes must be there.
You will always get null reference exception if some of the other nodes is missing in configuration:

There is an XML Schema file that prevents missconfiguration (XMLSchema.xsd)

Sc0tTy · 10-12-2011, 05:06 AM

Offcourse

But my logical thinking said if you remove ( or empty )the action node it should always run. But no problem there for me

Also what i wanted to ask , can you add RegexOptions.IgnoreCase to the regexp function call, this would save me ( and others quite a lot of work

)

Sc0tTy · 10-12-2011, 05:08 AM

Found another bug:
If you run the archiver not from the same directory like this:

C:\Users\Administrator>D:\ioFTPD\scripts\jeza.ioFT PD.ArchiveScript\jeza.ioFTPD.Archive.exe

It simply does nothing, this also is the case in Task Scheduler, i could create a .bat file for this, but it should be a simple fix in the script itself + other ppl won't have to do it aswel

jeza · 10-12-2011, 06:28 AM

Windows Task action has an option 'Start in'. Specify here 'D:\ioFTPD\scripts\jeza.ioFT PD.ArchiveScript' or where ever your script is located and it should work.
I will add RegexOptions.IgnoreCase