Archiver

[Rusher]

so what script would you guys recommend to use for moving releases ?

for example

/incomming/mp3/

lets say i want everything in that location that is older than 25 days to be moved to

/Archive/mp3/


it does NOT need to update symlinks or so. just move directories and files.

[Sc0tTy]

I'm looking for the same script , updating dupe log is not required but it needs to be able to move directory based on regexp

[peep]

Try looking for warchive. Old script from many years ago, but might still work.. https://sites.google.com/site/ioftpdscripts/home/warc

[jeza]

You can test this (http://jezatools.googlecode.com/files/jeza.ioFTPD.ArchiveScript.v3.2.69.549.zip) script for moving/deleting folders.
Microsoft .net framework 3.5 SP1 must be installed on machine where the script will be executed.

You can setup windows task for it or add a line to ioFTPD scheduler.
Example:
archive = 59 3 * * E:\server\ioFTPD\scripts\jeza.ioFTPD.ArchiveScript \jeza.ioFTPD.Archive.exe

It uses XML file for configuration (check Example.txt).
This script can:

move/delete folders older than X days


move/delete folders when total count of folders exceeds X


move/delete folders when free space on DISK is below X BYTES


move/delete folders when total used space on DISK is over X BYTES


move/delete folders when specified folder total used space is over X BYTES


I suggest using it on some test folder first :o

[Sc0tTy]

Yeh i didnt take a look @ WarChive because its so old an i'm running the latest ioFTPD
Jeza i'll take a look at it :) I'm a C# developer aswell ( in my spare time ) is it possible that you can add regexp to the options, that would be very usefull for me.

[jeza]

like for include or exclude specific folders by their names?
Right now you can exclude folders that start with specified string.

<task>
<skipPattern>.</skipPattern>
<skipPattern>_old</skipPattern>
<status>Enabled</status>
<actionType>Move</actionType>
<source>E:\\temp\\TestFiles</source>
<destination>E:\\temp\\archive</destination>
<action id="TotalFolderCount">
<value>21</value>
<minFolderAction>5</minFolderAction>
</action>
</task>

This will skip all folders that start with "." or "_old"

[Sc0tTy]

Its for a series archive and i want to archive per show.

So if a part in the release name is matched it has to be stored in the shows' folder

Archive\Tv\Series1
Archive\Tv\Series2

[jeza]

jeza.ioFTPD.Tools.v3.3.7.551.zip (http://jezatools.googlecode.com/files/jeza.ioFTPD.Tools.v3.3.7.551.zip)

This version has RegExpressionInclude and RegExpressionExclude setting.

May the Force be with you :)

[Rusher]

thx jeza !

[o_dog]

iotvarchive should work too ifit's jsut tvshows you want to archive. Think i have an updated version of it somewhere with more features.

[Sc0tTy]

o_dog , don't know that script so might as wel test jeza's script :)

[Sc0tTy]

[2011-10-06 23:29:22] System.IO.IOException: Source and destination path must have identical roots. Move will not work across volumes.
at System.IO.DirectoryInfo.MoveTo(String destDirName)
at jeza.ioFTPD.Archive.ConsoleApp.ExecuteArchiveTask( ArchiveTask archiveTask, DirectoryInfo directoryInfo)
at jeza.ioFTPD.Archive.ConsoleApp.Execute()
at jeza.ioFTPD.Archive.Program.Main(String[] args)

[jeza]

:eek:
I will fix it during the weekend...

[Yil]

Hmm. Jeza if you get a chance, let me know what "features", or even propose an interface, you would like to see out of ioFTPD to help support whatever scripts your working on. The existing shared memory interface is pretty useful to adjust vfs perms, userfiles, etc, and there was some talk about providing a C# wrapper to make that accessible, but even the iTCL interface has a number of holes in it. I haven't prioritized filling them because no-one was complaining and I figured I'd get around to it when I needed the functionality in ioYil, but that was only going to be iTCL only originally. When I saw that exception stuff about cross drive moves a light bulb went off in my head that you will have the exact same problems ioYil will have...

If you want to move a directory tree to another drive you will have to do a copy/delete, which is trivial in C# to do, BUT if you want to actually be able to always delete the old directory you'll have to kick users in the old directory like nxtools does which is something you probably can't do now without access to the shared memory interface. Internally the RNFR/RNTO code for directory moves actually goes a step farther and locks the directories so nobody can enter the old dir as it's easy to see clients/bots get kicked and immediately restart. Now that I think about it I'm not sure if the internal code actually kicks users or just locks it so the next file can't be accessed because I was thinking about a slight of hand technique where I would automatically move users to the new dir if symbolic paths was enabled. Either way you'll also have to arrange to output something every 10-30 seconds if this is called from a site command to keep clients happy. Are you aware of the EXEC module line prefixed commands? You can enable non-buffered immediate output, change vfs perms on files/dirs, etc by just sending output back to the server and I believe you can take advantage of that via scheduled EXEC tasks as well if you need to change perms which I presume would be useful if you need to make the archived dirs read-only or something.

What I intend to do is either expose the directory locking stuff and/or the internal directory move stuff to iTCL for my use and I guess you wouldn't mind this in the future either, but if you can think of anything else you'd like to see please let me know. The only other hint I can give you is that internally ioFTPD takes special care to make sure that it copies over the .ioFTPD file before moving any files or subdirs. This guarantees that permission information is intact so the moved directory doesn't expose anything during the move and I thought you might want to make sure you do the same.

[jeza]

new version : jeza.ioFTPD.Tools.v3.3.10.553.zip (http://jezatools.googlecode.com/files/jeza.ioFTPD.Tools.v3.3.10.553.zip)

Moving between volumes should work now. You can Copy, Move or Delete folders :idea:.

[jeza]

Yil
As you mentioned the correct way for the archive script is to change directory attributes and kick users from it, before taking any archive action.
Right now the script will fail when someone is browsing in source and after moving the folders all info about who uploaded it is gone.

Trough c# wrapper one should be able to have access to VFS permissions, managing user/groups, see who is online, kick users, get user/group stats...

[Yil]

Jeza: When you say the info about uploaded files is lost are you talking about the file/group/timestamp/perms of the files themselves or are you talking about the zipscript info such as ioNinja shows you when entering a directory?

The later is chattr data stored in the .ioFTPD file itself for the directory. If you create the target dir then immediately copy the .ioFTPD file into the dir like I suggested then this data probably makes it over. I say probably because without telling ioFTPD to mark the dir as stale and forcing it to re-read the .ioFTPD file it might not pick up the change and write out a blank entry with no chattr data in it if someone access the dir between creation and the copying of the .ioFTPD file. There's a number of other rare race conditions but that's the most likely because you're doing this all behind ioFTPD's back...

On the other hand, if you're talking about normal file perm stuff the problems go from really rare to annoyingly often. You're obviously aware that internally ioFTPD caches every directory before it's displayed/used by the server. That's necessary because that is when the permission information from the .ioFTPD file is applied to the directory itself and any files in the directory. To make sure the cache isn't stale because of external manipulation of the directory (i.e. files added/deleted in windows explorer, zipscripts extracting files, etc) the directory timestamp is compared against the cached entry and if they differ the cache is updated. Currently this means the directory is rescanned and the .ioFTPD reparsed, however if the .ioFTPD file notices that it has entries for non-existing files (and I think missing entries for files) it re-writes itself when done so it reflects the current directory.

What that actually means is problems. You see, if you move a file from the src dir to the dest dir and a user accesses the src dir again the .ioFTPD file gets re-written to reflect the deleted file. If you grabbed the .ioFTPD at the start, like I mentioned above, nothing about existing files is lost. The problem for you is much trickier at the dest dir though if you copied the .ioFTPD only at the start and someone enters the dest dir it will clear all the entries for the missing files. ioFTPD doesn't have this problem because it locks the dest and src dir so nobody can enter/touch them at all while things are being moved...

Now you see why I mentioned it would be really useful to just give you a ioFTPD copy dir function or expose dir locking facilities. But that would of only helped you if you were doing this in TCL, but in C# even the chattr get/set and vfs permission functions can't be used by you right now.

On the other hand you do have a 99% good enough solution I think. Grab the .ioFTPD file at the start, make a list of all files to copy by examining the dir, then make the dest dir, copy the .ioFTPD file into the target dir and make a backup of it there or just stash it in memory as it isn't large, then copy all the files from the list you made (you can deleted after each file), then re-write the destination .ioFTPD from the backup/memory copy at end. That should cover almost everything except permission changes while the copy is taking place and any new files added to the src/dest which in an archiver shouldn't really be happening... In TCL I'd suggest tricks like make the dest a hidden private dir, read-only, etc but just copying the .ioFTPD file at the start then re-writing the same info (not the current .ioFTPD file at src) at end should work for you.

[jeza]

OK. Tnx for your information. I will do it as you say.


Hmm. Jeza if you get a chance, let me know what "features", or even propose an interface, you would like to see out of ioFTPD to help support whatever scripts your working on.

I created a interface (https://oss.azurewebsites.net/forum/ioftpd-general/15114-tcl-shared-memory-stuff.html#post79955) of methods that could be useful i think...

[Sc0tTy]

Thanks!

I'll install it today and wi'll let you know :)

[Sc0tTy]

It works :)
(date older move)

I'll start configuring regexp moving and let you know.

PS: does it check if something is complete or not?
If not it would be a great option, since I don't want stuff moved to my archive when its incomplete.

Also when i unpacked the zip it had Archive and Archivescript folder , i used the latter.

[jeza]

great :)

I have no clue how to check if some release is complete or not.

Maybe before move, the script could check if there is some COMPLETE tag inside the folder that should be moved.

If you are using the script on some ftp site where daily new releases are being uploaded, then some sort of AUTO NUKE release script checks for that.

If you are using the script just for auto moving releases when downloaded from torrent for example, than i have no idea how to find out if a release was completed.

Always use the ArchiveScript folder

[Sc0tTy]

Found a bug, when not configuring the action node ( wich i want because i want it to always run ). i get the following error:

[2011-10-12 10:16:23] Starting with task: ['ArchiveType: Move, ArchiveStatus: Enabled, Source: '*****', Destination: '*****', Action: ']
[2011-10-12 10:16:23] System.NullReferenceException: Object reference not set to an instance of an object.
at jeza.ioFTPD.Archive.ConsoleApp.Execute()
at jeza.ioFTPD.Archive.Program.Main(String[] args)

I did a quick work around:
<action id="TotalFolderCount"> :: Action will execute if total folder count in source is greather than 33 andwill remove 33-22 oldest folders
<value>0</value>
</action>

Yeh i'm using ioNinja zip script ( v1 alpha ) so i have a COMPLETE file ( or in complete :P ) no torrent crap here.

[jeza]

If a task should not be executed set its status to Disabled.

Besides only regExpressionInclude and regExpressionExclude are not necessary in configuration, all other nodes must be there.
You will always get null reference exception if some of the other nodes is missing in configuration:

There is an XML Schema file that prevents missconfiguration (XMLSchema.xsd) ;)

[Sc0tTy]

Offcourse :) But my logical thinking said if you remove ( or empty )the action node it should always run. But no problem there for me :)

Also what i wanted to ask , can you add RegexOptions.IgnoreCase to the regexp function call, this would save me ( and others quite a lot of work :) )

[Sc0tTy]

Found another bug:
If you run the archiver not from the same directory like this:

C:\Users\Administrator>D:\ioFTPD\scripts\jeza.ioFTPD.ArchiveScript\jeza.i oFTPD.Archive.exe

It simply does nothing, this also is the case in Task Scheduler, i could create a .bat file for this, but it should be a simple fix in the script itself + other ppl won't have to do it aswel :)

[jeza]

Windows Task action has an option 'Start in'. Specify here 'D:\ioFTPD\scripts\jeza.ioFT PD.ArchiveScript' or where ever your script is located and it should work.
I will add RegexOptions.IgnoreCase

[Soxiz]

Yil could you explain the "EXEC module line prefixed command" that your where talking about?

[Yil]

Soxiz: check out the new thread here about EXEC module.

[jeza]

New version https://oss.azurewebsites.net/forum/new-scripts-announces-updates/15266-jeza-ioftpd-tools.html.

[Rusher]

New version...[/URL].


that link does not work for me =(

[jeza]

here is the thread with correct download links
https://oss.azurewebsites.net/forum/new-scripts-announces-updates/15266-jeza-ioftpd-tools.html

[Rusher]

Is it possible to have archiver use the RNFR / RNTO command for moving stuff on ftp to keep everything correct in dupe db ? or is there another way i can make it work ?

example :

i want to archive everything older than 10 days so archiver moves :

/mp3/2011-12-01/*

to

/archive/mp3/2011-12-01/*

but then when i run site dupe/search i want the enteries to be updated and point to new location.

edit : RNFR/RNTO doesnt work when use as "RNFR 2011-12-01" cause it doesnt execute that command sepertly for every subdir wich need to be done to have db to be updated.

[Sc0tTy]

Read jeza's post :)

[Yil]

RNFR/RNTO on a directory with subdirs not updating nxTool's database is a known bug. There is also no way to force nxTools to just rebuild a portion of it's database so it's more than a bit annoying. Unfortunately I can't think of any site script that handles an archiver moving stuff around correctly...

[jeza]

i plan to add updating of an dupe when running archive, but it will update only my own dupe script.
so if you are using some other dupe script, like nxdupe, i can not help you with :cry:

[Rusher]

so i'm ****ed.. i'm using nxtools for dupe now. but whenever you add support for your archiver i will take a look at that then. its a ***** having to move everything manually now :( and site search/dupe with nxtools is really fast and works just the way i want :D hope yours isnt to big of a change.

[Yil]

I've got a possible solution. Grab one of the SQLite .NET packages and just have the archiver update the nxTools dupeDB directly. Just make sure you don't use one huge transaction during the move as that would lock the DB while it's copying files.

Update: That might be complicated by the fact that nxTools uses a really old modified SQLite which ignores case. You can get that same behavior in current SQLite releases without any modifications by asking for it so we might need an "updated" version of nxTools. I can and should probably do this for everyone anyway...