ZR-Tools - 0day zip-cleaner, ban-script

[WarC]

todiloo,


so its finally released to the public. This is some small utils requested by Zer0Racer and is specially designed for him but some others might like them as well.

enjoy
/WarC

ZR-Tools v3.x Features

ZR-Clean Features
+ Clean within zip-files when uploading to ioFTPD.
+ Can be run manually as util in cmd or as script inside ioFTPD.
+ Clean zips and dirs recursively when run manually.
+ Removes comments from zipfiles.
+ Logs names and paths of all new (unknown) nfo-files.
+ Logs names and paths of all removed nfo-files.
+ Exclude paths where ZR-Clean should not be active.
+ Includes Zer0Racers lists with banned and accepted nfo-files.

ZR-Ban Features
+ Ban dirs and files from being uploaded in ioFTPD.
+ Section specific. Add your own sections to ZR-Tools.cfg.
+ Ability to use multiple ban-lists.
+ Exclude paths where ZR-Ban should not be active.

ZR-2F
+ Converts ZR-Tools ban-list to FlashFXP skiplist format.
+ Usage: ZR-2F.exe <ban-list> <skiplist name>
Skiplist name should be SKIPLIST.DAT


EDIT by Zer0Racer: Merged ZR-Tools 1.x 2.x and 3.x threads. Here is link to post about ZR-Tools v3.x, http://www.ioftpd.com/board/showthread.php?s=&postid=11786#post11786

[Zer0Racer]

Yes it's very good isn't it? :) I've been waiting for an app like this for a long time.

Another very nice feature is logging of unknown nfo-files. It logs filename and vfs path to the zipfile which contains that unknown file. This makes it very easy to spot new crappy nfos. And also easy to know which ones to add to the accepted list. The lists of unwanted and accepted (nfo)filenames that I have put together are left as they are. It's up to you to add new entries when needed. ;)

ZR-Clean.exe can also be executed from cmd line for cleaning your already existing 0day-archive recursively.

Read the manual included for further info.

WarC rules.

/Zer0Racer

[Giant]

Very nice and usefull this one.

One little problem i do have.
Must be a user error but i'll ask for advice anyway.

I made a Test.zip with 5 nfos that is on the banned list.
Cleaning this one from cmd works just fine. Trying to upload the nfos is blocked.
So far so good.
But when uploading this Test.zip i only see this response:
226--[ZR-Clean]--------------------------
226-Processing ZipFile...
226--------------------------------------

And the nfos isnt removed.

From the cmd line it looks like:
-[ZR-Clean]--------------------------
Processing ZipFile...
deleting: af.nfo
deleting: trope.nfo
deleting: bps-0wnz.nfo
deleting: HAPPY-BDAY.jpg
deleting: hol.nfo
-------------------------------------


Ideas and hints would be nice.
Using windows Xp with sp1.
ioFTPD the regged version
ioZS v1.02
ioA 0.7.4

[Zer0Racer]

Ah you're right Giant. We forgot to mention that ComSpec has to be added to ioftpd.env in ioFTPD/etc dir in order to get it to work as wanted. :)

Check your enviroment variables in windows to see which ComSpec-path is used in your computer.

Example for win2k:
ComSpec=C:\WINNT\system32\cmd.exe

WinXP should be ComSpec=C:\WINDOWS\system32\cmd.exe or something like that.

It should work now :)

WarC has added the additional info to manual.

[phoenixfr]

remind me the good old CFA ZipClean
But on a server side
as always , good job and very nice util for 0-day lovers

sugestion:
maybe you could use the same principle as in ZipClean : when there's only one nfo and it is not in the accepted list just automatically add it to the allowed list. You can also auto add to the banned list when you know that an accepted nfo is already there but you don't know the other nfos.

NB: haven't used the app, but just read the manual and ini so maybe. I don't know if such a level of automation is good . so maybe there's an other way of asking user approval to add the nfos found in the banned and accepted list.

[Giant]

Jupp works sweet now.
Thanks for the fast reply with info :)

[Zer0Racer]

In answer to phoenixfr's suggestion, ZR-Tools was never intended to have that kind of AI. That level of automation would take too much time to develop and would not be worth it, and not very accurate.

The example from zipclean that phoenixfr mentioned will actually not work. Some 0day-releases with multiple zips often has the rel-nfo in only the first or last zip-file. In the case where rel-nfo is in last zip, unwanted nfos could be injected in the other zips. Then if that nfo is not present in either banned or accepted list, when first zip is uploaded, automation would then add it to accepted list. What a mess!

I downloaded zipclean to have a look at it and found that the included purge.txt contained multiple entries of one or two nfos, and I saw ucf2000.nfo in there which is nfo for 0day-group UCF. Zipclean's list is not alphabetically sorted either.

ZR-Tools logs nfos that are not present in either banned or accepted list. The log also shows exactly what file contains that unknown nfo. Only thing you have to do is check that log once in a while to see if anything new and suspicous had turned up. Any half-good siteop keeps these lists up-to-date.

ZR-Tools is working exactly as I wanted it to from the start. Of course I have more ideas on how to develop it even further, but I'm very grateful and happy for how ZR-Tools is working right now. But who knows, maybe it will updated some time in the future, if WarC feels it is worth it. ;)

When my banned and accepted lists have grown for some time I will probably attach them here. If anyone of you out there has a load of unwanted nfo-filenames that are still appearing here and there these days, feel free to update the ZR-Tools cfgs (alphabetically plz) and post them here. Suggestions are welcome even though there is no planned new version of the util.

Please let WarC know what wonderful work he has done! :)

[iXi]

moin,
thx WarC for this realy nice util;)

I've got little ioa request on nuke a dir can add some stuff that users thay are currently uploading / leeching the release will be kicked so that the rename on nuke works... all the time?

thx cya

[WarC]

1. I updated ZR-Tools. Its bugfix that affects in rare occasions.

NOTE!!! in the default config files there is a rule ".io*"
remove that ASAP! or it will remove all permissionfiles.


2. pRoLuCkY that will work when new io comes out. dark0ne disabled the function to kick some versions back but it is back in next one.


/WarC

[DR. GiGa BYTe]

Hi,

i still have the same problem as Giant had... although i added the comspec variable...

i use Windows 2003 Server
ioFTPd v4.6.2
ioA v0.7.4
ioZS v1.02
ZR-Tools v1.3

any ideas how i can solve this problem?

thanx
dr.gb

[Zer0Racer]

Changes to ioftpd.env requires restart (of ioFTPD). And make sure you've entered the correct ComSpec path. To find out which path you got simply open cmd.exe in windows (Start.. Run.. cmd) and type set and press enter.

[DR. GiGa BYTe]

Hi,

finally i found the problem... i'am using spaces in my "real" and my virtual file system.
because of the spaces in the file system the exe does not find the target file (the spaces in the vfs doesn't matter).

may it be possible to fix this? so i have not to re-organize my directory structure.


thanx
DR.GB

this is a sample of my vfs

"F:\Root" /
"E:\Root\11 GAMEz\15 GAME STUFF" /11 GAMEz/15 GAME STUFF
"G:\Root\12 M0ViEz" /12 M0ViEz
"E:\Root\15MiSC" /15 MiSC
"H:\Root\21 UPL0ADz" /21 UPL0ADz
"H:\Root\17 REQUESTz" /17 REQUESTz

[WarC]

thats the absolutly funkiest dirnames I've ever seen but that is your choice :)

I'll check on it as soon as I get the time. Not a really high prio though atm

/WarC

[WarC]

1.4 is up. Should hopefully take care of this matter for you.

/WarC

[DR. GiGa BYTe]

Great!!!

Thanx 4 the fast help!

cya
DR.GB

[Zer0Racer]

ZR-Tools works great with ioFTPD v4.9.0. Just so you all know. :)

[phoenixfr]

there is no reason why it shouldn't work:)

[odog]

great script....
wold be nice if you could add something that can disable it in some dirs....like /Requests/* for exaple...

[WarC]

Will look into this with ZeroRacer since he is the ruler of what to add and not :)

/WarC

[odog]

Even nicer would be if you could add diffrent ban dirs on diffrent sections... =)

[Zer0Racer]

I've been thinking about both the ability to exclude paths and make ZR-Tools handle sections separately. Both are good ideas.

Another thing I've been thinking about is a seperate list for banning files on upload (that wouldn't be used when cleaning paths manually with ZR-Clean.exe) so you can deny upload of for example file_id.diz (leaving it to the zipscript to extract one from zip). At the same time file_id.diz shouldn't be deleted when cleaning path. This would just be an extra list (in addition to the ZR-Banned.cfg) that would be checked prior to checking regular ban-list. And it would only apply to [Pre] ZR-ban. Naming it ZR-Ban-F.cfg?

Back to the first two ideas...

Being able to specify path(s) that should be excluded from clean and ban is needed. You shouldn't have to disable the whole script just to upload something in requests or a maintenance-dir.

Separate ban-lists for different sections is not crucial. But I still think it's a good idea. That feature would make it alot easier of enforcing your site's rules. Banning certain groups in different sections and blocking different dirs containing certain text, for example being able to ban *complete* in all sections but not in dvdr. Or whatever other examples you can come up with.

So.. my answers are:
Excluding paths: YES, absolutely
Separate sections: YES, if not now.. then later
ZR-Ban-F.cfg: YES

I will discuss this further with WarC. Then it's his call if these new features should be added in the near future or not. Depends on how much he has to study and work on his other projects. And if we can come up with logical way on how to implement these features.

[b>d>>s]

m, but it deletes rejected .nfo's from zips but still leaves them in the dir ? some get locked and are undeletable ? namely NF2K2.NFO

?

[Zer0Racer]

Strange. Deleting in dirs has always worked just fine for me. I just made a little test.zip containing a couple of files + a couple of unwanted nfo's including nf2k2.nfo.

C:\test>dir
Volume in drive C is WIN2KSP3
Volume Serial Number is 60B5-1CF6

Directory of C:\test

2003-06-29 22:40 <DIR> .
2003-06-29 22:40 <DIR> ..
2003-05-14 20:25 9_559 nf2k2.nfo
2003-06-29 22:40 73_034 test.zip
2 File(s) 82_593 bytes
2 Dir(s) 781_182_976 bytes free


And then running ZR-Clean.exe to clean it up...

C:\ioFTPD\scripts\ZR-Tools>ZR-Clean.exe c:\test
-[ZR-Clean]--------------------------
Processing Path...
DELETE in dir : c:\test\nf2k2.nfo
deleting: kane.nfo
deleting: tct.nfo
deleting: nf2k2.nfo
-------------------------------------


If you're referring to stopping files from being uploaded...

[Pre]
stor = ..\scripts\ZR-Tools\ZR-Ban-F.exe
mkd = ..\scripts\ZR-Tools\ZR-Ban-D.exe


Uploading same test.zip + the nf2k2.nfo in the dir...

STOR test.zip
150 Opening BINARY mode data connection for test.zip.
226--[ZR-Clean]--------------------------
226-Processing ZipFile...
226-deleting: kane.nfo
226-deleting: tct.nfo
226-deleting: nf2k2.nfo
226--------------------------------------
STOR nf2k2.nfo
426 Connection closed; transfer aborted.
226 This file is banned via ZR-Ban by WarC
Transfer Failed!


So it should work. Plz double-check that everything is configured OK and that all your unwanted files are in ZR-Banned.cfg

[neoxed]

@b>d>>s

Yes I have that problem to, but it's not just that NFO. Some releases
pack there ZIP files a little differently, and since ZR-Tools uses
some 5 year old Zip manager program to extract, delete, update
Zips it doesnt support most of the newer generation Zip archives.

D:\ioFTPD\scripts\ZR-Tools>ZR-Clean.exe D:\Test\Adobe.Acrobat.v6.0.Standard.REA.NFO-ROR
zip warning: local and central headers differ for (null)
zip warning: offset 0--local = 0a, central = 14

zip error: Zip file structure invalid (roras601.zip)

zip error: Zip file structure invalid (roras601.zip)

zip error: Zip file structure invalid (roras602.zip)

zip error: Zip file structure invalid (roras603.zip)

zip error: Zip file structure invalid (roras604.zip)
zip warning: local and central headers differ for (null)
zip warning: offset 0--local = 0a, central = 14

You only see the errors when executing from command line, not
in the msg during the FTP/FXP transfer.

Maybe in ZR-Tools v2.0 we could see the use of Info-ZIP's DLLs.
Or possibly internal ZIP support, RamRam (aka FTPServerTools) has
managed to do so in his tools.

I find ZR-Tools to be somewhat slow in performance, which I would
attribute to the external ZIP tool. By integrating ZIP support into
the tools I'm sure we could see a tremendous speed increase.

Though this is quite a bit of extra work to do, it was just an idea.

Peace ;)

[Zer0Racer]

ZR-Tools uses the easiest way possible (freeware) to complete the tasks it was supposed to do. The problem with files not being deleted from the certain zip-files are because they are compressed using PK-ZIP. Not even WinRAR 3.x can delete files from those zips, says archive is corrupt. The zip needs to be repaired first.

About built-in zip-support... sounds a bit too complicated, at least for this app. But I can always speak to WarC about it. As I've said earlier, it was never intended to be such a big deal. Just a simple app to carry out some simple instructions. Don't know about the performance either. Think it works fast enough. If not, I can wait.

Easiest way to get rid of the archive corrupt thing is to wait for new Info-ZIP version 3.x that hopefully will have support for PK-ZIP archives.

[b>d>>s]

ok, sorry. the non deletion of banned files is a clash with SnypeTEST's SioFTPD :<

nfos are deleted from zips ok, but the ban dir and ban file fails

Skipping: EchoBase.nfo
Skipping: infection.nfo
Skipping: LOA.NFO
Skipping: rsn.nfo

says skipped but they still end up in dir

which sux, so if u can get together or somit to sort this it would be nice, lol

thx

hm, seems same problem with b0unty banfile/dir .itcl

[Zer0Racer]

You must've configured something wrong. Are you absolutely sure that you've set this in ioFTPD.ini?

[Pre]
stor = ..\scripts\ZR-Tools\ZR-Ban-F.exe
mkd = ..\scripts\ZR-Tools\ZR-Ban-D.exe

[b>d>>s]

ZR-T dels the files from the zip, but SioFTPD then still manages to extract it to the dir

?

226--[ZR-Clean]--------------------------
226-Processing ZipFile...
226-deleting: wizard.nfo
226--------------------------------------

but its still in dir afterwards as well, this is without it being in the dir on upload but just in the zip ???

[b>d>>s]

0M9

ok, here is the problem

i am v sorry lmao

u need to put zrclean before the SioFTPD,

this has fixed it.

*cough*

all fixed now, just need to make sure zr runs b4 SioFTPD, i dont think this is in instructions (b>d>>s tries to cover his ass)

i think this is only due to SioFTPD not ZR-Tools, but i dont like to say for sure :P

:D

[Zer0Racer]

HEHE ok.. no wonder you didn't get it to work. :D

Btw.. why is SioFTPD extracting junknfos in the first place?

[b>d>>s]

y not?

lol

[WarC]

ZR-Tools 2.0 have been delivered to donators!

New in this:
+completly rewritten
+internal zipfile support(354% speedboost)
+No more "error in zipfile" in some 0day-releases
+suggested configs from above implemented. based per section and so on.
+excluded paths support
+ability to use multiple ban-lists in same section (for easy administration)
+tested by the one and only Zer0Racer

those that have received it enjoy :)

[neoxed]

Wow! That's awesome. :D

Really great app for a AppRIP site. :)

[Zer0Racer]

Yeah ZR-Clean is it anyway. ZR-Ban can be used on any site to enforce the site rules, making it impossible to upload unwanted stuff.

There is also a new little app in ZR-Tools 2.0 called ZR2F, ZR-Tools 2 FlashFXP. It converts your ban-lists into FlashFXP skiplist format. Very useful.

[neoxed]

Nice, when is the public release due? Can't wait to give this a try. :)

Peace

[WarC]

havn't decided yet about public version. perhaps never. All comes to if there are any donations. Donate and a copy will be sent or keep waiting :)

[neoxed]

Damn, it's not possible for me to donate by PayPal or any other credit card based service.

Do you accept any donations another way?

(Yes, I would donate if I could, ZR-Tools and ioA are great scripits.)

Peace :)

[Zer0Racer]

A little note on nfo-filenames. One spam nfo in banned list is BM.nfo. It is indeed a spam nfo. But the other day I noticed a group named BM and BMDOX that use nfo-filename BM.nfo too. Smart. Well now you know.

[Bratell]

Yep, I noticed it too, probably the same way you did. :)

[neoxed]

ZR-Tools v2 is much faster, great work. :)

How about a small option to disable the ZR-Clean.log & ZR-Clean.nfo.log log files in the .cfg file.

Peace ;)

[WarC]

Glad you like it.

That feature can probably be added in next version. If not Zer0Racer gives a big no but I think not. Toggles are lovly :)

[WarC]

ZR-Tools 2.1 have been sent to donators.

NeoXed I god damn forgot to add your toggle :( shame on me.

[neoxed]

No rush ;)

Awesome work though. :)

[neoxed]

[Scripts]
zipclean = %..\scripts\ZR-Tools\ZR-Clean.exe %[$path]


[Ftp-SITE-Permissions]
zipclean = MV


A quick site command for ZR-Tools to scan the current dir :p

[Zer0Racer]

l33t skillz

Edit: for newer versions of ioFTPD
[FTP_Custom_Commands]
zipclean = %EXEC ..\scripts\ZR-Tools\ZR-Clean.exe %[$path]


[FTP_SITE_Permissions]
zipclean = MV

[neoxed]

lol

[Pharaoh]

I am using v1.4 and have no problem to ban almost any file...But...
There seem to be a problem to ban files like this:
( ExTrA ) - ( 78MB 8F - COMPLETE )

I tried the following in ZR-Banned.cfg to no avail:
*ExTrA*
*ExTrA*.*
ExTrA*
ExTrA*.*

Am I doing something wrong? How can I ban this kind of files?

TIA. :)

[Zer0Racer]

There seems to be an issue when names containing special characters ie. [] () ! etc. Not sure how extensive this is. I found this out myself a couple of weeks ago.

Have you tried uploading a file or dir named test.extra.test or something, without special characters, and see if bans or not?

Gonna have to get back to you later on.

[Zer0Racer]

WarC has been working on ZR-Tools v3.0 for some time now and we feel it is ready for release (to donators). ZR-Tools is more or less completely rewritten. Partly forced by unfortunate hdd-crash for WarC. :( Alot of stuff disappeared. Well, it was for the better - eventually. ZR-Tools works even better now. I've been testing it for N number of weeks and WarC has been fixing the bugs that have turned up along the way. Just needs some minor adjustments in manual and it needs to be packaged. Should be ready this week.

What is new? I'll try and answer that as good as I can and WarC can correct me if I'm wrong.

ZR-Tools v2.x didn't work very well when banning files/dirs with special characters like () [] or !, and the use of * was dodgy. WarC made an all new compare routine thingy that works great! :D

New options for ZR-Clean:
Disable_Log=0
# 1 disables ZR-Clean.log

Disable_File_Log=0
# 1 disables ZR-Clean.nfo.log (that logs all new unknown files)

Extract_nfo=1
# 1 extracts any nfo from zip that is a match in ZR-Clean_Accepted.txt if one isn't already present in dir. Now your zipscript won't have to keep track of all nfos!

Extract_file_id.diz=0
# As above. Maybe not needed since any decent zs extracts the file_id.diz. Implemented for those who might find it useful.

Benchmark=1
# 1 adds line in output telling how fast ZR-Clean checked/cleaned zip. For me it took 30ms to clean 4 files in a 5MB zip on upload.

File_Filter=*.nfo,*.jpg,*.gif
# File extension of files that should be logged in ZR-Clean.nfo.log. Useful to find files like bday.jpg, bub.jpg, eclipse-shock.jpg etc.


Also new is customization. Here's how I think it should look by default. Interesting? :)
[customize]
ZR-Ban_Header ="426--[ZR-Ban]-------------------------------------------------------"
ZR-Ban_File_Banned ="426 File %file is banned!"
ZR-Ban_Directory_Banned ="426 Directory %directory is banned!"
ZR-Ban_Footer ="426-----------------------------------------------------------------"
ZR-Ban_Close ="426 Connection closed; transfer aborted."

ZR-Clean_Header ="\n-[ZR-Clean]---------------------------------------------------------"
ZR-Clean_File_Not_Found =" File not found"
ZR-Clean_Processing_Path =" Processing path..."
ZR-Clean_Processing_File =" Processing file..."
ZR-Clean_Delete_In_Zip =" Delete in zip: %filename"
ZR-Clean_Delete_In_Dir =" Delete in dir: %path\%filename"
ZR-Clean_Delete_File_Failed =" Delete failed: %path\%filename"
ZR-Clean_Extracted_File =" Extracted: %filename"
ZR-Clean_Footer ="--------------------------------------------------------------------"

ZR-Clean_Delete_In_Zip_Log ="Delete in zip: %path - %filename"
ZR-Clean_Delete_In_Dir_Log ="Delete in dir: %path - %filename"
ZR-Clean_Log_New_Nfo ="%filename <-- %path"

New is the \* at the end of paths (WarC's new compare routine thingy). Just as in ZR-Tools v2.x multiple paths and ban-lists can be used. Multiple ban-lists are very good for admins who want to keep different types of files in separate lists. I have group demo exes in one file, group crc check exes in another (if I suddenly want to clean them out of zips I just add the list in cfg), maybe keep jpgs in one list, one list for all kinds of tags you want to ban (* [* !* -=* #* etc. Multiple paths to create global rules that work in all sections and/or to cover sections that are physically in different locations like raided vfs dirs.

The ban-lists that are included (created from scratch by me personally) contain 270+ filenames at this time. List of accepted nfos contains 210+ filenames.

That's about it. Oh yeah, ZR-Clean.exe can be run externally from command prompt to cleanup your current 0day site/archive. And it removes any zip comment.

Just imagine how much you can do with all of this. :)

Stay tuned...
/Zer0Racer


Show your support and get this great script. And WarC needs new hdd. :)
WarC's utils: http://warc.mine.nu/


EDIT: ZR-Tools v3.0 is out now to donators.

[DR. GiGa BYTe]

Nice...
The current version works great!
I'm really looking forward for the next version... let's see if you can top the last rls :D

[neoxed]

Looks awesome :D

[WarC]

I have now send new version to donators. If any of you didn't recieve it please let me know. Must say its a nice version that seems to work very well :)

Big thanks to ZeroRacer for all his testing and so on.

[The0ne]

nice :)

remind me when i have spare $$$ to donate, a little strapped on cash atm hehe.

[Optixxx]

*********************************************

Thx WarC for sending me the package

A honor to install it :)

greetz Optixxx

[WarC]

zr-tools 3.1 is now added to foundationlicense type and can be grabbed from memberpages.

I really hope that those that like it spend a buck on me :)

[WarC]

ZR-Tools 3.2 now added to memberarea.

Fixes when having multiple banlists for sections.

[Zer0Racer]

Just some more or less important info about ZR-Tools not stated in the manual.

You must use comma separated lists when specifying multiple banlists and/or paths. Do not separate by spaces.

Path=d:\ftp\upload\* e:\ftp\no_crc\* <-- INCORRECT!

Path=d:\ftp\upload\*,e:\ftp\no_crc\* <-- CORRECT!

And when adding ZR-Clean and ZR-Ban in ioFTPD.ini don't forget to use EXEC.

[WarC]

New version 3.3 is out for foundationusers.

[WarC]

Theres a new version coming of ZR-Tools....

New ziplib:
used (should take care of some odd zipfiles)
Info-zip will be used this time
New tool:
ZR-Search, used for searching for files in a path recursivly
and if wanted delete them. Also inside zip-files
Rewrite:
Due to the change of ziplib lots of the other subsystems
have also been modified to be faster and better

/WarC

[Linkster]

WarC, I fixed your account so you can now upload your scripts :)

[WarC]

thank you :)

[zpr]

awesome, new site, forum, io is coming back with scripts :)

[WarC]

New version is out. 3.5!

New features, new configoptions and more.

[SvensK]

Still rocks, thanks WarC.

[ESREVER]

@WarC It's sounds amazing -but I can't test it yet. The older releases are gone and the new one says:"Only available to Foundation users of ioFTPD". I don't know but as I downloaded ioFTPD I haven't paid for it, because it's OpenSource. Do you want to earn money with this script? SURE! Than give a link where the money goes by and I'll see... I'm interested in your work. cu <e>