Do me a favor and try my dupechecker with such an amount of files. Upload checking of 60000 files should take about 17 reads in a file, meanign within 0.2 seconds it can check a dupe out of 60000 files and yet it is still in a simple SORTED!! list. It works on dirs but I can extend it to work on files. The drawback is tho that after a save it takes more time to process... Have you considered using sqlite with an index in tcl? There is a tcl extension for sqlite that might give you the stuff you need. You would need some tcl scripting then tho. 60000 files with an average of lets say 14 characters is only a measly 830K file which is basically a small file. Reading such a file can be done super quickly. You can use DupeLister to make new dupelists. Please let me know if it is fast enough for you. I have been given reports of 400000 entries in it being handled within 2 seconds, so I assume in your case it'd be fast enough. If it is let me know then I'll see if I can add file support to it as well (shouldnt be hard at all).
DupeSearch and DupeLister are what you need for testing. OnDirCreated does dupe dir blocking I can make a OnPreFileUpload or something like that that tests a filename against a list like being created with DupeLister.
|