Getting recursive listings from ioftpd?

Tillmann · 11-22-2004, 09:12 AM

Hi,

I'm the author of FileArchiveSearch and SMBSpider . I'm currently adding FTP indexing capabilities to these search tools. For this, I need to get a recursive listing of the content of the ftp server.

No big deal, since I can request that from the server the standard way, using "LIST -lR". That works fine with glftpd, Serv-U, etc.

Problem is: It doesn't work with ioftpd. I did some further testing, and it seems that ioftpd simply ignores any arguments passed to LIST (e.g. LIST -t for sorting by time doesn't work - no big deal, since most ftp clients can sort directories on the client side).

Does anybody know a workaround to that problem?

A possible workaround would be to make a site command (e.g. SITE rls) which will call a normal "ls" binary (e.g. from cygwin). Has anybody done such a thing?

Of course such a workaround wouldn't consider the ioftpd VFS. In the long term, a LIST implementation that is on par with other FTP servers would be desirable.

bye,
Tillmann

esmandil · 11-22-2004, 09:36 AM

It has been discussed before:

http://www.ioftpd.com/board/showthre...recursive+list

thewarden · 11-23-2004, 07:32 AM

Most search \ sort \ and so on for ALL ftpd's are a wasted effort from the end users USE based on 2 main factors - speed and volume of "search"

There is only 1 item that works 100% perfect.

I understand your question, and good luck...

But all the ftpd's I am associated (io \ raiden \ serv-u) with that have small sub 2 terrabvytes online to as high as 6+ terrabytes online use one program that scans such sites in less than 30 seconds across a local lan of 2+ servers...

DO a search of FTPSERVERTOOLS and look for VFSLISTER and that is the item that does the job...setting it to run off every 15 mins or in 30 min time slots has no effect on the system and the lists are basicly fresh as can be.

Each list of every dir or file can be downloaded in 2-10 seconds and viewed in ex: ULTRA EDIT via what you are searching for and 90% of the time - this useage is faster than the ftpd results and far more precise.

I run this off on my site and 4 terrabytes takes a few seconds on standard dir listings - in depth dir\file takes about 30-40 seconds and that is of 10,000's of DIR's ... on the raid 5 array of 1500 gigs of RIP style - done in under 40 seconds and that is 5000+ dir's per a to z dir = 130,000+ dir's (test bed here for a business)

Might not be the answer your looking for - but this is the ONLY program I found that does the job...

Tillmann · 11-23-2004, 08:17 AM

Hi,

I know that most search utilities for ftpds suck, and that they have problems with speed and volume of search. That is why I've written my own (FAS/SMBSpider). It is blazingly fast, even if the index is substantially larger than 1 TB, and even if wildcards are used in search. My tool can also include offline media (CD, DVDR - like a web-based WhereIsIt) and SMB shares (via an SMB spider written in Python). And it comes with a web interface that is easy to use and platform independed as far as the client side is concerned. Damn, now I sound like a marketing guy.

Anyways - it just doesn't work with ioftpd, and that is the reason why I have posted this.

So, if I use vfslister, I get a dirlisting in the same format as a standard "ls -lR"?

bye,
Tillmann

thewarden · 11-23-2004, 10:00 AM

Quote:

Originally posted by Tillmann

Anyways - it just doesn't work with ioftpd, and that is the reason why I have posted this.

So, if I use vfslister, I get a dirlisting in the same format as a standard "ls -lR"?

bye,
Tillmann

Exactly as it is on the hard drive - and take a look at the options as it lets you combine ex:

c:\games
d:\games
and so on into a single ascii file sorted A to Z...it has depth and other items as well...best tools to add to any ftps are ftpservertools as they work where other do not.

Tillmann · 11-23-2004, 10:04 AM

Hi,

my question was, is the output formatted like ls -lR (LIST output from ftp server).

E.g.:
drwxrwxrwx 14 ioftpd ioftpd 4096 2004-11-23 16:02 games

bye,
Till

mr_F_2 · 11-23-2004, 12:52 PM

hey tillerman,

why don't you just write your own recursive-spanning routine?
You already have software that you're distributing publically, so you probably have the skill to write such a routine -- it's easier than asking all the FTPd server coders if they support list -R or are going to in the future, and it makes your program more scalable.

Tillmann · 11-23-2004, 01:09 PM

Hi,

right now I'm using lftp to retrieve listings. lftp is scriptable, but not to the extent that a recursive-directory-listing could be implemented.

So I'd have to write my own ftp client. But, even though the ftp standard is quite simple, that is a non-trivial task. Anybody who has tried to write an ftp client will confirm this (that's why there are countless ftp clients around, but 95% of them suck). There's also ftp libraries available for perl, but they won't support encrypted control connection/dirlistings (unless a special proxy is used which will take care of encryption; but that's a workaround not a solution).

Also, it's not a problem with "all the other ftp servers". It works with other ftp servers. With glftpd, with wuftpd, with proftpd, with pureftpd, even with crappy Serv-U.

It just doesn't work with ioftpd.

bye,
Till

esmandil · 11-23-2004, 04:09 PM

Quote:

Originally posted by Tillmann

There's also ftp libraries available for perl, but they won't support encrypted control connection/dirlistings

bye,
Till

I don't know if you care, but maybe somebody else can find this information useful: there is an ftp library for python which support encryption. It is pretty cool stuff

Syrinx · 11-28-2004, 02:02 PM

"This is unfortunately impossible, as in current form ioFTPD caches whole directory list before sending it to client. => Recursive listing would result to _huge_ buffer." from http://www.ioftpd.com/board/showthre...recursive+list

I am just curious that if a directory has many files such as 12345678 files,would ioftpd cache the entire file list before sending it?