Go Back   FlashFXP Forums > > > >

ioFTPD General New releases, comments, questions regarding the latest version of ioFTPD.

Reply
 
Thread Tools Rate Thread Display Modes
Old 08-21-2004, 07:37 PM   #16
Syrinx
Junior Member
 
Join Date: Jun 2004
Posts: 25
Default

I still get some questions about that result based on my knowledge of hardware.
1.what's the speed of the NIC?and 253 mb /sec means 253 mega byte per second or 253 mega bit per second?
2.The lastest(the fastest) harddisk can barely reach 80MB/sec reading and writing speed in a very ideal environment(mostly at 20 MB/s ~ 70 MB/s).Even the daemon can have that transfer speed,but I don't think there is harddisk can continiously write or read data at 127 mb/s.
Syrinx is offline   Reply With Quote
Old 08-22-2004, 03:24 AM   #17
ADDiCT
Senior Member
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: Aug 2003
Posts: 517
Default

When u read a file for the first time, windows keeps it cached in ram. When u read it again, it comes from the cache. On my system, i can read a 100MB cached file at about 1 GB/s.

When u want to test the maximum throughput of ioFTPD, u don't use harddisks, because those are the slowest component. U transfer a cached file, then the limit is 1 GB/s.

The client in this case, doesn't write the file to the harddisk, it just downloads it and then discards it.
ftp.exe: get file > nul

The speed of the NIC is unimportant here, because the tests are local transfers. Localhost -> localhost doesn't go through any network adaptor. (This way, u can develop and test networking applictions without being connected to a network). The speeds darkone got were most likely 253 MB/s and not 253 Mbit/s.

Please correct me if i'm mistaken
ADDiCT is offline   Reply With Quote
Old 08-22-2004, 01:02 PM   #18
Zer0Racer
Senior Member
ioFTPD Scripter
 
Join Date: Oct 2002
Posts: 703
Default

:drool:
Zer0Racer is offline   Reply With Quote
Old 08-22-2004, 01:26 PM   #19
darkone
Disabled
FlashFXP Registered User
ioFTPD Administrator
 
darkone's Avatar
 
Join Date: Dec 2001
Posts: 2,230
Default

Remember that I'm not developing this daemon (core) only with ide/sata disks and dsl connections in mind

And yes, figure was 253mb/sec... and I've managed to get few more mbytes more, so it's now closer to 270 than 250 Major problem with these tests, is that winsock is not able to use hardware acceleration of NIC to create tcp packets (crc-16 etc). Also, the test includes buffer copying that would not exist in real world situation, as core is now able to assign transfer buffers directly to hardware without going through winsock buffers.
darkone is offline   Reply With Quote
Old 08-26-2004, 10:17 AM   #20
darkone
Disabled
FlashFXP Registered User
ioFTPD Administrator
 
darkone's Avatar
 
Join Date: Dec 2001
Posts: 2,230
Default

Huhuu... I've broken the 300mb/sec barrier. All I had to do, is either to disable hyperthreading or set program's thread affinity to represent best case scenario. Excat transfer time for 800mb file was:

1 transfer: 5.1 seconds (313mb/sec)
2 transfer: 10.13 seconds (315mb/sec)

I'm currently finalizing the base api, and moving on to add ssl within a few days.
darkone is offline   Reply With Quote
Old 08-27-2004, 05:13 AM   #21
peep
Senior Member
FlashFXP Scripter
ioFTPD Foundation User
 
Join Date: Sep 2003
Posts: 132
Default

You're mad :-)
That's all I've got to say. Keep up the amazing work d1. sad to see I won't be around when the new core goes public (going away for about a year). Good luck with all this and I'll be dying to try it out when I get back!

peep
peep is offline   Reply With Quote
Old 08-27-2004, 07:37 AM   #22
richto
Member
FlashFXP Registered User
ioFTPD Foundation User
 
Join Date: Jan 2003
Posts: 59
Default

315 MB a sec? So thats roughly a 2.5 GBit connection - not bad!
richto is offline   Reply With Quote
Old 08-27-2004, 07:56 AM   #23
darkone
Disabled
FlashFXP Registered User
ioFTPD Administrator
 
darkone's Avatar
 
Join Date: Dec 2001
Posts: 2,230
Default

Yep, and because crc calculations, buffer splitting etc. are now done by cpu, it should be actually faster when it goes through NIC. (depending on driver performance & hardware acceleration capabilities of the card)

My integrated NIC (Intel PRO/1000 CT) seems to have following hardware capabilities:
- QOS managment
- Hardware flow control
- Up to 1024 receive buffers
- Up to 1024 send buffers
- TCP checksum (receive & send)
- IP checksum (receive & send)
- And some more.. (stupid thing shows them in finnish )

Which means that use of TCP protocol shouldn't add any overhead to CPU, when compared to sending RAW packets.
darkone is offline   Reply With Quote
Old 08-27-2004, 01:04 PM   #24
Mr_X
Senior Member
FlashFXP Registered User
ioFTPD Foundation User
 
Join Date: Sep 2003
Posts: 142
Default

I have a few questions:

How is it possible to improve so much performance? I don't think there is a lot of way to read a file and transfer it to a socket (perhaps I'm wrong).
What hardware do you use to get transfer so fast (MB, CPU, Hard disk, raid controller, RAM, network cards, ...)?
What windows version do you recommand to get best speed?
Could you explain how to code such function or do you have a website that explain it:
Code:
lpProc[2] = { DoSomething1, DoSomething2 };

void foo(int lOperation)
{
  lpProc[lOperation]();
}
Mr_X is offline   Reply With Quote
Old 08-27-2004, 03:21 PM   #25
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

Quote:
Originally posted by Mr_X
What hardware do you use to get transfer so fast (MB, CPU, Hard disk, raid controller, RAM, network cards, ...)?
If you were reading the rest of the thread, both darkone and ADDiCT mentioned these tests are done with the file cached in memory. See ADDiCT's post above.

Quote:
Originally posted by Mr_X
What windows version do you recommand to get best speed?
All 5.x versions of Windows (2000/XP/2003) should preform somewhat similar. Though, I'm sure Windows Server 2003 has a few under-the-hood tweaks that previous versions do not.
neoxed is offline   Reply With Quote
Old 08-27-2004, 03:45 PM   #26
darkone
Disabled
FlashFXP Registered User
ioFTPD Administrator
 
darkone's Avatar
 
Join Date: Dec 2001
Posts: 2,230
Default

How is it possible to improve so much performance? I don't think there is a lot of way to read a file and transfer it to a socket (perhaps I'm wrong).

On windows there are several ways of reading from handle; synchronous (ReadFile()), asynchronous , ReadFileEx(), overlapped ReadFile() using events, overlapped ReadFile() using io completion ports.. not to mention options that are available for sockets). New core uses the most advanced method available; single completion port, several simultanous reads/writes per socket/file, and several threads polling for the completion port. It makes it possible, for several cpus to process notifications that arrive in one completion port. I can't give latest functional sources for you to see, but what I can give is: http://www.ioftpd.com/~darkone/tmp/newcore.txt ... that's the first implementation of async processing line that I did. (latest version is much better, and in most cases it can process the operations inside thread safe areas within one quantum)


What hardware do you use to get transfer so fast (MB, CPU, Hard disk, raid controller, RAM, network cards, ...)?

2x 2.66Ghz Xeon, 2048Mb (Dual channel), cached file, local transfer (no NIC involved, which causes worse results that you'd get with a transfer through NIC)


What windows version do you recommand to get best speed?

Any version of windows is ok, just make sure you have you set windows to use memory for system cache rather than using it for programs. Server editions of windows have longer socket queue, which makes it better suitable for server use. However, io rarely needs listen queue - as it can accept up to 10connections simultanously - 10AcceptEx() requests are pending on each service socket. But in case you're going to run it as httpd server, which gets lots of short lived connections - you might hit the wall using XP or 2000 Professional.


ld you explain how to code such function or do you have a website that explain it:

It's standard function to pointer declaration. In some cases it's slower: if function can be inlined, and there's only few operators in array. However ie. php source code could be optimized this way, I remember seeing switch() operators of like 200 members there.... you can just imagine how long it takes to get to the last operator.
darkone is offline   Reply With Quote
Old 08-27-2004, 05:46 PM   #27
Mr_X
Senior Member
FlashFXP Registered User
ioFTPD Foundation User
 
Join Date: Sep 2003
Posts: 142
Default

Thx for those informations.
New io seems promising
Mr_X is offline   Reply With Quote
Reply

Tags
called, core, function, read, socket

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 07:19 PM.

Parts of this site powered by vBulletin Mods & Addons from DragonByte Technologies Ltd. (Details)