This time I felt too lazy to draw example of command flow using ms paint (I wonder if anyone actually understud any of my earlier drawings)... so instead I decided to write ascii version
There are two common approaches to create application that handles large number of simultanous tasks equally.
First, the easy one, is to create thread for each task and let the OS handle load balancing between the tasks.
Second, the hard one, is to split large task into several smaller tasks, and execute these smaller tasks one by one using pool of threads (or in parallel when possible). Next task is usually queued using First In First Out logic at the end of current task. Once thread completes one task, it begins to process next queued task - if any. Because new tasks are placed to end of queue, application is able to process any number of large tasks simultanously by processing these smaller tasks without using excessive amounts of threads.
Code:
Example of splitting large task (from 1.0):
--------------------------------+-------------------------------+---------------+
Task | Executing thread type | Thread switch |
--------------------------------+-------------------------------+---------------+
Start service | Worker thread | |
Create listen queue | Transfer processing thread | Yes |
Accept new connection | Transfer thread | |
Process new connection | Transfer processing thread | Yes |
Create new listen queue item | Transfer processing thread | |
Call common service callback | Transfer processing thread | |
Hostname lookup | Worker thread | Yes |
Call service specific callback | Worker thread | Yes |
... | ?? | |
--------------------------------+-------------------------------+---------------+
?) On thread switch, task is sent from one thread to another.
Thread types in 1.0:
--------------------------------+---------------------------------------+
Thread type | Thread count |
--------------------------------+---------------------------------------+
Worker thread | Logical CPU count * 2 + N (!1) |
Transfer thread | Physical CPU count * 2 (!2) |
Transfer scheduler thread | 1 |
Transfer processing thread | Physical CPU count |
Timer thread | 1 |
Window message thread | 1 |
--------------------------------+---------------------------------------+
!1) 'Worker thread' pool may grow temporarily, if all threads are in use and one ore more
threads is performing long blocking operation (THREAD_FLAG_BLOCKING is set).
!2) Half of 'transfer threads' may switch type on the fly to 'transfer processing thread'.
'Transfer threads' should not perform blocking or CPU intensive operations, instead they
should queue such operations to 'transfer processing threads'.
'Transfer processing threads' revert back to 'transfer threads', when there are no
more queued operations.
Boring.. hope I can get some sleep after writing this
