After reading http://www.python.org/dev/peps/pep-0371/ I was under
impression that performance of multiprocessing package is similar to
that of thread / threading. However, to familiarize myself with both
packages I wrote my own test of spawning and returning 100,000 empty
threads or processes (while maintaining at most 100 processes / threads
active at any one time), respectively.

The results I got are very different from the benchmark quoted in PEP
371. On twin Xeon machine the threaded version executed in 5.54 secs,
while multiprocessing version took over 222 secs to complete!

Am I doing smth wrong in code below? Or do I have to use
multiprocessing.Pool to get any decent results?

# multithreaded version

#!/usr/local/python2.6/bin/python

import thread
import time

class TCalc(object):

         def __init__(self):
                 self.tactivnum = 0
                 self.reslist = []
                 self.tid = 0
                 self.tlock = thread.allocate_lock()

         def testth(self, tid):
                 if tid % 1000 == 0:
                         print "== Thread %d working ==" % tid
                 self.tlock.acquire()
                 self.reslist.append(tid)
                 self.tactivnum -= 1
                 self.tlock.release()

         def calc_100thousand(self):
                 tid = 1
                 while tid <= 100000:
                         while self.tactivnum > 99:
                                 time.sleep(0.01)
                         self.tlock.acquire()
                         self.tactivnum += 1
                         self.tlock.release()
                         t = thread.start_new_thread(self.testth, (tid,))
                         tid += 1
                 while self.tactivnum > 0:
                         time.sleep(0.01)

if __name__ == "__main__":
         tc = TCalc()
         tstart = time.time()
         tc.calc_100thousand()
         tend = time.time()
         print "Total time: ", tend-tstart

# multiprocessing version

#!/usr/local/python2.6/bin/python

import multiprocessing
import time

def testp(pid):
         if pid % 1000 == 0:
                 print "== Process %d working ==" % pid

def palivelistlen(plist):
         pll = 0
         for p in plist:
                 if p.is_alive():
                         pll += 1
                 else:
                         plist.remove(p)
                         p.join()
         return pll

def testp_100thousand():
         pid = 1
         proclist = []
         while pid <= 100000:
                 while palivelistlen(proclist) > 99:
                         time.sleep(0.01)
                 p = multiprocessing.Process(target=testp, args=(pid,))
                 p.start()
                 proclist.append(p)
                 pid += 1
         print "=== Main thread waiting for all processes to finish ==="
         for p in proclist:
                 p.join()

[quick browsing through the code]

Ah, so there are 100 processes at time. 200secs still don't sound
strange.

Linux SLES (9.156.44.174) [15:18] root ~/tmp/src # ./run_benchmarks.py
empty_func.py

Importing empty_func
Starting tests ...
non_threaded (1 iters)  0.000005 seconds
threaded (1 threads)    0.000235 seconds
processes (1 procs)     0.002607 seconds

non_threaded (2 iters)  0.000006 seconds
threaded (2 threads)    0.000461 seconds
processes (2 procs)     0.004514 seconds

non_threaded (4 iters)  0.000008 seconds
threaded (4 threads)    0.000897 seconds
processes (4 procs)     0.008557 seconds

non_threaded (8 iters)  0.000010 seconds
threaded (8 threads)    0.001821 seconds
processes (8 procs)     0.016950 seconds

The limiting factor is not the creation time but the communication and
synchronization overhead between multiple threads or processes.

On my iMac 2.3Ghz dualcore. python 2.6

iTaulu:src torppa$ python run_benchmarks.py empty_func.py
Importing empty_func
Starting tests ...
non_threaded (1 iters)  0.000002 seconds
threaded (1 threads)    0.000227 seconds
processes (1 procs)     0.002367 seconds

non_threaded (2 iters)  0.000003 seconds
threaded (2 threads)    0.000406 seconds
processes (2 procs)     0.003465 seconds

non_threaded (4 iters)  0.000004 seconds
threaded (4 threads)    0.000786 seconds
processes (4 procs)     0.006430 seconds

non_threaded (8 iters)  0.000006 seconds
threaded (8 threads)    0.001618 seconds
processes (8 procs)     0.012841 seconds

With python2.5 and pyProcessing-0.52

iTaulu:src torppa$ python2.5 run_benchmarks.py empty_func.py
Importing empty_func
Starting tests ...
non_threaded (1 iters)  0.000003 seconds
threaded (1 threads)    0.000143 seconds
processes (1 procs)     0.002794 seconds

non_threaded (2 iters)  0.000004 seconds
threaded (2 threads)    0.000277 seconds
processes (2 procs)     0.004046 seconds

non_threaded (4 iters)  0.000005 seconds
threaded (4 threads)    0.000598 seconds
processes (4 procs)     0.007816 seconds

non_threaded (8 iters)  0.000008 seconds
threaded (8 threads)    0.001173 seconds
processes (8 procs)     0.015504 seconds

non_threaded (1 iters)  0.000004 seconds
threaded (1 threads)    0.000159 seconds
processes (1 procs)     0.001067 seconds

non_threaded (2 iters)  0.000005 seconds
threaded (2 threads)    0.000301 seconds
processes (2 procs)     0.001754 seconds

non_threaded (4 iters)  0.000006 seconds
threaded (4 threads)    0.000581 seconds
processes (4 procs)     0.003906 seconds

On my system, as measured by timeit, spawning and joining a thread
takes 111 usecs, while forking and waiting for a process takes 260.
Slower, but not catastrophically so.

As others have mentioned, it's not suprising
that spawning even 100 processes took some
time.

Bottom line: multiprocessing should not be used this way.
(nor should threading).

You might also find it interesting to examine a theoretical OS that is
optimized for process overhead.  In other words, what is the minimum
overhead possible?  Can processes be as small as threads?  Can entire
threads be only a few bytes (words) big?

I'm presently looking at Virtual Synchrony and
other distributed processing architectures - but
circuits is meant to be general purpose enough
to fit event-driven applications/systems.

cheers
James

I have often seen generators used to
facilitate coroutine and coooperative
programming though.

In Economics, you can divide a market into vertical and horizontal
dimensions, vertical being the chain from raw resources to finished
products, and horizontal being market coverage.  With a similar
division in tasks, horizontal units would handle incoming events,
prepare them, then pass them on to a next unit, which processes a
little, then passes it on, like an assembly line (bucket brigade/
alternating current); and vertical units would take one incoming
event, and see it through start to finish (direct current).  You don't
have to use that depiction.

The terminology is transposed from that of a distributed matrix
multiplication task depiction, where horizontal works start-to-finish,
and vertical takes handoffs from its predecessor.

(snip)

That being said, circuits employs both an event-driven
approach as well as a Component architecture.
In your analogy it is both horizontal and vertical.

As I continue to develop circuits and improve it's
core design as well as building it's ever growing set
of Components, I try to keep it as general as
possible - my main aim though is distributed
processing and architectures. (See the primes example).

Thanks for sharing your interest :)

On Terminal A:

jmills@atomant:~/circuits/examples$ ./primes.py -o primes.txt -b
127.0.0.1:8000 -p 1000 -w

Total Primes: 1001 (23/s after 44.16s)
Total Events: 43373 (983/s after 44.16s)
Distribution:
 c1096b40-7606-4ba7-9593-1385e14ef339: 348
 8313b43f-d45d-4a0a-8d87-e6a93d3dfb0b: 653

On Terminal B:

jmills@atomant:~/other/circuits/examples$ ./primes.py -b
127.0.0.1:8001 -s 127.0.0.1:8000

The example uses circuits to distribute work
amongst any arbitrary number of nodes connected
to the system. This is all manually implemented
by simply taking advantage of the circuits framework
with my basic understanding ( so far ) of distributed
processing, virtual synchrony, and other techniques.

This is the same thing just run on a single node (no
distribution):

jmills@atomant:~/circuits/examples$ ./primes.py -o primes.txt -b
127.0.0.1:8000 -p 1000

Total Primes: 1001 (17/s after 62.13s)
Total Events: 28579 (460/s after 62.13s)
Distribution:
 701be749-9833-40a4-9181-5ee18047b1ad: 1001

As you can see, running 2 instances almost halves
the time. If you do try this though, you'll note that
the CPU isn't being used very heavily at all - This
could be improved - but the example merely
demonstrates distributed event processing and
syncronization.

On Dec 31, 2008, at 7:19 PM, Paul Rubin wrote:

http://semanchuk.com/philip/sysv_ipc/
http://semanchuk.com/philip/posix_ipc/

The problem with your code is that you never start more than one
process at once in the multiprocessing example.  Just check ps when it
is running and you will see.

My conjecture is that this is due to the way fork() works under unix.
I think that when the parent forks it yields the CPU to the child.
Because you are giving the child effectively no work to do it returns
immediately, re-awakening the parent, thus serialising your jobs.

If you give the children some work to do you'll see a quite different
result.  I gave each child time.sleep(1) to do and cut down the total
number to 10,000.

$ ./test_multiprocessing.py
== Process 1000 working ==
== Process 2000 working ==
== Process 3000 working ==
== Process 4000 working ==
== Process 5000 working ==
== Process 6000 working ==
== Process 7000 working ==
== Process 8000 working ==
== Process 9000 working ==
== Process 10000 working ==
=== Main thread waiting for all processes to finish ===
Total time: 101.382129192

$ ./test_threading.py
== Thread 1000 working ==
== Thread 2000 working ==
== Thread 3000 working ==
== Thread 4000 working ==
== Thread 5000 working ==
== Thread 6000 working ==
== Thread 7000 working ==
== Thread 8000 working ==
== Thread 9000 working ==
== Thread 10000 working ==
Total time:  100.659118176

So almost identical results and as expected - we ran 10,000 sleep(1)s
in 100 seconds so we must have been running 100 simultaneously.

If you replace the "time.sleep(1)" with "for _ in xrange(1000000):
pass" you get this much more interesting answer on my dual core linux
laptop, showing nicely the effect of the contention on the python
global interpreter lock and how multiprocessing avoids it.

$ ./test_multiprocessing.py
== Process 1000 working ==
== Process 2000 working ==
== Process 3000 working ==
== Process 4000 working ==
== Process 5000 working ==
== Process 6000 working ==
== Process 7000 working ==
== Process 8000 working ==
== Process 9000 working ==
== Process 10000 working ==
=== Main thread waiting for all processes to finish ===
Total time: 266.808327913

$ ./test_threading.py
== Thread 1000 working ==
== Thread 2000 working ==
== Thread 3000 working ==
== Thread 4000 working ==
== Thread 5000 working ==
== Thread 6000 working ==
== Thread 7000 working ==
== Thread 8000 working ==
== Thread 9000 working ==
== Thread 10000 working ==
Total time:  834.81882