Need help with threads

[Damian Menscher]

I'm trying to use threads to run various processes that require some
I/O. This is apparently broken. Here's a simple program:

=================================================

#!/usr/bin/python
import time
import thread
import popen2

def b(n):
print "running b(%s)"%n
r,w = popen2.popen2('cat')
w.write("test %s"%n)
w.close()
print "process %s read %s"%(n,r.readline())

for n in range(10):
thread.start_new_thread(b, (n,)) ### with threads
#b(n) ### without threads

time.sleep(1) ## Allow some time for threads to finish

=================================================

So this should start up, fork off 10 threads that just do basic stuff,
then wait a second for everything to finish.

Running *without* threads (by uncommenting the b(n) line) shows
typical (correct) execution:

[menscher@qcdhome t2]$ ./threads.py
running b(0)
process 0 read test 0
running b(1)
process 1 read test 1
running b(2)
process 2 read test 2
running b(3)
process 3 read test 3
running b(4)
process 4 read test 4
running b(5)
process 5 read test 5
running b(6)
process 6 read test 6
running b(7)
process 7 read test 7
running b(8)
process 8 read test 8
running b(9)
process 9 read test 9

But running *with* threads gives me:

[menscher@qcdhome t2]$ ./threads.py
running b(6)
running b(2)
running b(4)
running b(0)
running b(5)
running b(1)
Killed

Where the job hogged 99% of CPU and had to be killed (even a ^C
didn't stop it).

I'm guessing there might be a namespace conflict in the I/O, but I'm
new to python in general and can't develop a workaround.

While I'm posting... is there an elegant way to wait till all threads
have finished? I can't count on things finishing in a fixed time for
my application.

Ideas? Please??

Damian Menscher
--
--==## Grad. student & Sys. Admin. @ U. Illinois at Urbana-Champaign ##==--
--==## <mens...@uiuc.edu> www.uiuc.edu/~menscher/ Ofc:(217)333-0038 ##==--
--==## Physics Dept, 1110 W Green, Urbana IL 61801 Fax:(217)333-9819 ##==--

[Aahz Maruch]

In article <_nOV6.9330$ki5.1...@vixen.cso.uiuc.edu>,

Damian Menscher <mensche...@uiuc.edu> wrote:
>
>I'm trying to use threads to run various processes that require some
>I/O.

http://starship.python.net/crew/aahz/
--
--- Aahz <*> (Copyright 2001 by aa...@pobox.com)

Hugs and backrubs -- I break Rule 6 http://www.rahul.net/aahz/
Androgynous poly kinky vanilla queer het Pythonista

"Just because I'm selling you into slavery doesn't mean we can't be
friends." B-movie cliche Hall of Fame (_Bounty Hunter: 2002_)

[Jeff Shannon]

Damian Menscher <mensche...@uiuc.edu> wrote in message news:<_nOV6.9330$ki5.1...@vixen.cso.uiuc.edu>...

> I'm trying to use threads to run various processes that require some
> I/O. This is apparently broken. Here's a simple program:
>
> =================================================
>
> #!/usr/bin/python
> import time
> import thread
> import popen2
>

<snip>

> While I'm posting... is there an elegant way to wait till all threads
> have finished? I can't count on things finishing in a fixed time for
> my application.
>
> Ideas? Please??
>
> Damian Menscher

My suggestion would be to use the 'threading' module instead of the 'thread'
module--it's a higher level, cleaner interface.

import threading

def ThreadFunc(my_arg):
print "Thread given argument of: %s" % str(my_arg)

for n in range(10):
threadname = "MyThread%d" % n
newthread = threading.Thread( target=ThreadFunc, name=threadname, args=(n,) )
newthread.start()

the threading module defines a Thread class whose constructor takes several
keyword arguments, some of which are optional. Most important is the target
argument, which is the function that comprises the body of the thread. This
function is passed a tuple of arguments specified by args. Check the
docstrings and help file of the threading module for more details.

Also, if you wish to pause until a particular thread is finished, you can
use the Thread.join() method, which also has an optional timeout parameter--
i.e., mythread.join(30) will pause until mythread finishes execution or 30
seconds have passed, whichever comes first, thus allowing you to have a chance
to kill or recover from a deadlocked thread. (Timeouts are a good thing.)

p.s. I suspect that your problem may have been due to some subtle deadlock
or race condition between your threads, though I'm not sure. Any time any
thread accesses a global resource, it should be protected by some sort of
lock or semaphore. Technically, this probably applies even to the print
statement in my example, but..... :)

Jeff Shannon
Technician/Programmer
Credit International