Multiple thread program problem

[Mohan Mohta]

Hello
I am trying to create multiple thread through the below program but I am getting an error

#! /usr/bin/python
import os
import subprocess
import thread
import threading
from thread import start_new_thread

def proc(f) :
com1="ssh -B "
com2=line.strip('\n')
com3= " uname -a"
co=str("ssh -B ")+ str(com2) + str(" uname -a")
subprocess.call(co,shell=True)
print "----------------------------"
return

f = open('/tmp/python/1')
for line in f:
t=thread.start_new_thread(proc(f),())
t.start()
f.close()
c = raw_input(" Type anything to quit")


Execution output:
Linux abc.myhomenetwork.com 2.6.18-348.25.1.el5 #1 SMP Thu Apr 10 06:32:45 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
----------------------------
Traceback (most recent call last):
File "./readfile1.py", line 19, in <module>
t=thread.start_new_thread(proc(f),())
TypeError: first arg must be callable

[Sam Raker]

proc(f) isn't a callable, it's whatever it returns. IIRC, you need to do something like 'start_new_thread(proc, (f,))'

[MRAB]

The first argument should be the function. You're calling proc(f) and
the passing its result.

Also, start_new_thread starts the thread running immediately and
returns its identifier, an int.

[Mohan Mohta]

On Wednesday, June 3, 2015 at 4:01:13 PM UTC-5, Sam Raker wrote:
> proc(f) isn't a callable, it's whatever it returns. IIRC, you need to do something like 'start_new_thread(proc, (f,))'

If I execute something like
t=thread.start_new_thread(proc,(f))

I get:

Traceback (most recent call last):
File "./readfile1.py", line 19, in <module>

t=thread.start_new_thread(proc,(f))
TypeError: 2nd arg must be a tuple

[Joonas Liik]

You think "(f)" makes a tuple, but it does not.

the parentesis is not the tuple constructor, the comma is

try:

t=thread.start_new_thread(proc,(f,))

[Mohan Mohta]

Thanks for the pointer waffle.
The program executes now but still not the way I want it.
I think I will need to tweak it a bit as the code is executing with the same argument from the file /tmp/python/1 multiple times whereas it needs to be executed only ones but in parallel. Let me figure that out.


Once again thanks for all the help provided on this thread.

[sohca...@gmail.com]

Check your usages of "line" and "f". You have spots where you probably meant "line" instead of "f", and others where you have "f" where you probably meant "line".

[M2]

Here is my logic:
f is where the entire file is getting loaded
which is also passed as argument in the function proc
line has a single line from the file which is then stripped off the new line character and assigned to com2 variable which helps in using it in the subprocess.call

[Cameron Simpson]

On 03Jun2015 17:04, M2 <mohan...@gmail.com> wrote:
>On Wednesday, June 3, 2015 at 6:56:47 PM UTC-5, sohca...@gmail.com wrote:
>> On Wednesday, June 3, 2015 at 4:45:52 PM UTC-7, M2 wrote:
>> > On Wednesday, June 3, 2015 at 5:34:31 PM UTC-5, Waffle wrote:
>> > > You think "(f)" makes a tuple, but it does not.
>> > > the parentesis is not the tuple constructor, the comma is
>> > > try:
>> > > t=thread.start_new_thread(proc,(f,))
>> >
>> > Thanks for the pointer waffle.
>> > The program executes now but still not the way I want it.
>> > I think I will need to tweak it a bit as the code is executing with the same argument from the file /tmp/python/1 multiple times whereas it needs to be executed only ones but in parallel. Let me figure that out.
>> >
>> >
>> > Once again thanks for all the help provided on this thread.
>>
>> Check your usages of "line" and "f". You have spots where you probably meant "line" instead of "f", and others where you have "f" where you probably meant "line".
>
>Here is my logic:
>f is where the entire file is getting loaded

In the main code, yes.

>which is also passed as argument in the function proc

But why? f is not using in proc. Only line is.

>line has a single line from the file which is then stripped off the new line character and assigned to com2 variable which helps in using it in the subprocess.call

That end is fine.

I would be passing only "line" to proc, not "f" at all.

Suggestion: move your main code into its own function. That will make all the
variables in it "local". Your proc function is presently relying on "line"
being global, which generally bad and a recipe for disaster in multithreaded
code.

Moving the main code into its own function will (1) get rid of the global
variables and (2) force you to consider exactly what you need to pass to
"proc", and that will help reveal various logic issues.

Cheers,
Cameron Simpson <c...@zip.com.au>

>>>How do you blip the throttle and wave? Do you blip it real high, then wave
>>>before the revs drop back?
>>Blip = right hand; Wave = left hand. Do both simultaneously. QED.
>Doesnt this make the bike lurch forward thru the intersection?
Not if the disk lock is in place...
- Dean Woodward <de...@agora.rdrop.com>

[M2]

Thanks Cameron.
I do not see the duplication in the execution now.
I do see it is not consistent by executing all the threads ; it might be due to the fact I am using
subprocess.call(co,shell=True)
Per my understanding the above does not keep track of threads it just spawns a thread and leaves it there.
I might need to use the function start(), join() to ensure it picks up all the argument

For the record now my new code is

#! /usr/bin/python
import os
import subprocess
import thread
import threading

import sys
from thread import start_new_thread

def proc(col) :
subprocess.call(col,shell=True)

return

f = open('/tmp/python/1')
for line in f:

com1="ssh -B "
com2=line.strip('\n')
com3= " uname -a "
co=str("ssh -B ")+ str(com2) + str(" uname -a")

t=thread.start_new_thread(proc,(co,))
f.close()


Thanks again for the help

[Cameron Simpson]

On 03Jun2015 19:59, M2 <mohan...@gmail.com> wrote:
>On Wednesday, June 3, 2015 at 7:38:22 PM UTC-5, Cameron Simpson wrote:
>> I would be passing only "line" to proc, not "f" at all.
>>
>> Suggestion: move your main code into its own function. That will make all the
>> variables in it "local". Your proc function is presently relying on "line"
>> being global, which generally bad and a recipe for disaster in multithreaded
>> code.
>>
>> Moving the main code into its own function will (1) get rid of the global
>> variables and (2) force you to consider exactly what you need to pass to
>> "proc", and that will help reveal various logic issues.

[...]

>
>Thanks Cameron.
>I do not see the duplication in the execution now.
>I do see it is not consistent by executing all the threads ; it might be due to the fact I am using
> subprocess.call(co,shell=True)
>Per my understanding the above does not keep track of threads it just spawns a thread and leaves it there.

subprocess does not start a thread at all. It starts a new external process.
Your variable "t" is the Thread.

>I might need to use the function start(), join() to ensure it picks up all the argument

Because you have called .start_new_thread, you do not need to call .start().

Whether you call .join() on the Thread later is up to you. When you call
.join(), the caller will block until the Thread terminates. So you won't want
to do that immediately if you hope to run several subprocesses in parallel with
this setup.

Thank you for including your current code; most helpful. I have some other
remarks, below the code which I'm keeping here for reference:

>For the record now my new code is
>#! /usr/bin/python
>import os
>import subprocess
>import thread
>import threading
>import sys
>from thread import start_new_thread
>
>def proc(col) :
> subprocess.call(col,shell=True)
> return
>
>f = open('/tmp/python/1')
>for line in f:
> com1="ssh -B "
> com2=line.strip('\n')
> com3= " uname -a "
> co=str("ssh -B ")+ str(com2) + str(" uname -a")
> t=thread.start_new_thread(proc,(co,))
>f.close()

First up, your open/read/close stuff is normally written like this in Python:

with open('/tmp/python/1') as f:
for line in f:
... loop body here ...

This is because the return from open() is in fact a "context manager", an
object designed to work with the "with" statement. Specificly, the "exit" step
of this particular context manager calls .close() for you. Aside from being
more succinct, a context manager has the other advantage that the exit action
is _always_ called when control leaves the scope of the "with". Consider this
example function:

def foo(filename):
with open(filename) as f:
for line in f:
... do stuff with line ...
if "quit" in line:
return

That will return from the function if the string "quit" occurs on one of the
lines in the file, and not process any following lines. The important point
here is that if you use "with" then the open file will be closed automatically
when the return happens. With your original open/loop/close code, the "return"
would bypass the .close() call, leaving the file open.

The next remark I would make is that while Threads are very nice (I use them a
lot), if all your thread is doing is dispatching a subprocess then you do not
need a thread. The separate process that is made by subprocess runs
automatically, immediately, without waiting for your program. So you can invoke
a bunch of subprocesses in parallel, and not go near a thread.

However, because you have invoked the subprocess with .call(), your "proc"
function inherently waits for the subprocess to complete before returning and
therefore you need a thread to do these in parallel.

The .call() function is a convenience function; the alternative is to use the
.Popen() constructor directly:

def proc(col):
P = subprocess.Popen(col, shell=True)
return P

Then you can have your main loop replace "t=..." with:

P = proc(co)

At this point, proc starts the subprocess but does not wait for it (versus
"call", which does the wait for you). So you can dispatch all these
subprocesses in parallel. Then after your loop you can wait for them to finish
at your leisure.

Cheers,
Cameron Simpson <c...@zip.com.au>

Nonsense. Space is blue, and birds fly through it. - Heisenberg

[Gary Herron]

On 06/03/2015 01:41 PM, Mohan Mohta wrote:
> Hello
> I am trying to create multiple thread through the below program but I am getting an error
>
> #! /usr/bin/python
> import os
> import subprocess
> import thread
> import threading
> from thread import start_new_thread
>
> def proc(f) :
> com1="ssh -B "
> com2=line.strip('\n')
> com3= " uname -a"
> co=str("ssh -B ")+ str(com2) + str(" uname -a")
> subprocess.call(co,shell=True)
> print "----------------------------"
> return
>
> f = open('/tmp/python/1')
> for line in f:
> t=thread.start_new_thread(proc(f),())

You are calling the function f yourself, but what you want here is for
the thread to call f once it's running.
So do

t=thread.start_new_thread(proc, (f,)) # Procedure to call and a
tuple of args for it.

> t.start()

I don't think the thread has a start method.

> f.close()
> c = raw_input(" Type anything to quit")
>
>
> Execution output:
> Linux abc.myhomenetwork.com 2.6.18-348.25.1.el5 #1 SMP Thu Apr 10 06:32:45 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
> ----------------------------
> Traceback (most recent call last):
> File "./readfile1.py", line 19, in <module>
> t=thread.start_new_thread(proc(f),())
> TypeError: first arg must be callable

You should probably also consider using the higher-level threading
module rather than the lower level thread module.

(Also consider using Python3 instead of Python2.)

Gary Herron





--
Dr. Gary Herron
Department of Computer Science
DigiPen Institute of Technology
(425) 895-4418

[M2]

Awesome Cameron.
It works the way I want it to work.

Thanks a lot guys.
Here is the new code:

#! /usr/bin/python
import os
import subprocess
import thread
import threading
from thread import start_new_thread

def proc(col) :
P=subprocess.Popen(col, shell=True)

return

f = open('/tmp/python/1')
for line in f:

com1="ssh -B "
com2=line.strip('\n')

#com3= " /var/scripts/health/health.sh"
com3= " uname -a "
co=str("ssh -B ")+ str(com2) + str(com3)
P=proc(co)
f.close()


Thanks a ton guys this should get me started on the greater program I am trying to write.

[Cameron Simpson]

On 04Jun2015 10:20, M2 <mohan...@gmail.com> wrote:
>Awesome Cameron.
>It works the way I want it to work.

Glad to hear it. A few small remarks:

>Thanks a lot guys.
>Here is the new code:

[...]
>from thread import start_new_thread

You're not using this any more. You may want to tidy this up.

>def proc(col) :
> P=subprocess.Popen(col, shell=True)
> return

You are not returning P. But ...

> co=str("ssh -B ")+ str(com2) + str(com3)
> P=proc(co)

Here you are saving the return value as P (local to this function). My
suggestion, I know. The point here is that in later code you might save all the
P values and call P.wait() for them at some point after the loop, waiting for
them to terminate. Or you may not care. Your call.

>f.close()

Consider adding the suggested "with" form. Shorter, easier to read, more
reliable.

Cheers,
Cameron Simpson <c...@zip.com.au>

Ride fast
Die fast
Leave no usable organs
- Tom Warner <t...@dfind.demon.co.uk>