Passing functions as parameter (multiprocessing)

[Jean-Michel Pichavant]

Fellows,

I'm having problems understanding an issue with passing function as parameters.

I'm sending some functions to the multiprocessing module (python 2.5 with the proper backport).
I'm iterating on a list of functions, however it seems that only the last function implementation is used for
all the subprocesses.

Here's a code that triggers the issue:

import multiprocessing

def f1():
print 'I am f1'
def f2(foo):
print 'I am f2 %s' % foo

workers = [
(f1,tuple()),
(f2,(5,)),
]

procs=[]
for func, parameters in workers:
# here it should be decorated, but for this example to be kept simple, the function is only wrapped, doing nothing special
def subproc(*args, **kwargs):
return func(*args, **kwargs)
procs.append(multiprocessing.Process(target=subproc, args=parameters))

for proc in procs:
proc.start()
for proc in procs:
proc.join()

Here's the result:
> run test.py
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.2.1-py2.5-linux-i686.egg/multiprocessing/process.py", line 237, in _bootstrap
self.run()
File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.2.1-py2.5-linux-i686.egg/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "test.py", line 17, in subproc
return func(*args, **kwargs)
TypeError: f2() takes exactly 1 argument (0 given)
I am f2 5

It looks like the first subprocess is called with f2 instead of f1.

Any idea ?

JM

-- IMPORTANT NOTICE:

The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[MRAB]

I believe the problem is that 'subproc' calls 'func', which is rebound
on the each iteration.

[Peter Otten]

Jean-Michel Pichavant wrote:

> I'm having problems understanding an issue with passing function as
> parameters.

> Here's a code that triggers the issue:
>
>
> import multiprocessing
>
> def f1():
> print 'I am f1'
> def f2(foo):
> print 'I am f2 %s' % foo
>
> workers = [
> (f1,tuple()),
> (f2,(5,)),
> ]
>
> procs=[]
> for func, parameters in workers:

> def subproc(*args, **kwargs):
> return func(*args, **kwargs)
> procs.append(multiprocessing.Process(target=subproc, args=parameters))

Python has late binding, and when the loop has finished the name func is
bound to f2. You have created multiple subproc functions, but that doesn't
matter as they all invoke func aka f2.

A possible fix:

def make_subproc(func):

def subproc(*args, **kwargs):
return func(*args, **kwargs)

return subproc

procs=[]
for func, parameters in workers:

procs.append(multiprocessing.Process(target=make_subproc(func),
args=parameters))

[Oscar Benjamin]

On 13 November 2012 12:51, Peter Otten <__pet...@web.de> wrote:
> Jean-Michel Pichavant wrote:
>
>> I'm having problems understanding an issue with passing function as
>> parameters.
>
>> Here's a code that triggers the issue:
>>
>>
>> import multiprocessing
>>
>> def f1():
>> print 'I am f1'
>> def f2(foo):
>> print 'I am f2 %s' % foo
>>
>> workers = [
>> (f1,tuple()),
>> (f2,(5,)),
>> ]
>>
>> procs=[]
>> for func, parameters in workers:
>
>> def subproc(*args, **kwargs):
>> return func(*args, **kwargs)
>> procs.append(multiprocessing.Process(target=subproc, args=parameters))

I don't know if this is to do with the way that the code was
simplified before posting but this subproc function wrapper does
nothing (even after Peter fixed it below). This code is needlessly
complicated for what it does.

> Python has late binding, and when the loop has finished the name func is
> bound to f2. You have created multiple subproc functions, but that doesn't
> matter as they all invoke func aka f2.
>
> A possible fix:
>
> def make_subproc(func):
> def subproc(*args, **kwargs):
> return func(*args, **kwargs)
> return subproc
>
> procs=[]
> for func, parameters in workers:
> procs.append(multiprocessing.Process(target=make_subproc(func),
> args=parameters))

The above is one way to wrap a function but it wraps the function into
a function that is exactly the same as the original function. I assume
that the reason for having the subproc function was to be able to bind
parameters to the function like so:

def make_subproc(func, args):
def subproc():
return func(*args)

return subproc

procs=[]
for func, parameters in workers:

procs.append(multiprocessing.Process(target=make_subproc(func, parameters)))

However, as you have already noticed, multiprocess.Process already has
the machinery to do this. If you pass in a value for args you can

leave out the subproc/make_subproc function and write:

procs=[]
for func, parameters in workers:

procs.append(multiprocessing.Process(target=func, args=parameters))

A loop like this can just be a list comprehension:

procs = [multiprocessing.Process(target=func, args=args) for func,
args in workers]


Oscar

[Peter Otten]

Oscar Benjamin wrote:

> I don't know if this is to do with the way that the code was
> simplified before posting but this subproc function wrapper does
> nothing (even after Peter fixed it below). This code is needlessly
> complicated for what it does.

Jean-Michel's Post had the following comment preceding his subproc()
definition

>>> # here it should be decorated, but for this example to be kept
>>> simple, the function is only wrapped, doing nothing special

which I unhelpfully removed...

[Jean-Michel Pichavant]

> --
> http://mail.python.org/mailman/listinfo/python-list
>

Anyway thank you all, I solved the issue with Peter's code.
func was rebound after each iteration, though I though it wasn't.

I would not have figured it out by myself, my mind was lost in the mysteries of subprocesses, decorated and wrapped methods.

Cheers,