multiprocess passing arguments double asterisks

[pic...@gmail.com]

import <span class="highlight" style="padding-left: 0px; padding-right: 0px;">multiprocess</span>ing as mp

def bar(**kwargs):
for a in kwargs:
print a,kwargs[a]

arguments={'name':'Joe','age':20}
p=mp.Pool(processes=4)
p.map(bar,**arguments)
p.close()
p.join()



Errors:

Traceback (most recent call last):
File "post.py", line 9, in <module>
p.map(bar,**arguments)
TypeError: map() got an unexpected keyword argument 'age'

How do I pass the dictionary arguments?

[pic...@gmail.com]

[Thomas Nyberg]

On 10/23/2016 03:12 AM, pic...@gmail.com wrote:
> import <span class="highlight" style="padding-left: 0px; padding-right: 0px;">multiprocess</span>ing as mp
>
> def bar(**kwargs):
> for a in kwargs:
> print a,kwargs[a]
>
> arguments={'name':'Joe','age':20}
> p=mp.Pool(processes=4)
> p.map(bar,**arguments)
> p.close()
> p.join()

What are you trying to do? The map method is similar to the map built-in:

https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map
https://docs.python.org/2/library/functions.html#map

map(function, iterable, ...)
Apply function to every item of iterable and return a list of the results...

You can't apply it to keyword arguments like this. There are some
different SO threads talking about this sort of thing:

http://stackoverflow.com/questions/13499824/using-python-map-function-with-keyword-arguments
http://stackoverflow.com/questions/10212445/python-map-list-item-to-function-with-arguments
http://stackoverflow.com/questions/16874244/python-map-and-arguments-unpacking

Maybe those (especially the last one) are helpful.

Cheers,
Thomas

[pic...@gmail.com]

Thanks for the reply.

The code snippet given by Peter is not very clear

I would like to multiprocess a function which is written in python of the form bar(**kwargs) which returns a value. This example does not return anything

Would you please use my example for the map function?

I appreciate your help,

[Thomas Nyberg]

On 10/24/2016 12:45 PM, pic...@gmail.com wrote:
> Thanks for the reply.
>
> The code snippet given by Peter is not very clear
>
> I would like to multiprocess a function which is written in python of the form bar(**kwargs) which returns a value. This example does not return anything
>
> Would you please use my example for the map function?
>
> I appreciate your help,
>

I'm honestly not totally sure what you want to do. However, say you want
to do the following (btw this is basically what Dennis said i nhis last
email, but maybe I can help clarify):

kwargs = {'param1': val1, 'param2': val2})

Then you'd like to have the following two operations performed in
separate processes:

bar(param1=val1)
bar(param2=val2)

In that case, I guess I would do something like the following. First
define bar_wrapper as follows (*I haven't tested any code here!):

def bar_wrapper(pair):
key, val = pair
return bar(**{key: val})

Then I would probably do something like

map(bar_wrapper, kwargs.items())

I.e. basically what I'm doing is taking the key-val pairs and producing
a list of them (as tuples). This is something that you can apply map
too, but not with the original function. So then the wrapper function
converts the tuple back to what you want originally.

Hopefully I'm understanding correctly and hopefully this helps.

Cheers,
Thomas

[pic...@gmail.com]

Thomas,

I have strings & numpy.ndarray as arguments. The wrapper function works great for strings.

Here's what I'm trying to do...
**************************

I have a serial python fn of the form

def bar(**kwargs):
a=kwargs.get("name")
print a
self.a1d=ma.asanyarray(kwargs.get('a1d'), dtype=float)
(****more calculations--takes a long time to compute
returns an object)

I am trying to run this function in parallel.

Here's my Main program

import numpy as np
import numpy.ma as ma
from delegate import parallelize
from hashlib import sha1
a1d=np.zeros((32))
b1d=np.zeros((32))
p=open("/tmp/pdata","rb")
pdata=np.load(p)
for i in range(0,10):
a1d=pdata['t1d']
b1d=pdata['gz1d']
print a1d,b1d
kwargs={'name':'special','a':a1d,'b':b1d}
val=parallelize(bar,kwargs)
***************************************
Error:
line 30, in <module>
val=parallelize(bar,kwargs)
delegate.py", line 311, in process
item, items = items[0], items[1:]
KeyError: 0
**************************************
Error: (with the wrapper function)
val=parallelize(bar_wrapper,kwargs.items())
TypeError: unhashable type: 'numpy.ndarray'
***************************************

How do I pass the string & numpy.ndarray to the function bar(**kwargs)?

Thank you for suggestions & help,






a1d=np.zeros((32))
b1d=np.zeros((32))
p=open("/tmp/pdata","rb")
pdata=np.load(p)
for i in range(0,100):
a1d=pdata['t1d']
b1d=pdata['gz1d']
print a1d,b1d
kwargs={'name':'special','a':a1d,'b':b1d}
val=parallelize(bar,kwargs)

[MRAB]

'parallelize' expects a function and a list of items. It calls the
function with each item in the list in parallel, passing the item as the
argument of the function.

That _single_ argument item can be a tuple or a dict.

[ric...@gmail.com]

--if the dict has numpy arrays it fails..

In the above example if
kwargs={'name':'special','a':2}
val=parallelize(bar_wrapper,kwargs.items()) it works fine

But if

kwargs={'name':'special','a':a1d,'b':b1d}

it fails (a1d & b1d are numpy arrays)