Idiom for map! over Arrays, DArrays and SharedArrays

[Douglas Bates]

A core calculation in some methods of fitting generalized linear models (GLMs) is updating a vector (the mean vector) by applying a scalar function (called the "inverse link") to the elements of another vector (the linear predictor).  Other, similar, updates are also involved in the Iteratively Reweighted Least Squares (IRLS) algorithm.

I would like to be able to apply this operation in parallel if the vectors in question are in either DArray or SharedArray storage.  Is there a general idiom for doing this or am I better off using method dispatch to handle the different cases?  I can assume that the chunks of the vectors for different processes are aligned.

[Amit Murthy]

I can add map! for DArray and SharedArrays. Would that suffice. If the mapping function is not an anonymous one, it would have to be pre-loaded on all the workers though.

[Simon Kornblith]

Unfortunately the performance of map! is still not great. Without method lookup hoisting/specialization on function arguments, the overhead versus a loop is quite substantial.

Until we have better performance for functions passed as arguments, it would be great to have DArray/SharedArray support in NumericExtensions, since that would provide a general means to do efficient reductions/mapping in parallel.

Simon