Re: [cython-users] Map vs for iteration in Cython

[Stefan Behnel]

Brian Knapp, 01.10.2012 05:04:
> I like map, reduce, and filter for readability, but these can only take
> Python objects as the function. I was unable to coerce a cdef function to
> satisfy the compiler, so I had to use either a cpdef or regular Python
> function when using map.

Using cpdef instead of plain def here adds overhead. The call is a Python
function call anyway.


> Using the equivalent list comprehension doesn't
> have this limitation and is significantly faster when using C calls.

Yes, that's expected. map() is the normal Python builtin (which is also
implemented in C, but needs to treat the callable as an arbitrary object).

Also note that map() behaves differently in Python 2 and Python 3, which
makes it harder to replace by an optimised implementation.


> Is there a way to get most of the performance of a for loop implementation
> such as:
> cdef int length = len(list)
> cdef int i = 0
> for i in xrange(length):
> call(list[i]

That's not a use case for map(), though. You are not collecting the result
in a list or otherwise using it in any way here. Also, I'd simplify this to

for item in list:
call(item)

(and then give "list" a better name).

Personally, I find list comprehensions more readable than map() most of the
time. It's a different thing with reduce().

You could write your own C version of map() and reduce() that simply takes
a C function pointer as first argument instead of a Python object. They
aren't more than a couple of lines each.

Stefan

[Robert Bradshaw]

On Mon, Oct 1, 2012 at 7:16 PM, Brian Knapp <knap...@gmail.com> wrote:
> I'd like to implement a versions then. I'm interested in maximizing
> performance headroom for a traversal problem that justifies the functional
> style.
>
> It seems like all I need is:
> cmap (fun, iterable)
>
> At first glance I'm thinking using a function pointer and a typed memory
> view for iterable is a good solution. However it's unclear that cymap would
> not need to know fun's signature and the type of elements in the view. Is
> there a potential compiler problem (many execution paths) for a general
> implementation of the proposed cmap?

Yes, you would need to explicitly specify the type (which could be a
fused type). However, it's unclear you need to write such a generic
framework. Also raw pointers are a pain in C as they cannot enclose
any state (like a closure), which is usually solved in C by also
passing a void* data member. I would verify that your bottleneck lies
here before going this route.