Why is the Cython version of serializer 1,000x times slower than the F# one?

[mra...@gmail.com]

Cython compiled code: https://github.com/mrakgr/Poker-In-Spiral/blob/7d3f884f3b39f76981b387c61e6c65f39e683f80/01-Serialization/array_bench.pyx

F# compiled code: https://github.com/mrakgr/Poker-In-Spiral/blob/7d3f884f3b39f76981b387c61e6c65f39e683f80/02-Serialization%20in%20Fsharp/array_bench.fsx

Python pickler combinator tutorial: https://github.com/mrakgr/Poker-In-Spiral/blob/7d3f884f3b39f76981b387c61e6c65f39e683f80/01-Serialization/serialization/array.ipynb

In fact the Cython version is at most 2x faster than raw Python. When I compile the loops as tail recursive functions it is even slower. I thought it might be worth trying compiling with 03 but that does change the performance. I've made very minor changes between the two versions (.NET arrays start off zero inited, it does not have views), and even looking at the compiled outputs the code is very similar.

I've made the Cython backend over half a year ago and have been using it since then. Despite that, this is the first time I've actually bothered benchmarking it on a non-trivial task. My plan was to write a series of articles on how to make a poker bot and highlight Spiral's performance benefits along the way, but I can hardly do that given the current situation. If this cannot be improved upon I'll scrap the Cython backend and make a native C++ one with something special for Python interop.

[mra...@gmail.com]

Cython generated file html link: https://github.com/mrakgr/Poker-In-Spiral/blob/449bca766075c8d8058a3509e7552861cd94c5c7/01-Serialization/array_bench.html

There isn't much explanation other that the library interop code is causing performance issues. For example, `numpy.zeros` call does an enormous amount of ref counting activity. I thought that calls like this would be optimized since they are statically known, but I guess not. There is nothing I can do personally about this.

[Robert Bradshaw]

Creating objects like Mut0 is also more expensive than, say,
manipulating structs.

Given that you know the size of several of your arrays, and they're
not multi-dimensional, you could consider using C arrays/pointers
instead.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/6381f013-37fd-47c6-aab8-1a595cd7b03bn%40googlegroups.com.

[Marko Grdinic]

I'd have liked to use structs, but Cython ones do not allow Python objects in them. C arrays would be beyond the reach of the runtime system and would require manual memory management. There is no point in it. The way I envisioned the series was to make a Python tutorial, and then demo the Spiral code which has a similar high level of expressiveness, but greatly improved performance. Since the performance is not there  without having to write C style code, there is no point in doing that.

Actually, I am surprised that all the inlining of the primitives did not improve the performance much. Even with the heap allocation that is already being done I'd have expected it to be far faster than Python. There is nothing I can do about the Cython generated code, but I might want to rewrite it in C++ while keeping the expected allocation profile just to see how much the unexpected factors impact performance before starting work on that backend.

You received this message because you are subscribed to a topic in the Google Groups "cython-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cython-users/Jd_Eqw2hWD0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/CADiQ%2BQANmYtc7kaYMjvHr3wz5mf57yFOgqi8pkqsziq-9sABsg%40mail.gmail.com.