Profiling

[Peter Schay]

Hello!

I have a few quick questions about profiling.

I tried profiling my project recently with 3.0.12 and was not seeing any cython symbols in the output of cProfile and pstats.  I upgraded to Cython 3.1 and voila!  All the Cython functions are in the output now.  Is this expected?   This testing is using Python 3.13.3 on macOS.  

Next question- I have functions that get called millions of times and the profiling is clearly affecting the time, and it looks like a messy range of tiny functions with I high overhead up to larger ones with more accurate accounting.

Are there other profiling tools you can recommend I should learn and use to see a clearer accounting without overhead?   

I saw the discussions  on GitHub about the new Python monitoring having issues in 3.12; will there be new profiling possibilities with Cython + 3.14 that I can try?

Thank you very much.

Regards,

Peter 

[da-woods]

Hi Peter,

Some quick version history to start with:

Python 3.12 switched to the new sys.monitoring profiling, and cProfile immediately started using that. At that point they didn't provide a C interface so Python 3.12 doesn't work with Cython at all. Note that some tools continue to use the old-style profiling though.

Python 3.13 added C support for the new sys.monitoring profiling, which we added support for in Cython 3.1. So, yes - it's expected that cProfile in Py 3.13 would only start working in Cython 3.1

I don't believe Python 3.14 is adding anything notable here, so don't expect any changes.

------

I don't specifically have any recommendation for other Python profiling tools but someone else might.

The other thing you can try is to use a C profiler instead. You might find it more productive because it's a bit lower overhead, especially for many small cdef functions. For this you mostly need to compile with debug symbols. You do have to pick through Cython's name mangling though, and the Python call-stack too. I don't know what the options are on macOS, but I usually use perf on Linux, or the one built into Visual Studio on Windows. Personally I usually skip straight to a C profiler for Cython code, but then I'm more interested in "how have we slowed down Cython" so it may not be quite the level you're after.

David

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cython-users/CALKyFja_vWZ9Kw1teuoLxQEG91rD3UbXe5%2B%3DuOKkY7jWsG0qwQ%40mail.gmail.com.

[Peter Schay]

Thanks for the helpful info.  It's great to have cProfile working again and also amazing to see how much progress is being made on Cython and Python these days. 

I will try Linux perf too.

To view this discussion visit https://groups.google.com/d/msgid/cython-users/a98b8b29-a2b7-4261-9c63-db50c1a14cd4%40d-woods.co.uk.