Exception handling in cython cdef declarations

[Raghav RV]

Hi,

I have several cython cdef functions with various return types. (void / int / etc...)

Some of these functions say a, b, c call a function `safe_realloc` which will raise a python `MemoryError` based on a condition.

Now there are several other functions say d, e, f which may or may not call a, b and c.

I imperatively need the program to crash if there is a MemoryError raised when safe_realloc is called at any point at any level of nested call. Currently in our codebase this is not the case and the error goes "Ignored"...

The cython docs state that if except * is used at the declaration of cdef functions that do not return python object,

"This form causes Cython to generate a call to PyErr_Occurred() after every call to <fn>".

How costly is this `PyErr_Occured` will it induce significant slow ups if "except *" is used blindly for all cdef declarations (a, b, c, d, e, f, safe_realloc) ?

Thanks!

[Chris Barker]

How costly is this `PyErr_Occured` will it induce significant slow ups if "except *" is used blindly for all cdef declarations (a, b, c, d, e, f, safe_realloc) ?

In general, Python's Exception handling is pretty fast, but you'd have to profile to be sure.

But if you want to use python exceptions deep down in your code, you really do need to handle them.

If the overhead proves to be costly, then at the lower level you may want to revert to C-style error handling -- i.e. have your functions return an error code the you check when you call them.

in this case, your safe_realloc would need to return a value indicating that it was not successful, and you'll need to check that flag when you call it, perhaps raising a python exception higher up in the stack.

-CHB

 

Thanks!

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Robert Bradshaw]

I have seriously considered whether "except *" (or except? NONCE for certain types) should become the default, as this is a common trap and source of surprise. It would be interesting to see some benchmarks. We also might want syntax for disabling the exception checking. 

[Raghav RV]

Thanks Chris and Robert for the replies!

[Venkat Raghav Rajagopalan (Raghav RV)]

I have seriously considered whether "except *" (or except? NONCE for certain types) should become the default, as this is a common trap and source of surprise. It would be interesting to see some benchmarks. We also might want syntax for disabling the exception checking. 

 

Hey Robert,

For our particular usage (in scikit-learn - ref: https://github.com/scikit-learn/scikit-learn/pull/8002#issuecomment-269369131), we experience significant slowdowns. This I surmise is the result of the function being called multiple times. However except -1 with int return types is pretty fast and comparable to speeds with no except declared.

Thanks!

[Venkat Raghav Rajagopalan (Raghav RV)]

I have seriously considered whether "except *" (or except? NONCE for certain types) should become the default, as this is a common trap and source of surprise. It would be interesting to see some benchmarks. We also might want syntax for disabling the exception checking.