Casting function pointers with except markers

[Thomas Kluyver]

Is it possible to cast a function pointer with an 'except -1' clause? I'm trying to do something like this:

info.flush = <herr_t (*)(H5FD_t *, hid_t, hbool_t) except -1>H5FD_fileobj_flush

But the -1 doesn't seem to be allowed syntax inside the <>:

info.flush = <herr_t (*)(H5FD_t *, hid_t, hbool_t) except -1 >H5FD_fileobj_flush

h5py/h5fd.pyx:209:80: Expected '>', found 'NEWLINE'

Using noexcept or except * seems to be valid syntax, but neither is exactly what I'm trying to express, although it might allow for a workaround.

Thanks,

Thomas

[D Woods]

It's probably a bug if we don't accept it (although may not be easy to fix).

Practically the solution is to create a `ctypedef` for the function pointer and cast to that. Something like (untested code):

ctypedef herr_t (*fileflush_ptr)(H5FD_t *, hid_t, hbool_t) except -1

info.flush = <fileflush_ptr>H5FD_fileobj_flush

[David Hoese]

I'm looking at implementing your suggestion in a branch that Thomas (the OP) started. It looks like Cython doesn't like any syntax with "except X" in it when it comes to ctypedef or casting. For example, we have:

cdef herr_t H5FD_fileobj_flush(H5FD_fileobj_t *f, hid_t dxpl, hbool_t closing) except -1 with gil:
    # TODO: avoid unneeded fileobj.flush() when closing for e.g. TemporaryFile
    (<object>f.fileobj).flush()
    return 0

Then from your suggestion:

ctypedef herr_t (*fileflush_ptr)(H5FD_t *, hid_t, hbool_t) except -1

and:

info.flush = <fileflush_ptr>H5FD_fileobj_flush

But then cythonizing gives:

info.flush = <fileflush_ptr>H5FD_fileobj_flush
             ^
------------------------------------------------------------

h5py/h5fd.pyx:213:13: Cannot assign type 'fileflush_ptr' to 'herr_t (*)(H5FD_t *, hid_t, hbool_t) noexcept'. Exception values are incompatible. Suggest adding 'noexcept' to type 'herr_t (H5FD_t *, hid_t, hbool_t) except -1'.

I read this as Cython not recognizing "except -1" as an except clause since it is suggesting adding "noexcept" to it.

Am I missing something?

Dave

[da-woods]

My reading of what's going on:

The type `fileflush_ptr` defines a function pointer with `except -1` as you have written. However `info.flush` is a different type with `noexcept`.

`fileflush_ptr` looks to be the same type as `H5FD_fileobj_flush` so the cast is unnecessary. However, because of the difference in exception specification it doesn't match `info.flush`.

I suspect `info.flush` is defined by some external interface which will not be expecting a Python exception? In which case `noexcept` is right. You therefore want to change `H5FD_fileobj_flush` to be `noexcept` to match this interface. If you do that you won't need a cast.

Hopefully that helps.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/059385b5-1ade-438f-9cec-2a2d738990d5n%40googlegroups.com.

[David Hoese]

Yep you're right. There was an `IF` block (yes, I know they're deprecated) that was checking the version of the wrapped C library and only one version block was decorated with `except X`s. I tried not casting but it turns out there are type differences:



info.flush = H5FD_fileobj_flush
^
------------------------------------------------------------

h5py/h5fd.pyx:212:13: Cannot assign type 'herr_t (H5FD_fileobj_t *, hid_t, hbool_t) except -1 nogil' to 'herr_t (*)(H5FD_t *, hid_t, hbool_t) except -1'



This is for the h5py library and I'm not familiar enough with the low-level parts of it or HDF5 (the C library it wraps) to know if this can be cleaned up. I'll see if I can track it down otherwise I'll go the ctypedef route as it is indeed a syntax error to include "except X" in a cast.

Dave

>> To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/059385b5-1ade-438f-9cec-2a2d738990d5n%40googlegroups.com <https://groups.google.com/d/msgid/cython-users/059385b5-1ade-438f-9cec-2a2d738990d5n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the Google Groups "cython-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/cython-users/FW6yboCJeow/unsubscribe <https://groups.google.com/d/topic/cython-users/FW6yboCJeow/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to cython-users...@googlegroups.com <mailto:cython-users...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/9fc073c7-305b-420f-9f1f-7269da6f1c41%40d-woods.co.uk <https://groups.google.com/d/msgid/cython-users/9fc073c7-305b-420f-9f1f-7269da6f1c41%40d-woods.co.uk?utm_medium=email&utm_source=footer>.

[da-woods]

FWIW I think it'd be fine to add noexcept to the definition of H5FD_fileobj_flush. It was almost certainly the old Cython 0.29.x behaviour anyway, so it's just keeping things as is. If there is an exception then all it'll do is swallow it and print a message.

I really recommend this instead of doing any casts. Casts are bad because the function will think it's raised a Python exception but the caller will ignore it. Just adding noexcept to the function definition does the right thing.

If you want to minimize changes then the directive `legacy_implicit_noexcept` will let you select the old behaviour globally.

[Thomas Kluyver]

On Mon, 27 Nov 2023 at 23:19, da-woods <dw-...@d-woods.co.uk> wrote:

the function will think it's raised a Python exception but the caller will ignore it. Just adding noexcept to the function definition does the right thing.

In this case, I do actually want the functions to raise an exception, even though the caller won't understand that. We're using 'except -1' to return -1 on errors, which indicates to the C code (HDF5) that the operation failed. HDF5 propagates the error condition back out to our Cython wrapper 'outside' the C code, which sees that something has gone wrong, notices that there's already a Python exception set, and lets it propagate.

By doing this, we get a Python traceback that correctly describes the Python stack both in the outer code, and inside the callbacks to Python, jumping over the layers of HDF5 C code inbetween. This has seemed to work pretty well for us so far. I accept it may be somewhat delicate, but the whole machinery of letting HDF5 call back into Python code to access file-like objects is pretty fragile in any case. I'm please David has managed to make what we were already doing work with Cython 3. :-)

Thanks,

Thomas

[Stefan Behnel]

Am 5. Dezember 2023 10:00:29 UTC schrieb Thomas Kluyver:
>In this case, I do actually want the functions to raise an exception, even
>though the caller won't understand that. We're using 'except -1' to return
>-1 on errors, which indicates to the C code (HDF5) that the operation
>failed. HDF5 propagates the error condition back out to our Cython wrapper
>'outside' the C code, which sees that something has gone wrong, notices
>that there's already a Python exception set, and lets it propagate.

That's a reasonable thing to do. I do the same thing in lxml in a couple of places. You should expect that to work.

Stefan