Getting my tracer working on Python 3.11

[Ionel Cristian Mărieș]

Hello everyone, I've written a tracer using cython and i'm trying to fix it up to run on Python 3.11, these are the relevant changes I've done so far to deal with the now-opaque frame objects:

https://github.com/ionelmc/python-hunter/compare/v3.5.1...master#diff-52ce3e22d2ddd16175600806e0fcacdedd30f7cc4564c7cbcf017290bbf1df4a

_compat.h is just a copy of https://github.com/python/pythoncapi-compat/blob/main/pythoncapi_compat.h

So this seems to almost run - I hope I got that stuff right, if not please comment.

What I'm stuck is a thing I use in the test suite to build on Python 3.11 - a Cython class that I use to patch some things around (for testing): https://github.com/ionelmc/python-hunter/blob/master/tests/eviltracer.pyx

On Cython 0.29 I get:

tests/eviltracer.c: In function ‘__Pyx_modinit_type_import_code’:
tests/eviltracer.c:3660:101: error: invalid application of ‘sizeof’ to incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
 3660 |   __pyx_ptype_6hunter_7_tracer_FrameType = __Pyx_ImportType(__pyx_t_1, "types", "FrameType", sizeof(PyFrameObject), __Pyx_ImportType_CheckSize_Warn);
      |                                                                                                     ^~~~~~~~~~~~~

plus some `size changed, may indicate binary incompatibility` errors, or warnings?!

I've tried Cython 3.0a11 I get:L

tests/eviltracer.c: In function ‘__Pyx_modinit_type_import_code’:
tests/eviltracer.c:5249:101: error: invalid application of ‘sizeof’ to incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
 5249 |   __pyx_ptype_6hunter_7_tracer_FrameType = __Pyx_ImportType(__pyx_t_1, "types", "FrameType", sizeof(PyFrameObject), __PYX_GET_STRUCT_ALIGNMENT(PyFrameObject),
      |                                                                                                     ^~~~~~~~~~~~~
tests/eviltracer.c:5249:144: error: invalid application of ‘__alignof__’ to incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
 5249 |   __pyx_ptype_6hunter_7_tracer_FrameType = __Pyx_ImportType(__pyx_t_1, "types", "FrameType", sizeof(PyFrameObject), __PYX_GET_STRUCT_ALIGNMENT(PyFrameObject),
      |                                                                                                                                                ^~~~~~~~~~~~~
tests/eviltracer.c:1967:47: note: in definition of macro ‘__PYX_GET_STRUCT_ALIGNMENT’
 1967 | #define __PYX_GET_STRUCT_ALIGNMENT(s) alignof(s)
      |                                               ^
 

error: invalid application of ‘sizeof’ to incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}

Those __pyx_ptype_variables don't seem to be used so I'm wondering how do solve this - I don't really want to patch those C files, seems wrong.

[da-woods]

Your basic basic problem is that you can't declare an extern cdef class for an opaque C type so

ctypedef class types.FrameType[object PyFrameObject]:
        cdef PyObject* f_trace

can't work.

There's two possible options

1. don't type it in Cython. Instead just use it as a regular python object and use that to access the `f_trace` attribute.

2. Explicitly include the internal header where PyFrameObject is defined (Python/internal/pycore_frame.h) using `cdef extern from`.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/6feec801-29f3-405c-9a8f-97829cabf18bn%40googlegroups.com.

[Ionel Cristian Mărieș]

There's two possible options

1. don't type it in Cython. Instead just use it as a regular python object and use that to access the `f_trace` attribute.

I could replace FrameType with object all over the place but then I would like to check that I'm not getting weird objects that are not PyFrameObject in that field, is there an easy way to do it?

I've tried something like "from types cimport FrameType" and later when I init the Event do "self.frame = <FrameType?> frame" but both fail at cytonization.

I can't typecheck for PyFrameObject right? (since that is a C struct from cython's perpective)

 

2. Explicitly include the internal header where PyFrameObject is defined (Python/internal/pycore_frame.h) using `cdef extern from`.

If i replace * with cdef extern from "internal/pycore_frame.h" then my build problem goes away but then I get this error at import time:

tests/test_tracer.py:88: in <module>
    from eviltracer import EvilTracer
tests/eviltracer.pyx:1: in init eviltracer
    # cython: language_level=3str, c_string_encoding=ascii
E   RuntimeWarning: types.FrameType size changed, may indicate binary incompatibility. Expected 56 from C header, got 120 from PyObject

But I want future-proof solution, it seems to me that I'll have to scratch my head all over again when Python 3.12 comes out if I start including internal headers...

 

[da-woods]

You can disable the size check by adding `check_size="ignore"` as an argument to your `FrameType` definition. In this case it's OK - Python looks to overallocate the frame beyond the basic structure so it's kind of expected. Of course this isn't hugely future-proof.

You can't do `from types cimport ...` since it's just a Python module. If you want a statically enforced Cython type-check without needing any implementation details I think you can do

ctypedef extern class types.FrameType[check_size="ignore"]:
    pass

(check_size="ignore" is useful here, because it'll assume the size should match that of an empty PyObject). Note that it's important that you leave the body as "pass", because otherwise you really have mislead it about something important. I haven't tried this fully though so it's possible I'm missing something that stops it from working.

Alternatively you can do the checks yourself with `isinstance` (nothing Cython-specific here, just as you would in Python)

I think you're going to struggle to get a completely future-proof solution because the Python developers consider frames an implementation detail and are inclined to change it. And there isn't too much Cython can do to shield you from that.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/a053be36-9247-4fdb-86d8-dcc098e611c9n%40googlegroups.com.