Union nested in struct (protobuf)

[Jeroen Kant]

Hello everyone,

I am trying to wrap a C++ library for use from Python that makes use of protobuf messages. Based on data gathered in Python, I would like to construct these messages in Cython and pass these message objects to the C++ library for use. The general message structure combines a struct with a union like below:

# messages.pb.h:

typedef struct _Messages_GeneralMessage {

    uint64_t timestamp;

    pb_size_t which_contents;

    union {

        Messages_SensorData sensor_data;

        Messages_DeviceEvent device_event;

    } contents;

} Messages_GeneralMessage;

I've attempted to define this message struct in my .pxd file like below. This was based on  a suggestion found on StackOverflow here combined with the documentation on interfacing with Extercal C Code.

# c_messages.pxd: 

cdef extern from "messages.pb.h":

    ctypedef struct Messages_SensorData:

        ...

    ctypedef struct Messages_DeviceEvent:

        ...

    ctypedef union Messages:

        Messages_SensorData sensor_data

        Messages_DeviceEvent device_event

    ctypedef struct Messages_GeneralMessage:

        uint64_t timestamp

        pb_type_t which_contents

        Messages contents

This however runs into trouble with the 'Messages' union I've defined when compiling the extension.

use_c_messages.cpp(2385): error C2065: 'Messages': undeclared identifier

It seems to be looking for the 'Messages' identifier in the .pb.h file, which is logically not defined there. Any thoughts on how to properly handle the Messages_GeneralMessage structure from the .pb.h file?

Cheers,

Jeroen

[D Woods]

It'd probably be useful to see a bit of the c++ code that's causing the error message. I wonder if you're triggering an auto-conversion to/from Python (or similar) which is causing "Messages" to appear? Normal access to Messages_GeneralMessage.contents really shouldn't require the name "Messages".

My other proposed workaround would be to declare the two fields in "Messages_GeneralMessage" with a custom cname:

ctypedef struct Messages_GeneralMessage:

        uint64_t timestamp

        pb_type_t which_contents

        Messages_SensorData sensor_data "contents.sensor_data"

        Messages_DeviceEvent device_event "contents.device_event"

(Code is untested - you may have to try putting the cname string in other places if I've misremembered). The idea is that Cython will generate "contents.sensor_data" instead of "sensor_data" and this should look it up correctly.

[Jeroen Kant]

Thank you for your response!  Your proposed workaround works perfectly, no need to change anything about the placement of the cname string. So for me personally, that does solve the problem. Thank you very much!

I can imagine the related c++ code could still be useful however to get some more insight in regards to the situation. I've included all of this below. If any extra information would be of help, please let me know!

The error is thrown on the third rule here:

static PyObject* __pyx_convert__to_py_Messages_SensorData(Messages_SensorData s);

static PyObject* __pyx_convert__to_py_Messages_DeviceEvent(Messages_DeviceEvent s);

static PyObject* __pyx_convert__to_py_Messages(Messages s);

static PyObject* __pyx_convert__to_py_Messages_GeneralMessage(Messages_GeneralMessage s);

This is the first reference to Messages. This throws the following errors:

use_c_messages.cpp(2385): error C2065: 'Messages': undeclared identifier

use_c_messages.cpp(2385): error C2146: syntax error: missing ')' before identifier 's'

We then get the same problem in the related section of the code:

  static PyObject* __pyx_convert__to_py_Messages(Messages s) {

    PyObject* res;

    PyObject* member;

    res = __Pyx_PyDict_NewPresized(2); if (unlikely(!res)) return NULL;

    member = __pyx_convert__to_py_Messages_SensorData(s.sensor_data); if (unlikely(!member)) goto bad;

    if (unlikely(PyDict_SetItem(res, __pyx_n_s_sensor_data, member) < 0)) goto bad;

    Py_DECREF(member);

    member = __pyx_convert__to_py_Messages_DeviceEvent(s.device_event); if (unlikely(!member)) goto bad;

    if (unlikely(PyDict_SetItem(res, __pyx_n_s_device_event, member) < 0)) goto bad;

    Py_DECREF(member);

    return res;

    bad:

    Py_XDECREF(member);

    Py_DECREF(res);

    return NULL;

  }

With the following errors:

use_c_messages.cpp(14728): error C2065: 'Messages': undeclared identifier

use_c_messages.cpp(14728): error C2146: syntax error: missing ')' before identifier 's'

use_c_messages.cpp(14728): error C2143: syntax error: missing ';' before '{'

use_c_messages.cpp(14728): error C2447: '{': missing function header (old-style formal list?)

use_c_messages.cpp(14775): error C3861: '__pyx_convert__to_py_Messages': identifier not found

Given your reply, I do think this implies a conversion to/from Python is triggered somehow?

Op dinsdag 12 oktober 2021 om 12:43:12 UTC+2 schreef D Woods:

[D Woods]

Yeah - that looks like the automatically generated conversion to a Python dict. That might happen if you return the struct from a `def` function, or maybe have made the struct a public attribute of a `cdef` class.

In this case it's unlikely to be a useful conversion since it doesn't know the details of the union type. It's probably worth working out what's using it (maybe by searching the cpp file for the call of __pyx_convert__to_py_Messages_GeneralMessage) since it's unlikely to be doing what you want. It's most likely just something you've forgotten to type, or something similar.

[Jeroen Kant]

Ah, that clarifies a lot! Thank you for pointing out the  __pyx_convert__to_py_Messages_GeneralMessage() search in the cpp file, that indeed helped to point to some code I was using to debug the current situation. Removing this code completely removed all uses of  __pyx_convert__to_py_Messages_GeneralMessage() from the cpp file, causing the original ctypedef union Messages method to work as well. Thanks a lot!

Op dinsdag 12 oktober 2021 om 22:10:44 UTC+2 schreef D Woods: