how to assign struct member values from a python dict

[Giovanni Torres]

I have a large C struct that I'd like to assign values using a python dictionary.  For example:

    typedef struct descriptor_t {
        char *name
        uint32_t user_id
        uint32_t group_id
        ....

(This struct has about 40 members)

Then, there is a function to initialize the descriptor:
   
    init_descriptor(descriptor_t *descriptor_msg)


I wrapped it in Cython and doing the following in the function:

cpdef int allocate(dict job_descriptor)
    """
    a = {"name": "foo", "user_id": 1000, "group_id: "1000"}
    cdef:
        descriptor_t descriptor_msg
    init_descriptor(&descriptor_msg)
    descriptor_msg.name = a["name"]
    descriptor_msg.group_id = a["group_id"]
    descriptor_msg.user_id = a["user_id"]
    c_allocate(&descriptor_msg)


This works, but I would prefer to do something like the following:

   for key in job_descriptor.keys():
       descriptor_msg[key] = a[key]
   c_allocate(&descriptor_msg)

But I get the error: Attempting to index non-array type 'descriptor_t'


Any suggestions on how to assign python dictionary key values to a C struct of the same key name?

Thanks,
Giovanni

[Jeroen Demeyer]

On 2016-11-24 18:31, Giovanni Torres wrote:
> Any suggestions on how to assign python dictionary key values to a C
> struct of the same key name?

I don't think you can do that in plain Cython. One possible option is
auto-generating Cython code generating 40 lines of the form

descriptor_msg.name = a["name"]

[Giovanni Torres]

The problem I run into is that users can define dictionaries with a random combination of key/value pairs.  Therefore, I couldn't just do:

    descriptor_msg.name = a["name"]

Or I could get a KeyError exception if "name" is not provided in the dictionary.  In this case, I would have to put each assignment inside a try/except block, but that would be ugly for a struct of 40 or so members.

I even tried:
  
    descriptor_msg.name = a.get("name")

But I get "Storing unsafe C derivative of temporary Python reference", which I understand I shouldn't do.

The only solution I've come up with so far is the try/except block:

    try:
        descriptor_msg.name = a["name"]
    except KeyError:
        descriptor_msg.name = NULL                 


I was hoping for a better way.

Thanks,
Giovanni

[Robert Bradshaw]

You could also do

descriptor_msg.name = a["name"] if "name" in a else NULL

(might need an explicit cast for that first clause). I suppose
__getindex__ could also come with the (possibly) unsafe temp warning,
but it's usually safe (especially if a is a dict).

[Giovanni Torres]

When I do this, I get:

     descriptor_msg.name = a["name"] if "name" in a else NULL

                                                                                  ^
     ------------------------------------------------------------
     test/a.pyx:965:75: Cannot convert 'void *' to Python object


If I change NULL to None, Cython is happy and it compiles, but then when I call the function and don't specify "name", for example, I get:

>>> import test
>>> a = {"foo": "bar"}
>>> test.submit(a)
TypeError: expected string or Unicode object, NoneType found

name is declared as "char *name", and the TypeError above points to the same line number 965 above with the if statement.

I'm using Cython 0.24.1.

Thanks,
Giovanni

[Robert Bradshaw]

On Sat, Nov 26, 2016 at 10:22 AM, Giovanni Torres

Sorry, by "extra cast" I meant

descriptor_msg.name = <char*>a["name"] if "name" in a else NULL

You could also do

if "name" in a:
descriptor_msg.name = a["name"]
else:
descriptor_msg.name = NULL

- Robert

[Giovanni Torres]

This is what I ended up doing.  Thanks!

- Giovanni

[Chris Barker]

Just curious, why doesn't:

descriptor_msg.name = a.get("name", None)

work?

(maybe with a cast?)

That's how you'd do it in a Python.

To the OP:

 -- a dict isn't really analogous to a struct in C -- maybe use a namedtuple? Or a cdef class?

If you really want to give your users  a dict interface, you could populate a namedtuple or class from the dict, and then use that to populate your C struct -- not all the fast, but is setting a struct field performance critical?

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Robert Bradshaw]

On Mon, Nov 28, 2016 at 9:27 AM, Chris Barker <chris....@noaa.gov> wrote:
> Just curious, why doesn't:
>
> descriptor_msg.name = a.get("name", None)
>
> work?
>
> (maybe with a cast?)
>
> That's how you'd do it in a Python.

Because you can't assign None to a char*. Automatically interpreting
None as NULL might lead to surprises. You could possibly do something
like

descriptor_msg.name = null_for_none(a.get("name", None))

> To the OP:
>
> -- a dict isn't really analogous to a struct in C -- maybe use a
> namedtuple? Or a cdef class?
>
> If you really want to give your users a dict interface, you could populate
> a namedtuple or class from the dict, and then use that to populate your C
> struct -- not all the fast, but is setting a struct field performance
> critical?
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris....@noaa.gov
>

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Chris Barker]

On Mon, Nov 28, 2016 at 10:37 AM, Robert Bradshaw <robe...@gmail.com> wrote:

> Just curious, why doesn't:
>
> descriptor_msg.name = a.get("name", None)
>
> work?
>
> (maybe with a cast?)
>
> That's how you'd do it in a Python.

Because you can't assign None to a char*.

aw -- I hadn't looked closely enough -- this is specifically about the char*

kinda makes me wonder if you want a NULL pointer there as the default -- maybe an "empty" string would be better? i.e. pointer to a single char of value 0.

But that's assuming the char* is being used for text, not arbitrary data, I guess.

-CHB

[Giovanni Torres]

On Monday, November 28, 2016 at 12:28:04 PM UTC-5, Chris Barker wrote:

To the OP:

 -- a dict isn't really analogous to a struct in C -- maybe use a namedtuple? Or a cdef class?

If you really want to give your users  a dict interface, you could populate a namedtuple or class from the dict, and then use that to populate your C struct -- not all the fast, but is setting a struct field performance critical?

I was thinking of using a dict because I thought maybe some sort of autoconversion was possible.  I did come across your post: https://groups.google.com/forum/#!topic/cython-users/xAaSFg_otW4

This post wasn't specifically about char* and NULLs vs NoneTypes, but that does come up while deciding what interface to give to users.  Below is a partial snippet of the struct I am working with and how I initialize it:


typedef struct job_desc_msg_t {
    char *account;
    char *acctg_freq;
    char *alloc_node;
    uint16_t alloc_resp_port;
    uint32_t alloc_sid;
    uint32_t argc;
    char **argv;
    char *array_inx;
    void *array_bitmap;
    time_t begin_time;
    uint32_t bitflags;
    char *burst_buffer;
    uint16_t ckpt_interval;
    char *ckpt_dir;
    char *clusters;
    char *comment;
    uint16_t contiguous;
    uint16_t core_spec;
    char *cpu_bind;
    uint16_t cpu_bind_type;
    uint32_t cpu_freq_min;
    uint32_t cpu_freq_max;
    uint32_t cpu_freq_gov;
    time_t deadline;
    [...]


cdef:
    job_desc_msg_t job_mesg
    submit_response_msg_t *resp_mesg

   slurm_init_job_desc_msg(&job_mesg)
   job_mesg.name = mydict["name"]
   job_mesg.comment = mydict["comment"]
 
   slurm_submit_batch_job(&job_mesg, &resp_mesg)



The struct has several members that are set by an external controller daemon.  In other cases, the controller will set default values when none are provided.  That's why I was hesitant to set char*'s as an empty string, i.e "", when the controller would either set its own default or probably leave it as NULL.  The other note is that users can supply any combination of values to that struct.   I will also look at using a cdef class instead of a dict.  If I use a dict, I'm still stuck with checking the types in case a user submits an int when it should have been a string, for example.

Thanks for the suggestions!  I'll give that a try and see how it goes.

Giovanni

[Chris Barker]

On Mon, Nov 28, 2016 at 4:36 PM, Giovanni Torres <giovann...@gmail.com> wrote:

  I will also look at using a cdef class instead of a dict.  If I use a dict, I'm still stuck with checking the types in case a user submits an int when it should have been a string, for example.

with a cdef class, Cython will take care of that for you :-)

-CHB