Dealing with typedef'd struct fields

[Floris Bruynooghe]

Hi All,

Apologies for the fairly long question, I hope I'm not entirely on the
wrong path.

I'm trying to read a struct's members from a file in /proc (on
Solaris).  Unfortunately some of the members of this struct are
actually typedef'd in header files meaning that cffi can not parse the
struct.  It looks something like this:

   typdef stuct psinfo {
       pid_t pr_pid;
       timestruct_t pr_start;
       /* more fields here */
   } psinfo_t;

As far as I understand the only way to really get to the fields in
psinfo_t is to write a separate function/library similar to this:

   #include <sys/proc.h>  /* declares psinfo_t, pid_t, etc. */

   struct simple_psinfo {
       long pr_pid;
       long long pr_start_sec;
       long long pr_start_nsec;
       /* more fields here */
   };

   struct simple_psinfo*
   read_psinfo(psinfo_t *psinfo)
   {
       struct simple_psinfo* simple;

       simple = malloc(sizeof(struct simple_psinfo));
       if (simple == NULL)
           return NULL;
       simple->pr_pid = (long)psinfo->pr_pid;
       simple->ps_start_sec = (long long)psinfo->pr_start.tv_sec;
       simple->ps_start_nsec = (long long)psinfo->pr_start_tv_nsec;
       /* more fields here */
       return simple;
   }

Here the compiler would warn if I do a cast which does not work I
guess, solving the problem with not knowing the type.  This is the
equivalent of the casts in PyLong_FromLong((long)psinfo->pr_pid) in
CPython's C API.

I think the corresponding cffi code to access this would then be:

   ffi.cdef("""
       typedef struct psinfo {
           ...
       } psinfo_t;
       struct simple_psinfo {
           long pr_pid;
           long long pr_start_sec;
           long long pr_start_nsec;
           /* more fields here */
       };
       struct simple_psinfo* read_psinfo(psinfo_t *psinfo);
   """)
   C = ffi.verify("""
       #include <sys/proc.h>
   """)
   psinfo = ffi.new('psinfo_t')
   with open('data', 'rb') as fp:
       fp.readinto(ffi.buffer(psinfo))
   simple_psinfo = C.read_psinfo(psinfo)
   print simple_psinfo.pr_pid
   del simple_psinfo

So my questions are:

Firstly, is this the correct way of doing this?

If so, is it acceptable to move the read_psinfo function into the
ffi.verify() call?  This would eliminate having to declare struct
simple_psinfo twice and having to do a bunch of stuff to build the
library, find it and pass it to cffi.

And finally, is there no way of doing this simpler?  It would be nice
if cffi could deal with opaque types and ask the compiler to do the
casting.  If there was some way to say "cast pid_t to long" then this
might be much simpler to do?  I don't think it's possible to just
declare to the casted types directly in the struc psinfo of ffi.cdef()
is it?


Thanks,
Floris

[Floris Bruynooghe]

Hello,

As a follow up post to my own question, I've got this to work kind of like I described earlier.  I've quoted the full working code below, but I'd still be interested to know if this is "the right way" to achieve this.

Specifically I do still have to declare the struct with only supported cffi types twice.  And secondly I'm still writing a conversion function in the .verify() block.

Any feedback on whether this is roughly the right way would be appreciated.

Regards,
Floris


ffi = cffi.FFI()

ffi.cdef("""
typedef struct psinfo { ...; } psinfo_t;

struct argv_psinfo {
    long pr_pid;
    char pr_psargs[100];
int argv_psiinfo(struct argv_psinfo *res, psinfo_t *psinfo);
""")
C = ffi.verify("""
#undef _FILE_OFFSET_BITS

#include <alloca.h>
#include <string.h>
#include <procfs.h>

struct argv_psinfo {
    long pr_pid;
    char pr_psargs[100];
};

int
argv_psinfo(struct argv_psinfo *res, psinfo_t *psinfo)
{
    res->pr_pid = psinfo->pr_pid;
    strlcpy(res->pr_psargs, psinfo->pr_psargs, 100);
    return 0;
}
""")

psinfo = ffi.new('psinfo_t*')
with io.open('/proc/{}/psinfo'.format(os.getpid()), 'rb') as fp:
    fp.readinfo(ffi.buffer(psinfo)
argv_psinfo = ffi.new('struct argv_psinfo *')
C.argv_psinfo(argv_psinfo, psinfo)
print(argv_psinfo.pr_pid, ffi.string(argv_psinfo.pr_psargs)

[Armin Rigo]

Hi Floris,

On Fri, Dec 6, 2013 at 3:20 PM, Floris Bruynooghe
<floris.b...@gmail.com> wrote:
> As a follow up post to my own question, I've got this to work kind of like I
> described earlier. I've quoted the full working code below, but I'd still
> be interested to know if this is "the right way" to achieve this.

Sorry for the delay in answering. You got the best workaround I can
think of, yes, but it's indeed a workaround.

A proper solution would be a bit involved, but possible. I'm thinking
about something like this: you'd say in the cdef:

typedef struct {
int... sec;
int... nsec;
} timestruct_t;
typdef stuct psinfo {
int... pr_pid;
timestruct_t pr_start;
char pr_psargs[...]; /* <- aready works */
} psinfo_t;

Here, "int..." would mean "this field is some integer, but I don't
know its size or signedness". You could also say

typedef int... pid_t;

and use pid_t as the field type. The same with "float..." or
"double...". There are complications to implement this, but now that
I think about it, if such "int..." types are only used as field types
in structs, then it's easier; so maybe we should start with supporting
only this case.


A bientôt,

Armin.

[Floris Bruynooghe]

Hello Armin,

On 15 December 2013 08:11, Armin Rigo <ar...@tunes.org> wrote:
> Hi Floris,
>
> On Fri, Dec 6, 2013 at 3:20 PM, Floris Bruynooghe
> <floris.b...@gmail.com> wrote:
>> As a follow up post to my own question, I've got this to work kind of like I
>> described earlier. I've quoted the full working code below, but I'd still
>> be interested to know if this is "the right way" to achieve this.
>
> Sorry for the delay in answering.

No worries.

> You got the best workaround I can
> think of, yes, but it's indeed a workaround.
>
> A proper solution would be a bit involved, but possible.

[...]

> Here, "int..." would mean "this field is some integer, but I don't
> know its size or signedness". You could also say
>
> typedef int... pid_t;

This would be very nice indeed and is kind of close to what I do
currently (thanks to some hints from fijal on irc). Essentially
instead of having functions that take an opaque struct and return an
newly allocated ones I just have a function which verifies the sizes
of the int-like typedefs and call that once:

ffi = cffi.FFI()
ffi.cdef("""
typedef long pid_t;

typedef struct psinfo {
pid_t pr_pid;
char pr_psargs[...];
} psinfo_t;
""")
CAPI = ffi.verify("""
#include <procfs.h>

int
check_types(void)
{
if (sizeof(long) != sizeof(pid_t))
return -1;
return 0;
}
""")
if CAPI.check_types() < 0:
raise RuntimeError('oops')

Doing this means the allocations are easier to manage as no functions
are allocating memory and I can just use ffi.new('psinfo_t*') and
file.readinto(ffi.buffer(...)) as in the examples to access a struct.
And this would become even easier with the typedef syntax you propose
as the check_types() function would not be needed.


When declaring "char pr_psargs[...]" I was actually quite surprised
this works (and beforehand I was doing two .verify() calls, one to
extract the size using "#define PRARGSZ ..." and one to use it).
Could someone describe how cffi can find out the size of this without
even knowing it's the size of the PRARGSZ macro? And are there any
limitations to this magic? I.e. is there a way to declare something
which makes it impossible for cffi to work this out?


Thanks,
Floris


--
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

[Chris Emerson]

On Sun, Dec 15, 2013 at 09:11:41AM +0100, Armin Rigo wrote:
> A proper solution would be a bit involved, but possible. I'm thinking
> about something like this: you'd say in the cdef:
>
> typedef struct {
> int... sec;
> int... nsec;

[...]

> Here, "int..." would mean "this field is some integer, but I don't
> know its size or signedness".

Is it worth considering how to generalise this slightly to cover something
like time_t, which IIRC is allowed to be any "arithmetic type", ie float
or integer?

Regards,

Chris

[Armin Rigo]

Hi Floris,

On Sun, Dec 15, 2013 at 2:45 PM, Floris Bruynooghe <fl...@devork.be> wrote:
> This would be very nice indeed and is kind of close to what I do
> currently (thanks to some hints from fijal on irc). Essentially
> instead of having functions that take an opaque struct and return an
> newly allocated ones I just have a function which verifies the sizes
> of the int-like typedefs and call that once:

Or use the documented hack:

ffi.cdef("#define SIZE_OF_PID_T ...")
lib = ffi.verify("#define SIZE_OF_PID_T sizeof(pid_t)")
assert lib.SIZE_OF_PID_T == 4 # or use if's to use e.g. int32_t or int64_t

> When declaring "char pr_psargs[...]" I was actually quite surprised
> this works (and beforehand I was doing two .verify() calls, one to
> extract the size using "#define PRARGSZ ..." and one to use it).
> Could someone describe how cffi can find out the size of this without
> even knowing it's the size of the PRARGSZ macro? And are there any
> limitations to this magic? I.e. is there a way to declare something
> which makes it impossible for cffi to work this out?

This is documented in the section about verify(). It's not based on
automatic extraction of the name of the macro or whatever, but
instead, it uses sizeof() as well: the length of the array is found by
getting "sizeof(((struct foo *)0)->pr_psargs)".


A bientôt,

Armin.

[Floris Bruynooghe]

On 18 December 2013 15:30, Armin Rigo <ar...@tunes.org> wrote:
> On Sun, Dec 15, 2013 at 2:45 PM, Floris Bruynooghe <fl...@devork.be> wrote:
>> This would be very nice indeed and is kind of close to what I do
>> currently (thanks to some hints from fijal on irc). Essentially
>> instead of having functions that take an opaque struct and return an
>> newly allocated ones I just have a function which verifies the sizes
>> of the int-like typedefs and call that once:
>
> Or use the documented hack:
>
> ffi.cdef("#define SIZE_OF_PID_T ...")
> lib = ffi.verify("#define SIZE_OF_PID_T sizeof(pid_t)")
> assert lib.SIZE_OF_PID_T == 4 # or use if's to use e.g. int32_t or int64_t

I don't really mind doing this in the C code, feels clearer to me as
the code is less spread out.

When you earlier said you might be able to support "typedef int...
foo_t" you said it could be regardless of signedness. I can
understand how you can figure out the size of the int-like type but
how can one verify or find out the sign?

>> When declaring "char pr_psargs[...]" I was actually quite surprised
>> this works (and beforehand I was doing two .verify() calls, one to
>> extract the size using "#define PRARGSZ ..." and one to use it).
>> Could someone describe how cffi can find out the size of this without
>> even knowing it's the size of the PRARGSZ macro? And are there any
>> limitations to this magic? I.e. is there a way to declare something
>> which makes it impossible for cffi to work this out?
>
> This is documented in the section about verify(). It's not based on
> automatic extraction of the name of the macro or whatever, but
> instead, it uses sizeof() as well: the length of the array is found by
> getting "sizeof(((struct foo *)0)->pr_psargs)".

Thanks for that explanation, makes sense.


Regards,

[Armin Rigo]

Hi,

On Thu, Dec 19, 2013 at 12:37 PM, Floris Bruynooghe <fl...@devork.be> wrote:
> When you earlier said you might be able to support "typedef int...
> foo_t" you said it could be regardless of signedness. I can
> understand how you can figure out the size of the int-like type but
> how can one verify or find out the sign?

Hum, I have to think more about this question in order to find some
clean solution. There are hackish ways:

struct foo s;
memset(&s, 0xff, sizeof(s));
if (s->fieldname < 0) { it's a signed field }


A bientôt,

Armin.