Alignment Issues with *ManagedStruct?
Chromatic
While adding support for handling keyboard events to the SDL bindings
(see the attached patch; it's not for applying, as the documentation is
lacking and the interface exposes too many details), I discovered that
the alignment of members within a struct matters quite a bit.
That is, to make the keyed struct work correctly, I had to add extra
bytes of padding in the appropriate places. This works, at least on my
platform, but I'm not sure how portable it is.
If you have the SDL headers installed on your system, I'm trying to make
the Parrot equivalent of an SDL_Event, the mess that it is. In
particular, I added one byte of padding before keysym and three bytes
before keysym.sym.
Is this indeed unportable? If so, is there a way to work around this
without writing C code for every afflicted structure? Is it a case of
making the struct PMCs smarter? Or have I fixed it with my nasty
workaround?
-- c
Leopold Toetsch
> That is, to make the keyed struct work correctly, I had to add extra
> bytes of padding in the appropriate places.
Have a look at the third initializer param - this is the offset of the
item in bytes.
(Albeit untested - seems you got the code to test it :)
> Is this indeed unportable? If so, is there a way to work around this
> without writing C code for every afflicted structure? Is it a case of
> making the struct PMCs smarter? Or have I fixed it with my nasty
> workaround?
NCI is per se unportable. But the *Struct PMC could of course be smarter
and calculate common align requirements. There are of course still
issues if you have to deal with packed structures.
> -- c
leo
Chromatic
> Have a look at the third initializer param - this is the offset of the
> item in bytes.
Oh, right. That completely slipped my mind.
> (Albeit untested - seems you got the code to test it :)
Okay, I'll turn this into a test case.
> NCI is per se unportable. But the *Struct PMC could of course be smarter
> and calculate common align requirements. There are of course still
> issues if you have to deal with packed structures.
Someone here mentioned that it's trivial to detect struct alignments in
C. If the configuration doesn't already do that, make it do that, then
add a bit of code to the struct PMC to align the element correctly?
set layout['member'], .DATATYPE_INT
push layout, 0
push layout, 0
set layout['alignedmember'], .DATATYPE_UINT16
push layout, 0
push layout, 'align'
Maybe that's not the right interface, but I'd like something nearly that
simple.
Is this one for the TODO list, or did I catch a pumpking in good
humours?
-- c
Leopold Toetsch
> set layout['member'], .DATATYPE_INT
> push layout, 0
> push layout, 0
Pushing zero as offset should do The Right Thing, that is proper
aligning the item. Tighter packed structures can be achieved with
explicit offset parameters.
Its basically an one-liner in unmanagedstruct.pmc:205
if (offs == 0 && toff % data_size)
advance (offs, toff) to data_size alignment
> Is this one for the TODO list, or did I catch a pumpking in good
> humours?
Takers wanted :)
> -- c
leo
Chromatic
> Pushing zero as offset should do The Right Thing, that is proper
> aligning the item. Tighter packed structures can be achieved with
> explicit offset parameters.
In this case, it doesn't, as the struct I'm emulating is:
typedef struct {
Uint8 type; /* SDL_KEYDOWN or SDL_KEYUP */
Uint8 which; /* The keyboard device index */
Uint8 state; /* SDL_PRESSED or SDL_RELEASED */
SDL_keysym keysym;
} SDL_KeyboardEvent;
SDL_keysym itself is a struct:
typedef struct {
Uint8 scancode; /* hardware specific */
SDLKey sym; /* SDL virtual keysym */
SDLMod mod; /* current key modifiers */
Uint16 unicode; /* translated character */
} SDL_keysym;
SDLKey and SDLMod are both enums.
As I understand it (and correct me if I'm wrong), SDL_keysym needs a
byte of padding on my architecture within SDL_KeyboardEvent. I can add
that padding manually, but I'm not convinced that'll be portable to
different architectures.
That's the kind of padding I'd like unmanagedstruct.pmc to be able to do
for me.
> > Is this one for the TODO list, or did I catch a pumpking in good
> > humours?
> Takers wanted :)
I'm happy to submit this as a TODO if I have the story straight.
-- c
Leopold Toetsch
> In this case, it doesn't, as the struct I'm emulating is:
> typedef struct {
> Uint8 type; /* SDL_KEYDOWN or SDL_KEYUP */
> Uint8 which; /* The keyboard device index */
> Uint8 state; /* SDL_PRESSED or SDL_RELEASED */
> SDL_keysym keysym;
> } SDL_KeyboardEvent;
> SDL_keysym itself is a struct:
> typedef struct {
> Uint8 scancode; /* hardware specific */
> SDLKey sym; /* SDL virtual keysym */
> SDLMod mod; /* current key modifiers */
> Uint16 unicode; /* translated character */
> } SDL_keysym;
> SDLKey and SDLMod are both enums.
> As I understand it (and correct me if I'm wrong), SDL_keysym needs a
> byte of padding on my architecture within SDL_KeyboardEvent.
Brr. I don't know. I've to ask my debugger for that :) If you have
SDL_keysym alone, that needs 3 bytes additional alignment. But don't ask
me about the combined structure.
But I'd say that there are four adjacent Uint8 taking one word, then is
another word (SDLkey sym). Or a structure starts aligned ...
We don't have a notion of nested structures. They could go just inline,
i.e. without extra syntax. Pointers to structures are of course missing
too, that's a different thingy.
> I can add
> that padding manually, but I'm not convinced that'll be portable to
> different architectures.
Automatic alignment based on the data type sizes is for sure better.
> -- c
leo
Chromatic
> > As I understand it (and correct me if I'm wrong), SDL_keysym needs a
> > byte of padding on my architecture within SDL_KeyboardEvent.
> Brr. I don't know. I've to ask my debugger for that :) If you have
> SDL_keysym alone, that needs 3 bytes additional alignment. But don't ask
> me about the combined structure.
Exactly my worry. Does it depend on the compiler? Architecture?
Libraries? C specification? Phase of the moon?
> But I'd say that there are four adjacent Uint8 taking one word, then is
> another word (SDLkey sym). Or a structure starts aligned ...
I ran this program in lieu of the debugger:
#include <stdio.h>
#include <stdlib.h>
#include <SDL/SDL_events.h>
int main ()
{
SDL_KeyboardEvent kbevent;
printf( "0x%08X event\n", &kbevent );
printf( "0x%08X event.type\n", &kbevent.type );
printf( "0x%08X event.which\n", &kbevent.which );
printf( "0x%08X event.state\n", &kbevent.state );
printf( "0x%08X event.keysym.scancode\n", &kbevent.keysym.scancode
);
printf( "0x%08X event.keysym.sym\n", &kbevent.keysym.sym );
printf( "0x%08X event.keysym.mod\n", &kbevent.keysym.mod );
printf( "0x%08X event.keysym.unicode\n", &kbevent.keysym.unicode );
return(0);
}
Here's the output on my machine:
0x7FFFF7E8 event
0x7FFFF7E8 event.type
0x7FFFF7E9 event.which
0x7FFFF7EA event.state
0x7FFFF7EC event.keysym.scancode
0x7FFFF7F0 event.keysym.sym
0x7FFFF7F4 event.keysym.mod
0x7FFFF7F8 event.keysym.unicode
> We don't have a notion of nested structures. They could go just inline,
> i.e. without extra syntax. Pointers to structures are of course missing
> too, that's a different thingy.
Inlining's not a problem; it only has to be done once. This is one spot
where C's lack of abstraction actually helps. :)
-- c
Uri Guttman
c> I ran this program in lieu of the debugger:
i recall an old trick i learned to get offsets into structs which would
be easier to read than hex addresses:
c> int main ()
c> {
c> SDL_KeyboardEvent kbevent;
c> printf( "0x%08X event\n", &kbevent );
c> printf( "0x%08X event.type\n", &kbevent.type );
<untested>
printf( "%d event.type\n", (char *)&kbevent.type ) - (char *)&kbevent ;
it can be made into a macro IIRC for ease of use.
that could be done with a perl script for any structure and it could
generate an offset map for the given environment. that map could be used
by parrot to access members in a portable way.
so the script would parse a structure, generate a set of those prints
(with the right format output) and compile it with the same tools as
parrot. run that little c program and save the output in a file. parrot
reads that file and uses it as a name to offset map for that structure.
want me to hack up this little script and c generation stuff? the hard
part is parsing the struct so i would have to assume some simple format
and not full c for the moment. the only thing needed by the parser is
all the member names.
uri
--
Uri Guttman ------ u...@stemsystems.com -------- http://www.stemsystems.com
--Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
Leopold Toetsch
> printf( "%d event.type\n", (char *)&kbevent.type ) - (char *)&kbevent ;
offsetof(struct, item)
is used inside parrot/jit/*
> want me to hack up this little script and c generation stuff? the hard
> part is parsing the struct so i would have to assume some simple format
> and not full c for the moment. the only thing needed by the parser is
> all the member names.
Very much appreciated.
I thought of that too. A Perl script that takes a C struct and emits an
*ManagedStruct initializer. WRT align: as such struct initializers are
in library code and used by different machines, I'd rather have the
alignment calculation inside the unmanagedstruct.pmc.
But as a last resort this script could regenerate the offsets for a
particular machine.
## &gen_struct()
# struct event_t {
# char x; /* optional comment */
# char y;
# int flags;
# };
## &end_gen
## autogenerated from above template
## don't modifiy - rerun gen_struct $file
.local pmc event_t_struct_init
.include "datatypes.pasm"
event_t_struct_init = new .OrderedHash
event_t_struct_init["x"] = .DATATYPE_CHAR # optional comment
event_t_struct_init[1] = 0 # no array of items
event_t_struct_init[2] = 0 # automatic offset
event_t_struct_init["y"] = .DATATYPE_CHAR
event_t_struct_init[4] = 0 # no array of items
event_t_struct_init[5] = 0 # automatic offset
event_t_struct_init["flags"] = .DATATYPE_INT
event_t_struct_init[7] = 0 # no array of items
event_t_struct_init[8] = 0 # automatic offset _or_ 4/8
## end autogen
gen_struct --force-align file.imc
could fill in the correct offset in the last line.
Running gen_struct the first time should produce the autogenerated part.
For PASM only, it could look like:
## &gen_struct(P23)
...
new P23, .OrderedHash
set P23["x"], ...
You might have a look at F<runtime/parrot/include/datatypes.pasm> which
is autogenerated too. *Struct.pmc doesn't yet handle all types, but this
will be fixed.
> uri
leo
Jens Rieks
> Uri Guttman <u...@stemsystems.com> wrote:
> > printf( "%d event.type\n", (char *)&kbevent.type ) - (char *)&kbevent ;
>
> offsetof(struct, item)
>
> is used inside parrot/jit/*
>
> > want me to hack up this little script and c generation stuff? the hard
> > part is parsing the struct so i would have to assume some simple format
> > and not full c for the moment. the only thing needed by the parser is
> > all the member names.
>
> Very much appreciated.
>
>
> I thought of that too. A Perl script that takes a C struct and emits an
> *ManagedStruct initializer. WRT align: as such struct initializers are
> in library code and used by different machines, I'd rather have the
> alignment calculation inside the unmanagedstruct.pmc.
structs, enums, unions and typedefs in order to create ManagedStructs
automatically.
If one goes even a step further, it should even be possible to create pasm/imc
wrapper for C functions automatically.
The tokeniser is already working, but the token processing is of course not
very simple, but I think I will have a first working alpha version within the
next few days.
> leo
jens
Leopold Toetsch
[ align issues ]
Nested structs are ugly. The alignment of the first item seems to depend
on the biggest item in the struct.
struct {
char // 0
struct {
char // 1
}
char // 2
}
struct {
char // 0
struct {
char // 4
int // 8
}
}
So we need some notion of nested structs or a hint for correct
alignment. I've checked in basic alignment handling for "normal"
cases, but not the second one above.
Below is a test program to experiment with that stuff.
leo
#include <stdlib.h>
#include <stddef.h>
#include <stdio.h>
// offsetof expands to ((size_t) &(( struct xt *)0)-> x )
struct xt {
char x;
struct yt {
char i;
int j; // use different items here
} _y;
char z;
} _x;
int main(void) {
printf("x : %d\n", offsetof(struct xt, x));
printf("i : %d\n", offsetof(struct xt, _y.i));
printf("j : %d\n", offsetof(struct xt, _y.j));
printf("z : %d\n", offsetof(struct xt, z));
return 0;
}
Leopold Toetsch
> gen_struct --force-align file.imc
It seems that this part finally has to consult the C compiler, i.e.
generate a test program and parse the offsetof() values of struct items.
The problem is nested structs, s. classes/unmanagedstruct.pmc for some
comments and the test program in another f'up in this thread.
$gcc-source/stor-layout.c seems to deal with that, but its a bit
unreadable :)
leo
Leopold Toetsch
> I'am currently writing a simple C-parser in imc. My plan is to extract
> all structs, enums, unions and typedefs in order to create
> ManagedStructs automatically. If one goes even a step further, it
> should even be possible to create pasm/imc wrapper for C functions
> automatically. The tokeniser is already working, but the token
> processing is of course not very simple, but I think I will have a
> first working alpha version within the next few days.
Sounds great. Not too simple to write that al in PIR ;)
leo
Tim Bunce
> >
> >
> > I thought of that too. A Perl script that takes a C struct and emits an
> > *ManagedStruct initializer. WRT align: as such struct initializers are
> > in library code and used by different machines, I'd rather have the
> > alignment calculation inside the unmanagedstruct.pmc.
>
> I'am currently writing a simple C-parser in imc. My plan is to extract all
> structs, enums, unions and typedefs in order to create ManagedStructs
> automatically.
> If one goes even a step further, it should even be possible to create pasm/imc
> wrapper for C functions automatically.
> The tokeniser is already working, but the token processing is of course not
> very simple, but I think I will have a first working alpha version within the
> next few days.
Have you looked at http://search.cpan.org/~grichter/ExtUtils-XSBuilder/ ?
Tim.
Uri Guttman
LT> Chromatic <chro...@wgz.org> wrote:
LT> [ align issues ]
LT> Nested structs are ugly. The alignment of the first item seems to depend
LT> on the biggest item in the struct.
that is a known rule for alignment of nested structs. since larger items
usually have strict alignment requirements on many platforms (misaligned
accesses either fail with bus faults or pay a massive penalty as on the
alpha), the compiler has to align the whole struct so that item will be
aligned. it can't just pad internally since that could vary. a char
followed by a 32 bit item could need 1,2, or 3 bytes of padding
depending on where the struct starts. so cc will force that struct to 32
bit alignement and pad 3 bytes after the char to keep the 32 bit item
aligned.
LT> So we need some notion of nested structs or a hint for correct
LT> alignment. I've checked in basic alignment handling for "normal"
LT> cases, but not the second one above.
you just need to do the same thing in offsetof but do it for the nested
struct elements. it won't necessarily be the same as when that struct is
top level so you can't cheat there.
it sounds like between the PIR solution and the cpan module that i don't
need to do my own hack. or should i still work on it? it would be an
external perl5/c solution that would be able to generate some form of
table of item offsets.
Leopold Toetsch
> LT> Nested structs are ugly. The alignment of the first item seems to depend
> LT> on the biggest item in the struct.
> that is a known rule for alignment of nested structs. since larger items
> usually have strict alignment requirements
Ok. Then I'll try to put that in unmanagedstruct.pmc. Seems not be too
hard.
> it sounds like between the PIR solution and the cpan module that i don't
> need to do my own hack. or should i still work on it? it would be an
> external perl5/c solution that would be able to generate some form of
> table of item offsets.
I'll do the alignment stuff. An util that actually generates the
initializer - as outlined in the previous posting - would be great
though.
Pointers to nested structs and function pointers as struct items are
already in CVS (at least, when returned from the external C lib) - I'll
update the docs/pmc/struct.pod tommorrow, including nested structures.
> uri
leo
Uri Guttman
LT> I'll do the alignment stuff. An util that actually generates the
<normally i would never comment on your english which is usually very
good but util(ity) is pronounced with a hard 'u' as in 'you'. so that
should be 'a' and not an>.
LT> initializer - as outlined in the previous posting - would be great
LT> though.
i am playing with ExtUtil::XSParser now and i have some problems with
it. i will be writing the author. does anyone on this list use it now? i
can send in what i have done. it seems to be more a work in progress
than something useable but i could be wrong. at least one method i need
to override (find_includes) was not described properly in the docs and i
had to scan the source to make it work. also the P:RD parser doesn't seem
to return a parse tree but some ok (1) values. any help would be
appreciated. i mainly need the c struct parser to give me a tree and i
can do the rest.
thanx,
Uri Guttman
LT> Uri Guttman <u...@stemsystems.com> wrote:
>> >>>>> "LT" == Leopold Toetsch <l...@toetsch.at> writes:
>> i am playing with ExtUtil::XSParser now
LT> ... but I hope that people out there are speaking Perl and can help with
LT> that issue.
mitchell charity pointed me to a much better module
Convert::Binary::C. it looks perfect and seems robust and it is very
well documented. i just installed (10747 tests!) it and will play with
it soon. if it does what it claims, it will be trivial to generate
anything we need. all it needs is a couple of calls to parse any c
source and one method will return a list of all the structs with a fully
broken out parse tree with offsets! i will hack up something basic
using it and we can tweak the output to whatever parrot needs.
Leopold Toetsch
> LT> I'll do the alignment stuff. An util that actually generates the
> <normally i would never comment on your english which is usually very
> good but util(ity) is pronounced with a hard 'u' as in 'you'. so that
> should be 'a' and not an>.
Hey - thanks for that. You did hear my talk in Paris :) I had really
bad English teachers. Grammatics wasn't one of my favorites either.
Anyway, if I'm stampering on someone's language feet, I'd like to
apologize in advance. Please just correct me. We don't even speak German
here ...
> LT> initializer - as outlined in the previous posting - would be great
> LT> though.
> i am playing with ExtUtil::XSParser now
... but I hope that people out there are speaking Perl and can help with
that issue.
> uri
leo
Uri Guttman
boy, was this easy with this module. all we need to do is mess around
with the output to get whatever leo needs.
with this struct (from leo with some minor changes):
struct xt {
char x;
struct yt {
char i,k;
int j;
} Y;
char z;
} X;
and this trivial script (i pass the filename on the command line):
#!/usr/local/bin/perl -w
use strict ;
use Data::Dumper ;
use Convert::Binary::C ;
my $cbc = Convert::Binary::C->new() ;
$cbc->parse_file( @ARGV ) ;
#print Dumper $cbc->compound() ;
my @structs = $cbc->compound() ;
foreach my $struct ( @structs ) {
print "$struct->{'type'} $struct->{'identifier'}\n" ;
foreach my $declaration ( @{$struct->{'declarations'}} ) {
my $type = $declaration->{'type'} ;
foreach my $declarator ( @{$declaration->{'declarators'}} ) {
my $decl_name = $declarator->{'declarator'} ;
my $offset = $declarator->{'offset'} ;
print "\t$type $decl_name : offset $offset\n" ;
}
}
}
i get this lovely output:
struct yt
char i : offset 0
char k : offset 1
int j : offset 2
struct xt
char x : offset 0
struct yt Y : offset 1
char z : offset 7
this was part of the structure parse tree (via Dumper):
$VAR2 = {
'context' => 'xyz.h(3)',
'declarations' => [
{
'declarators' => [
{
'declarator' => 'x',
'offset' => 0,
'size' => 1
}
],
'type' => 'char'
},
{
'declarators' => [
{
'declarator' => 'Y',
'offset' => 1,
'size' => 5
}
],
'type' => 'struct yt'
},
{
'declarators' => [
{
'declarator' => 'z',
'offset' => 6,
'size' => 1
}
],
'type' => 'char'
}
],
'pack' => 0,
'align' => 1,
'size' => 7,
'identifier' => 'xt',
'type' => 'struct'
};
BTW, this was on a sparc/solaris box.
so what output formats do we need?
the structures parse tree (from the compound method) has everything you
could want and is very simple to navigate. this is one amazing module
and it should get more publicity for sure.
Gordon Henriksen
> Nested structs are ugly. The alignment of the first item seems to depend
> on the biggest item in the struct.
>
> [...]
Yeah, the struct is aligned to align the largest element therein. e.g.:
struct { char; struct { char; char; char; } }
is at +0, +1, +2, +3
But: struct { char; struct { char; char; short; } }
is at +0, +2, +3, +4
And: struct { char; struct { char; short; char; } }
is at +0, +2, +4, +6
> So we need some notion of nested structs or a hint for correct
> alignment. I've checked in basic alignment handling for "normal"
> cases, but not the second one above.
Maybe you ought to capitulate to the hierarchical nature of it all and
simply push on another struct layout specifier to handle the nesting.
> struct xt {
> char x;
> struct yt {
> char i;
> int j; // use different items here
> } _y;
> char z;
> } _x;
>
> int main(void) {
> printf("x : %d\n", offsetof(struct xt, x));
> printf("i : %d\n", offsetof(struct xt, _y.i));
> printf("j : %d\n", offsetof(struct xt, _y.j));
> printf("z : %d\n", offsetof(struct xt, z));
> return 0;
> }
Mac OS X output is:
x : 0
i : 4
j : 8
z : 12
—
Gordon Henriksen
mali...@mac.com
Leopold Toetsch
> Maybe you ought to capitulate to the hierarchical nature of it all and
> simply push on another struct layout specifier to handle the nesting.
Exactly that's the plan:
.DATATYPE_STRUCT
.DATATYPE_STRUCT_PTR
are already in CVS.
leo
Paolo Molaro
> with this struct (from leo with some minor changes):
>
> struct xt {
> char x;
> struct yt {
> char i,k;
> int j;
> } Y;
> char z;
> } X;
>
> and this trivial script (i pass the filename on the command line):
> i get this lovely output:
>
> struct yt
> char i : offset 0
> char k : offset 1
> int j : offset 2
> struct xt
> char x : offset 0
> struct yt Y : offset 1
> char z : offset 7
> BTW, this was on a sparc/solaris box.
The offsets look incorrect. On basically every modern 32 or 64 bit
OS (with sizeof(int) == 4) it should look like:
struct yt (size=8, alignment=4)
char i : offset 0
char k : offset 1
int j : offset 4
struct xt (size=16, alignment=4)
char x : offset 0
struct yt Y : offset 4
char z : offset 12
lupus
--
-----------------------------------------------------------------
lu...@debian.org debian/rules
lu...@ximian.com Monkeys do it better
Leopold Toetsch
> The offsets look incorrect. On basically every modern 32 or 64 bit
> OS (with sizeof(int) == 4) it should look like:
Yeah. But in the meantime Parrot should calculate correct offsets :)
> lupus
leo
Leopold Toetsch
> boy, was this easy with this module. all we need to do is mess around
> with the output to get whatever leo needs.
s/leo/Joe R. Parrot Hacker/ - I can craft initializers by hand ;)
1) some script e.g. gen_struct (struct2pasm ...) located in tools/dev or
build_tools.
2) options:
--gen-pasm, -p (default, *if* src file =~ /\.pasm$/
--pmc=Px (for --gen-pasm only, default P15)
--gen-pir (default)
--named-initializer=0,1 (1)
--named-accessor=0,1 (0)
--out-file, -o=file (default: change input in place unless stdin)
3) Operation
3a) Take a C structure, spit out initializer
3b) take a commented C structure, add or change initializer
3b) is for source-files that may contain multiple C structures
like in my original posting WRT this util or like below
> with this struct (from leo with some minor changes):
## &gen_struct(--options, --pmc=P20, --gen-pasm)
# struct xt {
# char x;
# struct yt {
# char i,k;
# int j;
# } Y;
# char z;
#} X;
## &end_gen
## begin autogeneratedd bla bla
4) The output
4a) initializer name / P-register => $init_pmc
new P20, .OrderedHash # .PerlArray if --no-named-initializer
or
.local pmc struct_xt_init # --gen-pir
struct_xt_init = new OrderedHash
and
.include "datatypes.pasm" # for DATATYPE_ consts below
4b) data type of item
push $init_pmc, .DATATYPE_$type # --no-named-init...
or
set $init_pmc["$var"], .DATATYPE_$type # --name-init..
If item is a nested struct or a pointer to nested struct please
consult docs/pmc/struct.pod
4c) item count = 0, or N for array of items
push $init_pmc, $item_count
4d) offset - always 0 for now
push $init_pmc, 0
4e) nice to have: preserve C comments (put these after '#')
S. also t/pmc/nci.t for examples of initializers and src/nci_test.c for
the referenced C structs.
I hope above outline is somewhat clear else please just ask.
> uri
leo