Alignment and Padding / Packing of structs
Devon H. O'Dell
Someone asked about this in IRC and I wanted to know the answer since
I'm also looking at potentially using gccgo on Solaris/SPARC.
The specification says for alignment of structs: "For a variable x of
struct type: unsafe.Alignof(x) is the largest of all the values
unsafe.Alignof(x.f) for each field f of x, but at least 1."[1]
I assume that this means that if I have
type T struct {
a int64
b int32
}
any *T is guaranteed to be aligned on an 8-byte boundary.
However, nothing is explicitly said of packing or padding. I assume
that structs are packed by default (given the "_ float32 // padding"
member in [2]). I understand that one of the goals of the project is
to allow people to write portable applications with the same code, but
if we ever target SPARC or PPC:
type T struct {
a int32
b int64
}
can cause SIGBUS when writing to &T.b (and even though it works on
AMD64, it's very slow due to TLB misses) depending on the compiler
implementation. It also seems like it would be possible I asked Ian a
few days ago and he said that he'd heard gccgo works on Solaris/SPARC,
so I was curious how this works (perhaps there's something I'm missing
somewhere), or if that's just people getting lucky.
Regardless, it'd be nice to see specification on padding (since
alignment is specified), and if Go defaults to packed structs by
default, I think we should mention it somewhere for people who are
potentially using gccgo on platforms where padding absolutely matters
(I'm happy to put in the doc legwork after some enlightenment). Or if
we magically do this in gccgo, I'd like to understand how :).
Thanks,
--dho
[1]: http://golang.org/ref/spec#Size_and_alignment_guarantees
[2]: http://golang.org/ref/spec#Struct_types
Kyle Lemons
Julian Phillips
> Hi,
>
> Someone asked about this in IRC and I wanted to know the answer since
> I'm also looking at potentially using gccgo on Solaris/SPARC.
>
> The specification says for alignment of structs: "For a variable x of
> struct type: unsafe.Alignof(x) is the largest of all the values
> unsafe.Alignof(x.f) for each field f of x, but at least 1."[1]
>
> I assume that this means that if I have
>
> type T struct {
> a int64
> b int32
> }
>
> any *T is guaranteed to be aligned on an 8-byte boundary.
>
> However, nothing is explicitly said of packing or padding. I assume
> that structs are packed by default (given the "_ float32 // padding"
> member in [2]).
Doesn't seem so, in gc at least: http://play.golang.org/p/mHLAqfYW26
However, since it's not in the spec, the compiler is presumably free to
pack or not as it sees fit ...
--
Julian
Devon H. O'Dell
volgende geschreven:
You're right, I guess I could have gathered that information from
those three points. That's not the easiest bit to read, and I read
"for any type ... at least 1" and then threw that out the window when
reading 2, thinking "for any type ... unsafe.Sizeof(x)". Oops. So
yeah, alignment is fine in that it's not specified.
However, I think that the "_ float32 // padding" line should be
removed from the spec because it implies that compilers *do* pack
struct fields. Why else would you pad with a 32-bit value if you have
a type in the struct that might be 64-bit? Especially in the middle of
a struct. I guess I could come up with some reasons if you were
sending it over the wire, but it's contrived for an example. Due to
this implicit ambiguity and the fact that alignment is specified, I
feel like it might be nice to specify padding as optional as well.
It's a very tightly coupled concept, given the context.
> I assume that gcc is aware of this and arranges structs accordingly. Since
> gc doesn't target SPARC/PPC, it need not be burdened by the requirements of
> those architectures./
Ian just informed me that gccgo packs structs on SPARC on IRC, and I'm
just seeing Julian's email now about gc padding structs. Also ISTR a
few recent conversations on byte order (now that I think of it) where
it was specified to use []byte when packing is needed.
Anyhow, I think there's ambiguity in the spec because of the
implication of padding in the "Struct types" section. I'd like to
propose at least moving "_ float32 // padding" to the end of the
example struct in the spec.
I'll try to put something together that doesn't modify how the
language actually works today and see what people think.
--dho
Kyle Lemons
Devon H. O'Dell
Ok. I don't see how that's a useful demonstration if you don't know
what the rest of the memory around your struct will look like
(depending on what implementation of the compiler compiled it),
especially since you can't unpack []byte directly to a struct (since
apparently gc and gccgo pad them). It really only seems practically
useful in Go if you put padding at the end of a struct. Maybe I'm
short-sighted.
I'll drop the issue.
Thanks for the input and clarification on the topics!
--dho
Ian Lance Taylor
> Ian just informed me that gccgo packs structs on SPARC on IRC,
s/packs/pads/
Gccgo uses the C ABI to lay out any given struct, as described at
http://golang.org/doc/install/gccgo#Types . That is, gccgo adds padding
to structs in the same way that C does.
Incidentally, this fact is the reason that reflect.Type has both an
Align and a FieldAlign method: in some C ABIs, notably x86, the
alignment of a type when used as a struct field is not the same as the
alignment of the type when used as a standalone variable. With the gc
compiler, the Align and FieldAlign methods for a given Type always
return the same value.
Ian
Devon H. O'Dell
On Apr 4, 2012 8:08 PM, "Ian Lance Taylor" <ia...@google.com> wrote:
>
> "Devon H. O'Dell" <devon...@gmail.com> writes:
>
> > Ian just informed me that gccgo packs structs on SPARC on IRC,
>
> s/packs/pads/
Yes, thank you. Starting with the same two letters, I was doomed to do this at some point.
> Gccgo uses the C ABI to lay out any given struct, as described at
> http://golang.org/doc/install/gccgo#Types . That is, gccgo adds padding
> to structs in the same way that C does.
>
> Incidentally, this fact is the reason that reflect.Type has both an
> Align and a FieldAlign method: in some C ABIs, notably x86, the
> alignment of a type when used as a struct field is not the same as the
> alignment of the type when used as a standalone variable. With the gc
> compiler, the Align and FieldAlign methods for a given Type always
> return the same value.
Awesome, thanks for that info.
--dho
> Ian
LRN
If alignment and padding is platform dependent, how do you write and read structs to/from a socket?
For example, in C it is not uncommon to declare a struct, initialize its fields with values (converted to big-endian format), then write it into a socket in one call.
The receiver will, again, read the struct from a socket in one call, then will do big-endian-to-native format conversion on each field where it is applicable - and will get the same struct the sender had.
This only works when definitions of the struct on both the sender and the receiver are aligned in the same way (usually - 1-byte aligned).
I've encountered this problem recently, which is why i'm asking.
Also, MSVC uses a padding algorithm different from the one gcc has, which is why there's a -mms-bitfields option in gcc. It forces gcc to use MSVC-style padding, which is required if the caller wants to pass certain structs to W32 API.
How will that work in Go? (h-m-m-m...i can probably infer that from W32 port of Go by reading its source code...)
Aram Hăvărneanu
--
Aram Hăvărneanu
Rodrigo Kochenburger
Rémy Oudompheng
> Also, MSVC uses a padding algorithm different from the one gcc has, which is
> why there's a -mms-bitfields option in gcc. It forces gcc to use MSVC-style
> padding, which is required if the caller wants to pass certain structs to
> W32 API.
> How will that work in Go? (h-m-m-m...i can probably infer that from W32 port
> of Go by reading its source code...)
The most secure method would probably to write a dedicated function
what converts structs to array of bytes that emulates a given padding
algorithm.
Rémy.
minux
encoding/binary will use reflection and ignore any padding, right?
Kyle Lemons
Related question:
If alignment and padding is platform dependent, how do you write and read structs to/from a socket?
dem...@gmail.com
Ian Lance Taylor
> Is there any update to alignment/padding for structs now that we can write
> .so files that can be called by C programs? We would have to know it to tell
> the C compiler what to expect from go side imho.
your C code can #include. See
https://golang.org/cmd/cgo/#hdr-C_references_to_Go .
Ian
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Ali Demir
> email to golang-nuts+unsubscribe@googlegroups.com.
Ian Lance Taylor
> Header file won't have padding / alignment info. That would be a compiler
> setting on the C side that is using the header. So the programmer needs to
> setup C compiler in a way that is compatible with the go compiler. How would
> he know how to set it up? Do we just assume defaults are the same between Go
> and C?
fields are aligned as expected.
It's true that the cgo tool does assume that it knows the alignment
requirements of the basic types. These requirements are not normally
changed by compiler options. The alignment requirements of unusual
types, such as processor-specific vector types, are not relevant,
since Go can't represent them and they will therefore never appear in
the Go/C interface.
Do you have a specific concern, or is this purely abstract?
Ian
Ali Demir
Ian Lance Taylor
> Packing option is explicitly specified for VC++:
>
> https://msdn.microsoft.com/en-us/library/xh3e3fd0.aspx
>
> The user needs to make sure whatever they pick (or default) in their project
> matches what Go compiler used. So it may help to specify what go compiler is
> using.
the C compiler.
In general the Go compiler expects fields to be aligned to their size.
That is, a four byte field is aligned to a four byte boundary, and an
eight byte field is aligned to an eight byte boundary. As I said
earlier, the compiler emits padding fields so that everything will
work if the C compiler uses a smaller alignment. However, things will
fail if the C compiler uses a larger alignment. I'm not aware of any
such C compiler, but I guess it wouldn't shock me.
Ian