Reading and writing binary structures?

[Stéphane Bortzmeyer]

I was trying to write a network server for a binary protocol. What is
the "effective" and idiomatic to read and write structures from a
binary packet? I find only the binary package
<http://golang.org/pkg/encoding/binary/> which does not seem to work
with structs. I assume I have to create a struct with proper sizes and
padding and then? Using package unsafe? Any examples somewhere?

[Qtvali]

On Dec 15, 10:48 pm, Stéphane Bortzmeyer <bortzme...@gmail.com> wrote:
> padding and then? Using package unsafe? Any examples somewhere?

func (p *Word) at(i int) *Word {
return (*Word)(unsafe.Pointer(uintptr(unsafe.Pointer(p)) + uintptr(i)
*_S))
}

This was in pkg/big/arith.go; there are also assembler bindings.

[ygl]

please have a look of "gob": http://golang.org/pkg/gob/
which is used by "rpc" package.

Regards
Yigong

[Stéphane Bortzmeyer]

2009/12/16 ygl <ygl...@gmail.com>:

> please have a look of "gob": http://golang.org/pkg/gob/

The way I read it, gob only interacts with itself. I want a way to
talk a protocol which I did not design and do not control, whose
binary data structures are not self-described as in gob.

[Max Ushakov]

AFAIK "binary" package actually does work with structs. The struct
only needs to have only fixed-sized fields (which is natural since you
have to know all sizes in advance). If you know the binary layout of
the protocol you can mirror that in a struct and use the binary
package.

[ygl]

On Dec 16, 12:37 pm, Stéphane Bortzmeyer <bortzme...@gmail.com> wrote:
> 2009/12/16 ygl <yglg...@gmail.com>:

As far as i know (very possible i miss something:), you can follow the
model of gob:
1. use reflect to recursively iterate thru your structs's member
fields for its type info and data
or you can avoid reflect by adding some kind of Marshal/Demarshl
methods to each of you msg structs.
2. according to your target protocol wire-format, serialize struct's
data members into a bytes.Buffer
3. send the byte[] stream from bytes.Buffer to io interface

At the receving side, we can reverse the above steps. I guess that in
your target proto frame, very possible it has <msg_id, msg_len,
byte_data> as content, so we can use msg_id to decide which struct to
demarshal to.

In my understanding, gob uses reflect to make things flexible. You
dont need to define marshal/demarshal for each possible msg struct and
at recving side, allow some fuzziness of mapping data to destination
structs.

Regards
yigong

[Stéphane Bortzmeyer]

OK, thanks to everyone for your help. Today, what I am doing (and
which works) is:

import ("bytes"; "encoding/binary";)

...

type ProtocolHeader struct {
Id uint16;
Misc uint16;
Foo, Bar uint16;
}

...

var (
packet ProtocolHeader;
)
message := make([]byte, 1024);
n, remaddr, error := listener.ReadFrom(data);
buf := bytes.NewBuffer(data[0:n]);
binary.Read(buf, binary.BigEndian, &packet);
fmt.Printf("ID is %d\n", packet.Id);

and it works, the struct of type ProtocolHeader is correctly filled
in.

Is there a way in Go to have fields which are not multiple of a byte,
for instance 1-bit or 4bits and to use encoding/binary with them? It
does not seem so, I believe I'll have to read in a uint16 then play
with >> and && to extract my fields.