[go-nuts] Alternative to encoding/binary?

[Daniel Smith]

Yesterday I discovered that, at least for my use, binary.Write/Read spends more time encoding than writing to the disk. It's making my program (which I estimate will eventually generate about 24 gb of data) quite slow.

IOW, if I replace

err = binary.Write(a._file, binary.LittleEndian, &myStruct)

with

var test = make([]byte, sizeOfMyStruct)
...
_, err = a._file.Write(test)

my program runs much faster.

Is there any way to write an arbitrary struct to disk more quickly? Ideally I'd like to be able to treat the struct as a []byte without doing any encoding or reflection at all. I realize this will break the type-safe nature of the language. Is there a way to do this with the unsafe package?

Thanks!

--
Daniel Smith
http://www.schaumburggoclub.org/

[roger peppe]

you can do this easily with the unsafe package if you really need to:

func struct2bytes(x *myStruct) []byte {
return (*[unsafe.Sizeof(*x)]byte)(unsafe.Pointer(x))[0:]
}

the struct pointer first becomes an arbitrary pointer, then a pointer
to a fixed-size array of the correct size, then a slice.
the conversions incur no runtime overhead.

[Daniel Smith]

Thank you, that's exactly what I was looking for.

I'm surprised that encoding/binary is so slow.

On Mon, Jul 12, 2010 at 6:55 AM, roger peppe <rogp...@gmail.com> wrote:

you can do this easily with the unsafe package if you really need to:

func struct2bytes(x *myStruct) []byte {
       return (*[unsafe.Sizeof(*x)]byte)(unsafe.Pointer(x))[0:]
}

the struct pointer first becomes an arbitrary pointer, then a pointer
to a fixed-size array of the correct size, then a slice.
the conversions incur no runtime overhead.

[Andrew Gerrand]

On 13 July 2010 01:46, Daniel Smith <dan...@lukenine45.net> wrote:
> Thank you, that's exactly what I was looking for.
> I'm surprised that encoding/binary is so slow.

If you look at the implementation, it uses a lot of instructions to do
bit-shifting and such. It makes sense that it would be an order of
magnitude slower than just copying the data out of memory.

Andrew

[Russ Cox]

> If you look at the implementation, it uses a lot of instructions to do
> bit-shifting and such. It makes sense that it would be an order of
> magnitude slower than just copying the data out of memory.

The slowness comes from using reflect, which
allocates a lot of garbage as it walks the structures.
The bit shifting is essentially free and not the bottleneck.
Encoding/binary is for convenience, not for speed.

Russ

[David Roundy]

Is this something that either improvement of the reflect package or a
future better-optimizing compiler could improve upon?

David

On Mon, Jul 12, 2010 at 6:52 PM, Russ Cox <r...@golang.org> wrote:
> The slowness comes from using reflect, which
> allocates a lot of garbage as it walks the structures.
> The bit shifting is essentially free and not the bottleneck.
> Encoding/binary is for convenience, not for speed.
>
> Russ
>

--
David Roundy

[Russ Cox]

On Mon, Jul 12, 2010 at 19:11, David Roundy
<rou...@physics.oregonstate.edu> wrote:
> Is this something that either improvement of the reflect package or a
> future better-optimizing compiler could improve upon?

Yes. But I don't know exactly what that would mean.
The current reflect package is version 2 or version 3
depending on how you count, and we've used it enough
to know that there needs to be another try at some point,
and that the new version would ideally allocate less
and be easier to use. But we don't really know what it
would look like yet and are mostly letting the idea
percolate while we do other things.

Russ