Re: [go-nuts] gob size limit

[Rob Pike]

On Mon, Aug 20, 2012 at 3:02 PM, <the...@gmail.com> wrote:
> Hi, I'm hitting the gob size upper limit in gob/decoder.go on line 80 in
> tip. It's now capped at a hardcoded 1GB. I'm using gobs here to store a
> large data structure to disk. Is there a plan to relax this restriction?
> (It's now marked as a TODO) On a 64-bit machine, it shouldn't be a problem
> as far as I can tell. Many thanks, Max

It's easy to do DOS, either deliberately or by accident, by allowing
huge messages to flow through. If you have legitimate need for huge
messages, DO the TO.

-rob

[Rob Pike]

I'm beginning to think the TODO should BEDONE by adding the obvious
function or method, although I'm unsure whether it should be on the
package or the encoder/decoder, and so on. It's easy but subtle.

If we do that, the default could drop significantly. A gig is a stupid
big message and a small number of clients could bring down a server.

-rob

[Kyle Lemons]

My knee-jerk reaction is that it should be per-en/decoder, but that would require an odd return type for SetMaxSize (or whatever) to make the proverbial gob.NewDecoder(r).SetMaxSize(1024*1024*1024).Decode(&v) keep working.  Perhaps NewDecoderSize and such, more in line with bufio.Reader?

[David Symonds]

A method on the decoder sounds right to me. You would only want to
raise these limits with input that you somewhat trust.

[Zippoxer]

You mean reading a gig value consumes too much memory? Then why not read it in chunks?

d := gob.NewDecoder(conn)

var src io.Reader // will read the gig value directly from conn

err := d.Decode(&src)

...

dst, err := os.Create("file")

...

err = io.Copy(dst, src)

...

As an alternative to:

d := gob.NewDecoder(conn)

var src []byte // the complete gig value, in memory

err := d.Decode(src)

...

dst, err := os.Create("file")

...

err = io.Copy(dst, bytes.NewReader(src))

...

The only issue is that you have to finish reading from the reader in order to read further values. Unless you seek.

However, I couldn't see how this proposal could help enough to be worth the effort.