Re: [go-nuts] gob size limit

275 views
Skip to first unread message

Rob Pike

unread,
Aug 20, 2012, 7:19:40 PM8/20/12
to the...@gmail.com, golan...@googlegroups.com
On Mon, Aug 20, 2012 at 3:02 PM, <the...@gmail.com> wrote:
> Hi, I'm hitting the gob size upper limit in gob/decoder.go on line 80 in
> tip. It's now capped at a hardcoded 1GB. I'm using gobs here to store a
> large data structure to disk. Is there a plan to relax this restriction?
> (It's now marked as a TODO) On a 64-bit machine, it shouldn't be a problem
> as far as I can tell. Many thanks, Max

It's easy to do DOS, either deliberately or by accident, by allowing
huge messages to flow through. If you have legitimate need for huge
messages, DO the TO.

-rob

Rob Pike

unread,
Aug 20, 2012, 7:36:39 PM8/20/12
to the...@gmail.com, golan...@googlegroups.com
I'm beginning to think the TODO should BEDONE by adding the obvious
function or method, although I'm unsure whether it should be on the
package or the encoder/decoder, and so on. It's easy but subtle.

If we do that, the default could drop significantly. A gig is a stupid
big message and a small number of clients could bring down a server.

-rob

Kyle Lemons

unread,
Aug 20, 2012, 8:39:58 PM8/20/12
to Rob Pike, the...@gmail.com, golan...@googlegroups.com
My knee-jerk reaction is that it should be per-en/decoder, but that would require an odd return type for SetMaxSize (or whatever) to make the proverbial gob.NewDecoder(r).SetMaxSize(1024*1024*1024).Decode(&v) keep working.  Perhaps NewDecoderSize and such, more in line with bufio.Reader?

David Symonds

unread,
Aug 20, 2012, 8:55:52 PM8/20/12
to Kyle Lemons, Rob Pike, the...@gmail.com, golan...@googlegroups.com
A method on the decoder sounds right to me. You would only want to
raise these limits with input that you somewhat trust.

Zippoxer

unread,
Aug 21, 2012, 4:10:37 PM8/21/12
to golan...@googlegroups.com, the...@gmail.com
You mean reading a gig value consumes too much memory? Then why not read it in chunks?

d := gob.NewDecoder(conn)
var src io.Reader // will read the gig value directly from conn
err := d.Decode(&src)
...
dst, err := os.Create("file")
...
err = io.Copy(dst, src)
...

As an alternative to:

d := gob.NewDecoder(conn)
var src []byte // the complete gig value, in memory
err := d.Decode(src)
...
dst, err := os.Create("file")
...
err = io.Copy(dst, bytes.NewReader(src))
...

The only issue is that you have to finish reading from the reader in order to read further values. Unless you seek.

However, I couldn't see how this proposal could help enough to be worth the effort.
Reply all
Reply to author
Forward
0 new messages