Discussion of the Go Programming Language ()

themax | 21 Aug 2012 00:02

gob size limit

Hi, I'm hitting the gob size upper limit in gob/decoder.go on line 80 in tip. It's now capped at a hardcoded 1GB. I'm using gobs here to store a large data structure to disk. Is there a plan to relax this restriction? (It's now marked as a TODO) On a 64-bit machine, it shouldn't be a problem as far as I can tell. Many thanks, Max

headers

Rob Pike | 21 Aug 2012 01:19

Re: gob size limit

On Mon, Aug 20, 2012 at 3:02 PM,  <themax@...> wrote:
> Hi, I'm hitting the gob size upper limit in gob/decoder.go on line 80 in
> tip.  It's now capped at a hardcoded 1GB.  I'm using gobs here to store a
> large data structure to disk.  Is there a plan to relax this restriction?
> (It's now marked as a TODO)  On a 64-bit machine, it shouldn't be a problem
> as far as I can tell.  Many thanks, Max

It's easy to do DOS, either deliberately or by accident, by allowing
huge messages to flow through. If you have legitimate need for huge
messages, DO the TO.

-rob

headers

Rob Pike | 21 Aug 2012 01:36

Re: gob size limit

I'm beginning to think the TODO should BEDONE by adding the obvious
function or method, although I'm unsure whether it should be on the
package or the encoder/decoder, and so on. It's easy but subtle.

If we do that, the default could drop significantly. A gig is a stupid
big message and a small number of clients could bring down a server.

-rob

headers

Kyle Lemons | 21 Aug 2012 02:39

Re: gob size limit

My knee-jerk reaction is that it should be per-en/decoder, but that would require an odd return type for SetMaxSize (or whatever) to make the proverbial gob.NewDecoder(r).SetMaxSize(1024*1024*1024).Decode(&v) keep working. Perhaps NewDecoderSize and such, more in line with bufio.Reader?

On Mon, Aug 20, 2012 at 4:36 PM, Rob Pike <r <at> golang.org> wrote:

I'm beginning to think the TODO should BEDONE by adding the obvious
function or method, although I'm unsure whether it should be on the
package or the encoder/decoder, and so on. It's easy but subtle.

If we do that, the default could drop significantly. A gig is a stupid
big message and a small number of clients could bring down a server.

-rob

headers

David Symonds | 21 Aug 2012 02:55

Re: gob size limit

A method on the decoder sounds right to me. You would only want to
raise these limits with input that you somewhat trust.

headers

themax | 21 Aug 2012 20:48

Re: gob size limit

Thanks for the replies, the proposed solution would definitely work for my (rather simple) use case.

On Monday, August 20, 2012 8:55:52 PM UTC-4, David Symonds wrote:

A method on the decoder sounds right to me. You would only want to
raise these limits with input that you somewhat trust.

headers

Zippoxer | 21 Aug 2012 22:10

Re: gob size limit

You mean reading a gig value consumes too much memory? Then why not read it in chunks?

d := gob.NewDecoder(conn)

var src io.Reader // will read the gig value directly from conn

err := d.Decode(&src)

...

dst, err := os.Create("file")

...

err = io.Copy(dst, src)

...

As an alternative to:

d := gob.NewDecoder(conn)

var src []byte // the complete gig value, in memory

err := d.Decode(src)

...

dst, err := os.Create("file")

...

err = io.Copy(dst, bytes.NewReader(src))

...

The only issue is that you have to finish reading from the reader in order to read further values. Unless you seek.

However, I couldn't see how this proposal could help enough to be worth the effort.

On Tuesday, August 21, 2012 2:36:39 AM UTC+3, Rob Pike wrote:

I'm beginning to think the TODO should BEDONE by adding the obvious
function or method, although I'm unsure whether it should be on the
package or the encoder/decoder, and so on. It's easy but subtle.

If we do that, the default could drop significantly. A gig is a stupid
big message and a small number of clients could bring down a server.

-rob