How to handle timeout with json.Decode?

[Parker Evans]

I am running into an issue with using json.Decode on a net.Conn that I was wondering if someone had a solution for.

I am trying to use json.Decode to decode json objects from a net.Conn stream (specifically a unix socket).  However, in the spirit of being able to cancel the goroutine that is doing the decoding, I would like it not to block forever.  Thus, I am setting the read timeout of the net.Conn before each read.  However, I have run into an issue.  If json.Decode gets any non io.EOF error (or in some cases even io.EOF), the decoder will never work again (always returns that error when calling the Decode method and no way to clear it).  I tried to make an io.Reader that wrapped the net.Conn and looked for the timeout error, and instead returned a nil error with zero bytes and no error, but Decode will spin forever if the io.Reader it is reading from returns 0 bytes read but no error.

So I think I am stuck...  Unless there is another way to handle this (other than creating a new decoder after every timeout which seems like overkill), I will probably implement my own decoder.

Here is an example of what I mean: http://play.golang.org/p/lHI6cb8F3y

NOTE: The example in the link above won't run but it shows the gist of what I am trying to do.

Any help would be greatly appreciated, but I am ok if someone says I just have to implement my own decoder (or modify the standard one).

Thanks!
Parker

[aro...@gmail.com]

NewDecoder is cheap: http://golang.org/src/pkg/encoding/json/stream.go?s=540:577#L16

Seems fine to create a new one every time there is an error.

- Augusto

[Parker Evans]

Thanks for the reply.

While I agree that creating a new decoder is relatively cheap (at least initially), if I were to create a new decoder after every timeout, that could create a lot of garbage.  For instance, once you start decoding, the decoder is going to allocate a large slice buffer to put the data into while trying to unmarshal it (something like 512 bytes).  Each time you throw away the decoder, you will abandon that buffer and the other members of a decoder.  This will happen every second while you are sitting and waiting for data on the net.Conn (assuming a 1s timeout as in my example).  Seems like a lot of unnecessary garbage...

An additional issue with creating a new decoder every time is that the decoder could have buffered part of a json string.  If you throw away the decoder after a timeout, you will lose that whole json message.  That could be a big problem in my application.  In an ideal world (for me), the decode would return some kind of error if it tried to read from the provided io.Reader and got 0 bytes that I could check to see if its a timeout (like the net.Error I used in my example)... or would provide a timeout facility of its own.

Is having a periodic timeout not a common use case for a json Decoder?  I supposed I could just implement similar functionality with a buffered reader or something and unmarshal and some other trickery (though I hate to reinvent the wheel).  Was just wondering if this had already been tackled.

~Parker

[Dave Cheney]

I had a look to see why json.NewDecoder was allocating on the heap and reduced the example to 

https://code.google.com/p/go/issues/detail?id=7213

Maybe something can be done to make NewDecoder stack allocated, which essentially makes it free.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Parker Evans]

I have to admit I have not taken the time to see how much garbage is created if I keep creating new json Decoders once per second and calling Decode on them (with a read timeout on the stream).  I am making assumptions based on the implementation that I read through (don't know if the compiler will do something help things there).  Nor do I know the actual runtime penalty of doing this.

My main concern is with losing json packets.  If I throw away the decoder after every read timeout, we could be in a situation where we are timing out mid packet in which case making a new decoder would throw away the existing buffered data and effectively drop a json packet.  I can't know how much time I need because the timeout is setup before every read of the io.Reader (net.Conn).

~Parker

[aro...@gmail.com]

My intuition is that the GC overhead is vastly outweighed by the network I/O.  A small piece of garbage a few times a second is probably unnoticeable and not worth the optimization effort.

Losing data is a good point, though.  I believe there are two possible error conditions:

(1) an actual JSON decoding error, such as a syntax error.   In this case, you may have several JSON messages buffered in the now-defunct decoder.  You can use Decoder.Buffered to recover the buffered data.  Not sure how to recover from that point -- you'll have to search for the end of the message, but it's already determined to be malformed so hard to be confident about that.  Might be better to try to read distinct data packets from the net.Conn and then send them on for decoding.

(2) a timeout.  In this case, you know there's nothing left in the buffer, so starting fresh is not a problem.  You could still use Buffered and use a MultiReader to ensure than anything left is fed into the new decoder.

- Augusto