JSON decode streaming : Go forward when format is not json

[Thomas Solignac]

Hello,

I build a service receiving Json from websocket.

I use json decode.
I want my service to be robust !

var tmp interface{}

err = this.reader.Decode(&tmp)

My problem is :
When I got some error here, I came to infinite loop, because the decoder doesn't go forward.
It seems that it is trying to decode the same part of texte again and again.

I would like to send my error and go on.

How could I fix that ?
Till then I just (sadly) close the connection.

Thanks a lot for your help,
Thomas

[Jakob Borg]

You're decoding JSON objects from a stream. Knowing where the next object starts requires knowing where the current one ends. If the current JSON object can't be decoded, the decoder doesn't know where it ends, and it can't proceed to the next one.

Closing the connection is probably appropriate when the other side sends something that cannot be decoded.

//jb

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Thomas Solignac]

Thank you for answer.

I think that this behaviour is approriate too, considering this.
However, something remains strange.

How could the streamer know the difference between a partial stream and a not JSON string ?

Because here, the message sent is just "TEST".
And the error is : "invalid character 'T' looking for beginning of value"
With such error, it should go forward anyway.

Unless the decoder just "give up".

And I don't get this error elsewhere.

It is not essential but it puzzles me.

Note : Never tried to sent something like : "{test:"

[Tamás Gulácsi]

2017. január 19., csütörtök 18:55:22 UTC+1 időpontban Thomas Solignac a következőt írta:

Thank you for answer.

I think that this behaviour is approriate too, considering this.
However, something remains strange.

How could the streamer know the difference between a partial stream and a not JSON string ?

Because here, the message sent is just "TEST".
And the error is : "invalid character 'T' looking for beginning of value"
With such error, it should go forward anyway.

Unless the decoder just "give up".

And I don't get this error elsewhere.

It is not essential but it puzzles me.

Note : Never tried to sent something like : "{test:"

Le jeudi 19 janvier 2017 18:43:11 UTC+1, Jakob Borg a écrit :

You're decoding JSON objects from a stream. Knowing where the next object starts requires knowing where the current one ends. If the current JSON object can't be decoded, the decoder doesn't know where it ends, and it can't proceed to the next one.

Closing the connection is probably appropriate when the other side sends something that cannot be decoded.

//jb
 

A JSON stream is a (usually) '\n' separated string of JSON objects, which starts with either '[' or '{'.
So you can search for such chars, start the decoder there (watch out for its own buffering! Use r = io.MultiReader(dec.Buffered(), r) to recreate the rest.

[nost...@gmail.com]

A partial stream must still be a valid stream. Once the stream is in an invalid state, the decoder doesn't know how to proceed. Since none of the value types that can be represented by JSON can begin with the letter "T", the stream can be considered invalid soon as the decoder encounters that character.

There are some methods for JSON communication that define a way to have separate chunks within a stream. A common one is a stream that has individual JSON values separated by new lines. If you were implementing such a stream, a valid response might be to continue reading until you reach the next new line, and then attempt to start a new decode from there.

[Thomas Solignac]

Thank you both for your answers.

It's true that there is a convention for JSON streaming in which one each json message is followed by a '\n'.

There is + and - for this convention.

+ :
-> I can detect bad JSON

- :
-> The most important : It's a little more painful for final user. It's already rare to have a WS connection rather than HTTP, I'm not confortable with the idea tu push forward the first impression of weirdness.
-> Secondly, I can't use standard decoder package anymore for this. I need an other package, or do it my self. But the parsing is not trivial, as I can get '\n' or Json reserved character inside a string. Robustness brings complexity here.


However, such different answers let me get an accurate synthesis.
This allows me to make a decision :

I'm going to stay on the actual convention, with the standard decoder.

I added a new error message, indicating that the JSON message is not valid.
And I'm trying now to get that bad formatted message for more explicit error. (Maybe read directly the socket ? Let's set)


Thank you for this enlightening discussion !

[Jordan Krage]

Robustness brings complexity here.

Relevant: RFC7464 - JavaScript Object Notation (JSON) Text Sequences

Given that format, it should be possible to parse record boundaries reliably, and then use the std lib to unmarshal them individually.

[Thomas Solignac]

The Json Golang doc indicates that "Package json implements encoding and decoding of JSON as defined in RFC 4627". 

It's a deprecated RFC, and, contrary to the most recent you spoke about, It says nothing about truncated/invalid json.
It explains why the package doesn't provide such feature I guess !

Maybe, should I report as feature proposal to update the package for most recent RFC ?

Thank you