JSON data over socket being chunked!
2,286 views
Skip to first unread message
Rambo
Nov 21, 2010, 10:13:22 AM11/21/10
to nodejs
Hi everyone. I'll appreciate some help with this!
I built a mobile app which connects to a socket server in nodeJS and
sends json-encoded data. The problem is that I don't know how to deal
with buffers, because data is being sent in real time and sometimes
the JSON structure is chunked, so the server cannot parse it
correctly. How can I be sure that the JSON data the server receives is
well formed? Which is the limit for sending data over socket without
being corrupted? I mean... when is the 'data' event dispatched?
Thanks!
I built a mobile app which connects to a socket server in nodeJS and
sends json-encoded data. The problem is that I don't know how to deal
with buffers, because data is being sent in real time and sometimes
the JSON structure is chunked, so the server cannot parse it
correctly. How can I be sure that the JSON data the server receives is
well formed? Which is the limit for sending data over socket without
being corrupted? I mean... when is the 'data' event dispatched?
Thanks!
Guillermo Rauch
Nov 21, 2010, 10:32:55 AM11/21/10
to nod...@googlegroups.com
You need to use some encoding mechanism, since the `data` event of your stream can be emitted in a non-deterministic way.
For example, https://github.com/pgriess/node-netstring
Mihai Călin Bazon
Nov 21, 2010, 10:33:50 AM11/21/10
to nod...@googlegroups.com
You have a few variants:
1. You should design your protocol to account for the possibility of receiving the data in multiple chunks. For example:
- end each request with a byte that you know that cannot appear in JSON. For example a null byte. Then you know that you should just accumulate data into a buffer until a null byte comes in. When you get the null, discard it, convert the buffer to an UTF8 string and go JSON.parse.
- prepend each request with, say, "N:", where N is the number of bytes that follows. Then you know how many bytes to expect; accumulate them into a buffer and when you got to N, convert to UTF8 string and go JSON.parse.
2. You could also write a resumable JSON parser that parses data as it comes (and stops when there's no more data, but resumes on the next chunk) and calls an event handler when a full "thing" has been received and parsed (i.e. an array, a hash, etc.). That's a lot of work but quite rewarding if you never did it before. ;-)
3. But maybe it would be easier for you to change your code to use the Websocket protocol, and you can find at least a dozen libs that do the low level dirty for you. ;-)
Cheers,
-Mihai
--
Mihai Bazon,
http://mihai.bazon.net/blog
1. You should design your protocol to account for the possibility of receiving the data in multiple chunks. For example:
- end each request with a byte that you know that cannot appear in JSON. For example a null byte. Then you know that you should just accumulate data into a buffer until a null byte comes in. When you get the null, discard it, convert the buffer to an UTF8 string and go JSON.parse.
- prepend each request with, say, "N:", where N is the number of bytes that follows. Then you know how many bytes to expect; accumulate them into a buffer and when you got to N, convert to UTF8 string and go JSON.parse.
2. You could also write a resumable JSON parser that parses data as it comes (and stops when there's no more data, but resumes on the next chunk) and calls an event handler when a full "thing" has been received and parsed (i.e. an array, a hash, etc.). That's a lot of work but quite rewarding if you never did it before. ;-)
3. But maybe it would be easier for you to change your code to use the Websocket protocol, and you can find at least a dozen libs that do the low level dirty for you. ;-)
Cheers,
-Mihai
--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
--
Mihai Bazon,
http://mihai.bazon.net/blog
Rambo
Nov 21, 2010, 11:38:18 AM11/21/10
to nodejs
I'm actually using socket.io to send data back to the clients but the
source comes from another socket. I know it's not the best solution,
but I'm not quite sure if socket.io actually implements its own
protocol or is the same as any websocket implementation out there, for
example this one: http://jwebsocket.org/
So what I need to know is: can I use any web socket client to connect
to a socket.io server? If that is possible I could simply use a single
websocket server instead of 2 listening in 2 different ports! and
obviously that would solve my problem of data being chunked.
> > nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>
> > .
source comes from another socket. I know it's not the best solution,
but I'm not quite sure if socket.io actually implements its own
protocol or is the same as any websocket implementation out there, for
example this one: http://jwebsocket.org/
So what I need to know is: can I use any web socket client to connect
to a socket.io server? If that is possible I could simply use a single
websocket server instead of 2 listening in 2 different ports! and
obviously that would solve my problem of data being chunked.
> > .
Mikeal Rogers
Nov 21, 2010, 12:53:34 PM11/21/10
to nod...@googlegroups.com
\n can't appear unescaped in JSON so you can use it as a delimiter, it's what we do for the CouchDB continuous _changes feed.
Parsing becomes a simple matter of buffering the chunks until the next newline.
-Mikeal
--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
Konstantin Käfer
Nov 21, 2010, 2:10:16 PM11/21/10
to nod...@googlegroups.com
According to http://www.ietf.org/rfc/rfc4627.txt, whitespace including
newline characters is allowed before and after any token. That means
that you can't use \n to find complete JSON statements. Unescaped \n
is of course illegal within a string.
newline characters is allowed before and after any token. That means
that you can't use \n to find complete JSON statements. Unescaped \n
is of course illegal within a string.
Konstantin Käfer
Nov 21, 2010, 2:15:13 PM11/21/10
to nod...@googlegroups.com
Is there a sequential access parser (like SAX for XML) for JSON data?
2010/11/21 Konstantin Käfer <kka...@gmail.com>:
Rambo
Nov 21, 2010, 4:30:06 PM11/21/10
to nodejs
I need to implement a websocket on Android, so I tried the demo (using
emulator) on http://jwebsocket.org/ but i couldn't connect to my
socket.io server... what's wrong?? is it me or the documentation on
that site really sucks?? After reading all that shit I still don't
know how to implement a websocket client...
On 21 nov, 16:15, Konstantin Käfer <kkae...@gmail.com> wrote:
> Is there a sequential access parser (like SAX for XML) for JSON data?
>
> 2010/11/21 Konstantin Käfer <kkae...@gmail.com>:
>
>
>
>
>
>
>
> > According tohttp://www.ietf.org/rfc/rfc4627.txt, whitespace including
emulator) on http://jwebsocket.org/ but i couldn't connect to my
socket.io server... what's wrong?? is it me or the documentation on
that site really sucks?? After reading all that shit I still don't
know how to implement a websocket client...
On 21 nov, 16:15, Konstantin Käfer <kkae...@gmail.com> wrote:
> Is there a sequential access parser (like SAX for XML) for JSON data?
>
>
>
>
>
>
>
>
> > According tohttp://www.ietf.org/rfc/rfc4627.txt, whitespace including
> > newline characters is allowed before and after any token. That means
> > that you can't use \n to find complete JSON statements. Unescaped \n
> > is of course illegal within a string.
>
> > that you can't use \n to find complete JSON statements. Unescaped \n
> > is of course illegal within a string.
>
> > On Sun, Nov 21, 2010 at 19:53, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
> >> \n can't appear unescaped in JSON so you can use it as a delimiter, it's
> >> what we do for the CouchDB continuous _changes feed.
> >> Parsing becomes a simple matter of buffering the chunks until the next
> >> newline.
> >> -Mikeal
>
> >> On Sun, Nov 21, 2010 at 8:47 AM, Guillermo Rauch <rau...@gmail.com> wrote:
>
> >>> Yep, you can use a WebSocket client and parse the socket.io protocol.
> >>> Look at the websocket tests for an example:
>
> >>>https://github.com/LearnBoost/Socket.IO-node/blob/0.6.2/tests/transpo...
> >> \n can't appear unescaped in JSON so you can use it as a delimiter, it's
> >> what we do for the CouchDB continuous _changes feed.
> >> Parsing becomes a simple matter of buffering the chunks until the next
> >> newline.
> >> -Mikeal
>
> >> On Sun, Nov 21, 2010 at 8:47 AM, Guillermo Rauch <rau...@gmail.com> wrote:
>
> >>> Yep, you can use a WebSocket client and parse the socket.io protocol.
> >>> Look at the websocket tests for an example:
>
Nathan Rajlich
Nov 21, 2010, 7:27:56 PM11/21/10
to nod...@googlegroups.com
Try my Java-WebSocket. I know that people have used it to talk to Socket.IO servers before...
Justin Cormack
Nov 22, 2010, 4:42:03 AM11/22/10
to nod...@googlegroups.com
On my todo list... I havent found one yet.
Justin
Robert Kieffer
Nov 22, 2010, 9:05:18 AM11/22/10
to nodejs
Yup, I do this as well for the streaming HTTP response data I send
back in the Proccoli activity feed. The code for detecting/processing
newline-separated JSON blocks is pretty simple:
https://github.com/broofa/proccoli/blob/master/webroot/lib/Dechunker.js
On Nov 21, 9:53 am, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
> \n can't appear unescaped in JSON so you can use it as a delimiter, it's
> what we do for the CouchDB continuous _changes feed.
>
> Parsing becomes a simple matter of buffering the chunks until the next
> newline.
>
> -Mikeal
>
>
>
>
>
>
>
> On Sun, Nov 21, 2010 at 8:47 AM, Guillermo Rauch <rau...@gmail.com> wrote:
> > Yep, you can use a WebSocket client and parse the socket.io protocol.
> > Look at the websocket tests for an example:
>
> >https://github.com/LearnBoost/Socket.IO-node/blob/0.6.2/tests/transpo...
> > .
back in the Proccoli activity feed. The code for detecting/processing
newline-separated JSON blocks is pretty simple:
https://github.com/broofa/proccoli/blob/master/webroot/lib/Dechunker.js
On Nov 21, 9:53 am, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
> \n can't appear unescaped in JSON so you can use it as a delimiter, it's
> what we do for the CouchDB continuous _changes feed.
>
> Parsing becomes a simple matter of buffering the chunks until the next
> newline.
>
> -Mikeal
>
>
>
>
>
>
>
> On Sun, Nov 21, 2010 at 8:47 AM, Guillermo Rauch <rau...@gmail.com> wrote:
> > Yep, you can use a WebSocket client and parse the socket.io protocol.
> > Look at the websocket tests for an example:
>
>
> > I connect with pgriess' node-websocket-client to the socket.io server.
>
> > --
> > Guillermo Rauch
> >http://devthought.com
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "nodejs" group.
> > To post to this group, send email to nod...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>
> > I connect with pgriess' node-websocket-client to the socket.io server.
>
> > --
> > Guillermo Rauch
> >http://devthought.com
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "nodejs" group.
> > To post to this group, send email to nod...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > .
Marco Rogers
Nov 22, 2010, 12:16:14 PM11/22/10
to nodejs
Check out the stream parser in my evented-twitter lib. It takes in
arbitrary chunks and emits valid json strings. Underneath it uses a
buffered tokenizer that handles the icky bits for you. It's pretty
fast.
https://github.com/polotek/evented-twitter/blob/master/lib/json-stream-parser.js
https://github.com/polotek/evented-twitter/blob/master/lib/buffered-tokenizer.js
Note that you need to set a start function with onReady and then call
jsonparser.start().
:Marco
arbitrary chunks and emits valid json strings. Underneath it uses a
buffered tokenizer that handles the icky bits for you. It's pretty
fast.
https://github.com/polotek/evented-twitter/blob/master/lib/json-stream-parser.js
https://github.com/polotek/evented-twitter/blob/master/lib/buffered-tokenizer.js
Note that you need to set a start function with onReady and then call
jsonparser.start().
:Marco
Sean McQuillan
Nov 30, 2010, 4:43:57 AM11/30/10
to nod...@googlegroups.com
Hi,
All json values except number literals are wrapped in matched-pair characters. If you avoid using number literals in your stream you can trivially find the end of a json object by scanning for a matched paren, square bracket, or quote.
That said, your probably better off not building your own protocol and listening to the other advice in this thread :).
Cheers,
Sean
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages