ANN: JSON Stream

[Chris Dew]

I was quite concerned when I couldn't find a formal specification for the simple newline delimited JSON over TCP framing (which I have successfully used on multiple projects).

So I wrote one:

http://www.jsonstream.org

Have I missed a IETF or W3C RFC for this?

Comments very welcome...

Chris.

[Kevin Swiber]

On Tue, Jun 11, 2013 at 8:57 AM, Chris Dew <cms...@gmail.com> wrote:

I was quite concerned when I couldn't find a formal specification for the simple newline delimited JSON over TCP framing (which I have successfully used on multiple projects).

So I wrote one:

http://www.jsonstream.org

Have I missed a IETF or W3C RFC for this?

Hey Chris,

I think most streaming parsers just wait for the beginning object or array to close and don't necessarily need a delimiter between objects.

A lot of work in "streaming JSON over TCP" has been focused on binary serialization formats as a whole, not necessarily chunk delimiters.

- BSON: http://bsonspec.org/

- MessagePack: http://msgpack.org

- CBOR: http://tools.ietf.org/html/draft-bormann-cbor-01 (recent IETF Internet-Draft)

Cheers, 

--
Kevin Swiber

Projects: https://github.com/kevinswiber
Twitter: @kevinswiber

[Shane Holloway (IEEE)]

On Jun 11, 2013, at 9:52 AM, Kevin Swiber <ksw...@gmail.com> wrote:

On Tue, Jun 11, 2013 at 8:57 AM, Chris Dew <cms...@gmail.com> wrote:

I was quite concerned when I couldn't find a formal specification for the simple newline delimited JSON over TCP framing (which I have successfully used on multiple projects).

So I wrote one:

http://www.jsonstream.org

Have I missed a IETF or W3C RFC for this?

Hey Chris,

I think most streaming parsers just wait for the beginning object or array to close and don't necessarily need a delimiter between objects.

A lot of work in "streaming JSON over TCP" has been focused on binary serialization formats as a whole, not necessarily chunk delimiters.

I've taken a similar streaming approach to Chris, but using '\0' as a delimiter. It's easier to make work when you don't have a flexible/robust JSON parser because you can do chunking in a stateless way. When chunking using object/array open tokens, you either need to implement a stateful scanner to match nested opens/closes. Or a more robust JSON parser that is itself streaming, or a parser that can tell you where it stopped parsing the last packet. Oh, and not throw an error when it encounters multiple top-level items and partial JSON fragments.

It is much simpler (and faster) to split on characters that are not permitted in a JSON document. I chose '\0' because we were using some human-created plaintext JSON files that did have real '\n' whitespace.

[Kevin Swiber]

That makes sense.

I get that it's a lot simpler to implement, but I'm often in a position where I need a full-blown transfer protocol.

In HTTP, for instance, not only can I receive multipart content that defines boundaries separating each JSON object, but I can also get helpful metadata like Content-Length, Content-Encoding, etc.  (Granted, using this also ties one into a request/response model with the overhead of an HTTP parser.)

I'm curious what your use cases are for character-delimited JSON objects over-the-wire.  (I can speculate, but I find the real world is usually more interesting.)

Thanks,

[Shane Holloway (IEEE)]

That makes sense.

I get that it's a lot simpler to implement, but I'm often in a position where I need a full-blown transfer protocol.

In HTTP, for instance, not only can I receive multipart content that defines boundaries separating each JSON object, but I can also get helpful metadata like Content-Length, Content-Encoding, etc.  (Granted, using this also ties one into a request/response model with the overhead of an HTTP parser.)

I'm curious what your use cases are for character-delimited JSON objects over-the-wire.  (I can speculate, but I find the real world is usually more interesting.)

Certainly — real examples are far better!

I actually used '\0' delimited JSON streams to implement a dual-server variant of JSON-RPC2 over TCP sockets for [Bellite's Interprocess protocol](http://bellite.io/docs/api/#ipc). Creating small implementations was then straight forward for the host C++ server and the language bindings like [bellite-node](https://github.com/bellite/bellite-node), Python, Ruby & PHP. Using delimited streams made the work considerably easier since the built-in JSON parsing did not support JSON streaming for those languages.

[Dick Hardt]

For those interested in seeing potential new JSON standards, there is JSON-B, JSON-C and JSON-D

http://tools.ietf.org/html/draft-hallambaker-jsonbcd-00

That may be an appropriate WG for standardizing streaming of JSON objects

FWIW: In the past, I also used newlines to move mulitple JSON objects and allows the use of readline libraries to read in an object at a time. It worked pretty well

-- Dick

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Floby]

var split = requrie('split')

jsonStream.pipe(split());