handling streams in native c++ addons

[rhasson]

I'm working on a C++ NLP binding and wanted to implemented a streaming interface so that text could be added to a stream in JS, my native binding will receive it, extract the text process it and emit a "data"events with the responses.

How would I go about reading a stream that was either passed in as an argument or was piped in via "pipe"?  I'm assuming I'll have to implement the pipe interface as well which I'm not sure how to do yet

Thanks,

Roy

[Isaac Schlueter]

I'd recommend not doing the "stream" bit in C++, and instead have a
much simpler binding layer that does a single action, and then use
JavaScript to wrap up the queue of writes and emitted data blobs.

Check out the zlib binding in src/node_zlib.cc and lib/zlib.js in
node's source. It does something similar to this.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

[Nathan Rajlich]

I second Isaac's recommendation. That is exactly how the Encoder class in node-lame is written. Essentially the C bindings are 1-1 mappings to the C functions themselves. Then creating the node "Stream" API around those is done entirely in JS. Works great!

[rhasson]

Isaac and Nathan, 

Thanks for the feedback, I'll follow your guidance by keeping the binding a simple 1-1 mapping and doing the streaming interface in JS.

Roy

[Duncan Wong]

Sorry for digging up an old post

I'm thinking about writing a C++ streaming json parser addon.  The implementations I've found so far are done in javascript and are perhaps much slower than buffering the full response and doing JSON.parse (a couple of streaming json authors say that the speed is 2 to 3 times slower.

I'm thinking about feeding both the hooks and the stream itself in the addon so that it can emit only the events that I'm interested in.  The alternative would be to pass in each chunk to the C side, but from what I understand, constant intercommunication between the C binding and javascript is "slow".

My use case is to create an http proxy that aggregates from multiple restful json sources.  For certain service calls, it'd have to read and aggregate some of the calls.  I wish do as much streaming as I can to reduce the amount of overhead this proxy would induce as an additional http layer.

Clearly more complexity involved, but do you think the speed tradeoff is worth it especially in regards to http performance anyways?

Thanks!