Questions/suggestions for nodejs stream object mode

[Chaoran Yang]

Hi all,

I'm trying to implement a stream in object mode. It's a very complex stream that allows user read and write database records. But for this discussion, you can just imagine it as a duplex stream that reads and writes database records in a table.

[Question] I've read the API doc on streams. It's a bit unclear about the usage of highWaterMark in object mode. Does it limit the number of objects that it internally buffered? If not, how do I control the memory usage of the internal buffer?

[Suggestion] The API doc said "a readable stream in object mode will always return a single item from a call to readable.read(size), regardless of what the size argument is", which feels very unintuitive for me. I think it should be more intuitive if readable.read(size) returns "size" items.

[Suggestion] Currently, The special value null still retains its special value for object mode streams. That is, for object mode readable streams, null as a return value from stream.read() indicates that there is no more data, and stream.push(null) will signal the end of stream data (EOF). However, in the stream I want to implement, a null object is exactly what I want to return to the consumer, and it should not indicate that there is no more data. Wouldn't it be more appropriate to use "undefined" as the special value to indicate the end of stream data?

Thanks,

Chaoran Yang

[greelgorke]

Unfortunatelly the objectMode doesn'T mean "any kind of object", so you can't use it to stream anything than Buffer or String instances. This mode in fact a "i won't concat the chunks to one big, but let them as you push them".  This mode is good for some cases e.g. when you parse something and want return well delimited tokens, as lines or xml-tags. But still it handles just binary or string data. So i assume the watermarks are working just the same as before. If you want to us real object streams, look into https://github.com/dominictarr/event-stream

[Peter Rust]

I'm no expert on streams, but when I was at nodeconf I did ask substack whether I could stream objects with node streams and IIRC his response was that that was what objectMode was for. It didn't sound at all like objectMode was just for strings/binary data.

[Peter Rust]

The docs (http://nodejs.org/api/stream.html#stream_object_mode) seem pretty clear: "Normally, Streams operate on Strings and Buffers exclusively. Streams that are in object mode can emit generic JavaScript values other than Buffers and Strings."

[greelgorke]

last time i tested it was buggy. but it seems like it's working now as expected. great!

[Floby]

Yes it's working now.

[Benjamin Goering]

I agree with both suggestions.

Another hole is the non-objectMode special behavior of readable.push(''); http://nodejs.org/docs/latest/api/stream.html#stream_stream_push

When called by a Readable implementation, it is supposed to signal that the stream is done getting data from its upstream source, but there was no data to fetch. The Readable internals then ensure that another _read() loop is called (including checking first if the stream has been paused).

When in objectMode, though, this is ambiguous. .push('') Could mean that the implementation would like to push out the empty string, since that's a valid object.

I actually need to change this this week (break the spec?) in my browser-based implementation of objectMode streams3.

https://github.com/Livefyre/stream

Currently, .push(), ie pushing undefined, is treated like .push(null), which signals EOF and ends the Readable. Instead, I'm going to make .push() be the equivalent of .push('') in non-objectMde streams.

This would actually be bad for your case. Just like you sometimes expect .read() to return null, theoretically some user may want to .push() the value 'undefined'. I'm kind of fine making that sacrifice, but curious if you or anyone has a better idea. It could be .push(readable.NULL) or something ( :/ ). Similarly, if you needed to be able to .read() out the value 'null', .read() could return something like readable.NULL. But these seem gross..

[Chaoran Yang]

I agree with you on the hole of readable.push(''). I think it would be better to have a separate API for these special cases, e.g. a readable.end() to indicate the stream has ended, and readable.empty() to indicate the same thing as readable.push('').

With respect to the return value. Yes, theoretically some user may want to return the value 'undefined' without indicating the end of a stream. But in practice I have never meet such a case. However, I've meet numerous cases where the value 'null' is an actual return value, other than indicating the end of a stream. I agree use something like readable.NULL (or readable.EOF) would be a solution. But seems gross, and IMO, an overkill.

-Chaoran

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/hhtuFJs0_9Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Floby]

objectMode streams can push falsey values (other than null) since 10.19 or 10.20, I don't remember.

[Ben Noordhuis]

On Thu, Oct 24, 2013 at 3:12 PM, Floby <floren...@gmail.com> wrote:
> objectMode streams can push falsey values (other than null) since 10.19 or
> 10.20, I don't remember.

That's correct, it was fixed in v0.10.19.