0.9x: writeable stream: objectMode

[Aaron Heckmann]

In node < 0.9, WriteStreams converted non-Buffers to Buffers transparently. 

https://github.com/joyent/node/blob/v0.8.20-release/lib/fs.js#L1557-L1561

In 0.9, this is not the case, which leaves me a bit confused with the ReadableStreams objectMode option. I'd like to pipe a ReadableStream in objectMode to a writable stream and rely on conversion to Buffers without needing to create a transform stream, the way I could in 0.8x. Is this by design?

http://nodejs.org/docs/v0.9.8/api/stream.html#stream_new_stream_readable_options

--
Aaron

@aaronheckmann

[Nathan Rajlich]

Aaron, I'm a little confused at exactly what you're trying to do, but if you have a Readable stream outputting "objects" of some kind (since it's in objectMode), and you pipe() to a writable stream that is *not* in objectMode (i.e. it's expecting strings and/or buffers), then what would you expect the Writable stream to do? How would it know how to convert your Object from the readable stream into a Buffer?

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dan Milon]

Judging from the code Aaron linked, neither does v0.8 know how to
convert the object into a buffer. It relies on #toString().

On 02/16/2013 12:45 AM, Nathan Rajlich wrote:
> Aaron, I'm a little confused at exactly what you're trying to do,
> but if you have a Readable stream outputting "objects" of some kind
> (since it's in objectMode), and you pipe() to a writable stream
> that is *not* in objectMode (i.e. it's expecting strings and/or
> buffers), then what would you expect the Writable stream to do? How
> would it know how to convert your Object from the readable stream
> into a Buffer?
>
> On Fri, Feb 15, 2013 at 12:14 PM, Aaron Heckmann

> <aaron.h...@gmail.com <mailto:aaron.h...@gmail.com>>

> wrote:
>
> In node < 0.9, WriteStreams converted non-Buffers to Buffers
> transparently.
>
> https://github.com/joyent/node/blob/v0.8.20-release/lib/fs.js#L1557-L1561
>
> In 0.9, this is not the case, which leaves me a bit confused with
> the ReadableStreams objectMode option. I'd like to pipe a
> ReadableStream in objectMode to a writable stream and rely on
> conversion to Buffers without needing to create a transform
> stream, the way I could in 0.8x. Is this by design?
>
> http://nodejs.org/docs/v0.9.8/api/stream.html#stream_new_stream_readable_options
>
>
>
>

> -- Aaron @aaronheckmann <https://twitter.com/#!/aaronheckmann>

>
>
> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>
>
You received this message because you are subscribed to the Google
> Groups "nodejs" group. To post to this group, send email to

> nod...@googlegroups.com <mailto:nod...@googlegroups.com> To

> unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com

> <mailto:nodejs%2Bunsu...@googlegroups.com> For more options,

> visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --- You received this message because you are subscribed to the
> Google Groups "nodejs" group. To unsubscribe from this group and
> stop receiving emails from it, send an email to
> nodejs+un...@googlegroups.com

> <mailto:nodejs%2Bunsu...@googlegroups.com>. For more options,

[Aaron Heckmann]

Previously a WriteStream created with fs.createWriteStream converted non-Buffers to buffers by concating an empty string with the data which gave us an opportunity to run formatting logic on the data by setting a custom toString method. This no longer works because fs.WriteStream._write rejects any non-Buffer, even in objectMode.

Seems like a bug but I'm just starting to familiarize myself with the new streams and maybe I'm "doing it wrong".

Thanks for the assistance.

--
Aaron

@aaronheckmann

[Isaac Schlueter]

I'd think that writing non-string/buffer data into an fs stream has
probably always been an error. At best, it's undefined behavior
(which you may be relying on, but it's still not guaranteed by the
API).

Relying on custom toString() is an antipattern. It reeks of excessive
magic. Just cast to a string explicitly if that's what you mean to
do.

On Fri, Feb 15, 2013 at 7:53 PM, Aaron Heckmann

[Aaron Heckmann]

I agree my test was magical.

I'm still unclear about objectModes purpose in fs.WriteStream. Is it really more of a

stringMode?

--
Aaron

@aaronheckmann

[Nathan Rajlich]

objectMode is for the *implementor* of the Writable stream. I.e. if I'm writing a writable stream subclass, which is meant to accept Objects instead of bytes (strings/buffers), then I'd use "objectMode". When objectMode is on, the _write() callback function will receive the Objects that have been written, rather than a flattened Buffer object (from the buffers/strings written in non-objectMode).

Readable streams also have "objectMode" but it's basically the same concept: the readable outputs Objects instead of bytes, and thus the .read(n) function will simply return an Object, ignoring the "n" parameter.

Hope that makes sense!

[Aaron Heckmann]

Thanks all!

Aaron

@aaronheckmann

[Ket]

I done it on the client side before piping through.

Hope that makes sense!

stringMode?

>>> nodejs+unsubscribe@googlegroups.com

>>> For more options, visit this group at
>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "nodejs" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to nodejs+unsubscribe@googlegroups.com.

>>> For more options, visit https://groups.google.com/groups/opt_out.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

nodejs+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Aaron

@aaronheckmann