Spawned processes as transform streams

Ryan Schmidt

unread,

Jun 13, 2013, 9:23:08 PM6/13/13

to nod...@googlegroups.com

I understand that a process that I spawn with require('child_process').spawn() *has* three streams: stdin, stdout, stderr.

http://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options

But I've now read about require('stream').Transform and it feels like for certain types of programs (compression programs like bzip2 or image conversion programs like ImageMagick) I would want the spawned process to *be* a stream -- a transform stream.

http://nodejs.org/api/stream.html#stream_class_stream_transform

I've tried to find examples of how to wrap a spawned child process in a transform stream, and I haven't found any, which makes me think I'm going about this the wrong way.

I want to be able to do something like this:

var rs; // a readable stream, maybe a file
var ws; // a writable stream, maybe an http response
var convert; // a transform stream that uses ImageMagick's convert program
rs.pipe(convert).pipe(ws);

Am I wrong to want this?
If not, how to I do this?

Marco Rogers

unread,

Jun 14, 2013, 2:01:50 AM6/14/13

to nod...@googlegroups.com

You've got the right idea. But you're right that it can be difficult to work out the details. I've been giving talks on this recently, and I've got a repo that may help. Check out my example gzip stream. There are slides and other examples in there that may also be helpful.

https://github.com/polotek/txjs-2013-streams-talk/blob/master/streams/gzipstream.js

Keep in mind that this is an example. Don't use this for gzipping. Gzip streaming is now supported by the core zlib module in node core.

:Marco

Andrey

unread,

Jun 14, 2013, 2:14:18 AM6/14/13

to nod...@googlegroups.com

You can use https://github.com/sidorares/exec-stream

var es = require('exec_stream'); var convert = es('imagemagick', ['options']); rs.pipe(convert).pipe(ws);

Marco Rogers

unread,

Jun 14, 2013, 2:19:14 AM6/14/13

to nod...@googlegroups.com

On Thu, Jun 13, 2013 at 11:14 PM, Andrey <andrey....@gmail.com> wrote:

You can use https://github.com/sidorares/exec-stream
var es = require('exec_stream'); var convert = es('imagemagick', ['options']); rs.pipe(convert).pipe(ws);

This looks great. Do you have plans to upgrade it with the latest APIs? It's not obvious, but it will pay to support the updated streams. Using the base classes like I've done in my example gets you all the goodness minimal less fuss. Wrap it in your nice module and it's a one-stop solution.

:Marco

On Friday, 14 June 2013 11:23:08 UTC+10, ryandesign wrote:
I understand that a process that I spawn with require('child_process').spawn() *has* three streams: stdin, stdout, stderr.

http://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options

But I've now read about require('stream').Transform and it feels like for certain types of programs (compression programs like bzip2 or image conversion programs like ImageMagick) I would want the spawned process to *be* a stream -- a transform stream.

http://nodejs.org/api/stream.html#stream_class_stream_transform

I've tried to find examples of how to wrap a spawned child process in a transform stream, and I haven't found any, which makes me think I'm going about this the wrong way.

I want to be able to do something like this:

var rs; // a readable stream, maybe a file
var ws; // a writable stream, maybe an http response
var convert; // a transform stream that uses ImageMagick's convert program
rs.pipe(convert).pipe(ws);

Am I wrong to want this?
If not, how to I do this?

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/sbvdxC6i5Yw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Marco Rogers
marco....@gmail.com | https://twitter.com/polotek

Life is ten percent what happens to you and ninety percent how you respond to it.
- Lou Holtz

Andrey

unread,

Jun 14, 2013, 2:49:14 AM6/14/13

to nod...@googlegroups.com

Yes, I want upgrade it to streams2. Alternatively there should already be some way to compose proper stream2 duplex stream using existing Dominic Tarr modules (duplex/mux-demux/event-stream etc)

Ryan Schmidt

unread,

Jun 14, 2013, 6:21:07 AM6/14/13

to nod...@googlegroups.com

On Jun 14, 2013, at 01:01, Marco Rogers <marco....@gmail.com> wrote:

> You've got the right idea. But you're right that it can be difficult to work out the details. I've been giving talks on this recently, and I've got a repo that may help. Check out my example gzip stream. There are slides and other examples in there that may also be helpful.
>
> https://github.com/polotek/txjs-2013-streams-talk/blob/master/streams/gzipstream.js
>
> Keep in mind that this is an example. Don't use this for gzipping. Gzip streaming is now supported by the core zlib module in node core.

Thanks! This is great. I'd gotten about half of that example written before becoming confused. I haven't made my own streams before. Seeing a complete example is very helpful.

On Jun 14, 2013, at 01:14, Andrey <andrey....@gmail.com> wrote:

> You can use https://github.com/sidorares/exec-stream

Having this wrapped up in a module is fantastic, since it is quite a few things to get right. Thank you! I had searched for "spawn-stream" and not found anything, and hadn't made the leap to search for "exec-stream" yet.

Now to try these solutions out and find out what, if anything, is done with stderr. Looking at the code briefly, it doesn't look like stderr is handled.

unread,

Jun 17, 2013, 3:02:31 AM6/17/13

to nod...@googlegroups.com

Upon further investigation, I've found that in fact the example silently discards the last chunk of data. 0.10.4 and earlier silently ignored this problem, while 0.10.5 and later now correctly raise an error about that.

I found a fix for the example and submitted a patch here:

https://github.com/polotek/txjs-2013-streams-talk/issues/1

Reply all

Reply to author

Forward