Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
callbacks vs. streams
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  Messages 26 - 36 of 36 - Collapse all  -  Translate all to Translated (View all originals) < Older 
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Tim Caswell  
View profile  
 More options Apr 28 2012, 10:07 am
From: Tim Caswell <t...@creationix.com>
Date: Sat, 28 Apr 2012 09:07:57 -0500
Local: Sat, Apr 28 2012 10:07 am
Subject: Re: [nodejs] Re: callbacks vs. streams

If you want to follow node idioms there are three patterns as Bruno said.
 They are:

Callback: I have an async, non-blocking operation that can take a while.  I
want to be notified when it finishes.  If there was an error instead of
throwing, the error will be passed to the first argument of the callback.
The callback is always the last argument in the original call.  The
callback will only fire one, but will always fire eventually. (Sometimes,
depending on your coding style, you can even add a constraint that the
callback never fires in the same tick the async function is called.  This
is usually done with a nextTick)  Callbacks are the async equivalent to a
normal function call that either returns a value or throws an error.

Event Emitter: I have some object that can emit events 0 or more times.
 These events may have different names too.  Maybe I want to allow multiple
consumers to pick and choose the data they are interested in.  If there is
an error, the event emitter will emit an "error" event.  If there are no
listeners to that event node will throw the error so that it doesn't go
undetected.

Stream: A stream is closer to a callback than an event emitter even though
it's implemented using Event Emitter.  A stream is some query that emits a
finite stream of data.  Usually this is text or binary data.  Streams are
meant to be used to pipe data between processes and sockets. where data
can't be shared via references and you don't want to buffer everything at
every point.  Streams have a very specific interface that involves
.write(), .end(), .pause(), .resume(), "drain", "data", "end" events and
more.  All the Stream prototype adds is a .pipe() function to readable
streams that makes it easy to pipe data from a readable stream to a
writable stream.  While a stream can also be used as a generic event
emitter, the primary reason for using the stream interface it to transport
a stream of homogeneous data that has a finite end between endpoints that
can't see each other directly.

Now I said these are the three node idioms.  Obviously, there are
also JavaScript idioms like the sync iterator callback in
Array.prototype.forEach.

Also I've been known to use vanilla callbacks that get called repeatedly
internally to some libraries for efficiency reasons.  When I write a
parser, for example, I usually write it as a simple function:

    function makeParser(onData) {
      ...
      return write(chunk) {
        .. when something is found
        onData(data)
      }
    }

I don't need an event emitter when all that's ever going to come out are
data events (this is a simple parser)  I just create a single closure.  But
this isn't a stream, if I want my parser to pause it's upstream data source
when whatever it's writing to is busy, I'll need to implement the stream
interface and have a lot more code. (Or the code using this parser can just
watch use the parser as a pure tool and create a stream itself that wraps
the parser)

There are many options, but since the question was about node idioms, they
are one-shot-callback last with err argument, event emitters, and finite
data streams.

On Sat, Apr 28, 2012 at 8:04 AM, Dominic Tarr <dominic.t...@gmail.com>wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Bruno Jouhier  
View profile  
 More options Apr 28 2012, 10:32 am
From: Bruno Jouhier <bjouh...@gmail.com>
Date: Sat, 28 Apr 2012 07:32:05 -0700 (PDT)
Local: Sat, Apr 28 2012 10:32 am
Subject: Re: [nodejs] Re: callbacks vs. streams

Just posted a git with an implementation: https://gist.github.com/2519472


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Oliver Leics  
View profile  
 More options Apr 28 2012, 11:25 am
From: Oliver Leics <oliver.le...@gmail.com>
Date: Sat, 28 Apr 2012 17:25:23 +0200
Local: Sat, Apr 28 2012 11:25 am
Subject: Re: [nodejs] Re: callbacks vs. streams

On Sat, Apr 28, 2012 at 4:07 PM, Tim Caswell <t...@creationix.com> wrote:
> Stream: A stream is closer to a callback than an event emitter even though
> [..]
> Streams are
> meant to be used to pipe data between processes and sockets. where data
> can't be shared via references and you don't want to buffer everything at
> every point.

Is the following statement always true?

"A prime number generator that implements the stream-interface but
never pipes its data to another process or socket should better not
implement the stream-interface."

(I don't want to troll or something like that, I really want to know
your opinions about that)


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tim Caswell  
View profile  
 More options Apr 28 2012, 12:29 pm
From: Tim Caswell <t...@creationix.com>
Date: Sat, 28 Apr 2012 11:29:26 -0500
Local: Sat, Apr 28 2012 12:29 pm
Subject: Re: [nodejs] Re: callbacks vs. streams

On Sat, Apr 28, 2012 at 10:25 AM, Oliver Leics <oliver.le...@gmail.com>wrote:

In that case I would say that a stream interface is overkill.  But like
most things, it depends on what you're interfacing with and your personal
preference for abstraction.  If the code you're interfacing with expects a
stream interface, even though it's in-process, there is something to be
said for a consistent interface.

A prime number generator is a "generator".  That means you pull data from
it when you want the data.  There is no outside force creating the numbers
pushing them to you.  The generator Bruno wrote would be a good abstraction
for an on-demand prime generator. (Or use harmony generators if you prefer
it as a language feature)

If the generator uses threads and is generating primes as fast as possible
in a worker, then maybe an event emitter or stream would make sense.  In
that case there is a real event source.  Some other code running in another
thread notifies you when it has data.  If the data is small enough to
buffer and you don't need the primes till they are all generated, then use
a single callback.  It's less times to synchronize the two threads.  If
there is a ton of data and it's infeasible to buffer or wait till it's all
done, then emit data as it happens (Event Emitter).  If you want to pipe
the data to some other process or write to disk or something that had hard
bandwidth constraints, then use a full stream and implement a way to pause
the number generator when the target is busy.  That way you don't end up
generating numbers faster than you can write them and blow up in memory.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Isaac Schlueter  
View profile  
 More options Apr 28 2012, 12:54 pm
From: Isaac Schlueter <i...@izs.me>
Date: Sat, 28 Apr 2012 09:54:58 -0700
Local: Sat, Apr 28 2012 12:54 pm
Subject: Re: [nodejs] Re: callbacks vs. streams
This is actually much more simple than you're all making it.  The
Node.js API is remarkably consistent on this point.

1. Callback

A callback is for cases where you have a single "question", which will
be answered by either an error, or optionally some data.

getSomeThing(arg, arg, arg, function (er, someThing) {
  if (er) return ohnoes(er)
  // .. now do something with the someThing

})

Callbacks should be called *exactly* once.

2. EventEmitter

EventEmitter : Callback :: Class : Function

An EventEmitter is used in cases where a single object may emit a
bunch of different messages, and you want to handle some of them.  If
an EventEmitter encounters an error, it emits an 'error' event, which
throws if it is unhandled.

var ee = new EventEmitter
ee.on('someEvent', function (data) {
  // some event happened, and wants to tell me about data

})

ee.on('error', function (er) {
  ohnoes(er)

})

EventEmitters are highly optimized, and very cheap to create and use.
(They need to be, because node internally creates bazillions of them.)

You can easily have a class that extends EventEmitter to add
additional semantics:

function Foo() {}
util.inherits(Foo, EventEmitter)
Foo.protoype.getPrimes = function (n) {
  if (this._gettingPrimes) {
    this.emit('error', new Error('already getting some!'))
  }
  this._gettingPrimes = true
  // go and get some primes.
  // when you have one:
  this.emit('prime', thePrime)
  // maybe when you're done:
  this._gettingPrimes = false
  this.emit('end')

})

3. Streams

Streams : Time :: Array : Space

A Stream is a special EventEmitter subclass designed for dealing with
data flowing into or out of some underlying interface, such as a file
or socket.  It's used throughout Node, though a bit clumsily in
places, and the interface and usage is scheduled to be cleaned up in
v0.9.

Readable streams have a .pipe() method that can be used to send the
bytes coming *out* of the readable stream *into* the writable stream.

For this prime generator thing, presumably each prime number takes
some relevant amount of time to generate, but it doesn't sound much
like a Stream, per se.  I'd subclass EventEmitter, and just define
semantics for your events that makes sense for your specific use case.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Glenn Block  
View profile  
 More options Apr 28 2012, 1:42 pm
From: Glenn Block <glenn.bl...@gmail.com>
Date: Sat, 28 Apr 2012 10:42:11 -0700
Local: Sat, Apr 28 2012 1:42 pm
Subject: RE: [nodejs] Re: callbacks vs. streams
So given the case of a multi-step process where each step needs to be
performed async you would recommend callbacks/named callbacks right?

I ask because I was in a discussion with someone who argued that named
callbacks were discouraged for these scenarios and that event emitters
should be used. The other arg was using event emitters meant he could
clearly see in one place how everything is wired up.

My argument was that event emitters are more for pub/sub type scenarios
where there are 0 to n listeners. I felt it was overkill / not the
right abstraction as in this case there is one and only one listener.

Sent from my Windows Phone
From: Isaac Schlueter
Sent: 4/28/2012 9:55 AM
To: nodejs@googlegroups.com
Subject: Re: [nodejs] Re: callbacks vs. streams
This is actually much more simple than you're all making it.  The
Node.js API is remarkably consistent on this point.

1. Callback

A callback is for cases where you have a single "question", which will
be answered by either an error, or optionally some data.

getSomeThing(arg, arg, arg, function (er, someThing) {
  if (er) return ohnoes(er)
  // .. now do something with the someThing

})

Callbacks should be called *exactly* once.

2. EventEmitter

EventEmitter : Callback :: Class : Function

An EventEmitter is used in cases where a single object may emit a
bunch of different messages, and you want to handle some of them.  If
an EventEmitter encounters an error, it emits an 'error' event, which
throws if it is unhandled.

var ee = new EventEmitter
ee.on('someEvent', function (data) {
  // some event happened, and wants to tell me about data

})

ee.on('error', function (er) {
  ohnoes(er)

})

EventEmitters are highly optimized, and very cheap to create and use.
(They need to be, because node internally creates bazillions of them.)

You can easily have a class that extends EventEmitter to add
additional semantics:

function Foo() {}
util.inherits(Foo, EventEmitter)
Foo.protoype.getPrimes = function (n) {
  if (this._gettingPrimes) {
    this.emit('error', new Error('already getting some!'))
  }
  this._gettingPrimes = true
  // go and get some primes.
  // when you have one:
  this.emit('prime', thePrime)
  // maybe when you're done:
  this._gettingPrimes = false
  this.emit('end')

})

3. Streams

Streams : Time :: Array : Space

A Stream is a special EventEmitter subclass designed for dealing with
data flowing into or out of some underlying interface, such as a file
or socket.  It's used throughout Node, though a bit clumsily in
places, and the interface and usage is scheduled to be cleaned up in
v0.9.

Readable streams have a .pipe() method that can be used to send the
bytes coming *out* of the readable stream *into* the writable stream.

For this prime generator thing, presumably each prime number takes
some relevant amount of time to generate, but it doesn't sound much
like a Stream, per se.  I'd subclass EventEmitter, and just define
semantics for your events that makes sense for your specific use case.

--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Volkmann  
View profile  
 More options Apr 28 2012, 2:02 pm
From: Mark Volkmann <r.mark.volkm...@gmail.com>
Date: Sat, 28 Apr 2012 13:02:08 -0500
Local: Sat, Apr 28 2012 2:02 pm
Subject: Re: [nodejs] Re: callbacks vs. streams

On Sat, Apr 28, 2012 at 9:07 AM, Tim Caswell <t...@creationix.com> wrote:
> A stream is some query that emits a finite stream of data.

So do we have agreement that if a stream never emits an "end" event
then something went wrong and this is never intentional? Maybe this is
one criteria for choosing between EventEmitter and streams. Choose
EventEmitter when there is no guaranteed end of data.

--
R. Mark Volkmann
Object Computing, Inc.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Isaac Schlueter  
View profile  
 More options Apr 28 2012, 2:09 pm
From: Isaac Schlueter <i...@izs.me>
Date: Sat, 28 Apr 2012 11:09:16 -0700
Local: Sat, Apr 28 2012 2:09 pm
Subject: Re: [nodejs] Re: callbacks vs. streams

On Sat, Apr 28, 2012 at 10:42, Glenn Block <glenn.bl...@gmail.com> wrote:
> So given the case of a multi-step process where each step needs to be
> performed async you would recommend callbacks/named callbacks right?

If each step is independent of each other step, then callbacks are fine.

obj.doThing(function (er, ok) {
  // ok, done now
  obj.doAnotherThing(function (er, ok) {
    // done with the other thing now
  })

})

is a bit more humane than:

obj.doThing()
obj.once("didthing", function () {
  obj.doAnotherThing()
  obj.once("didanotherthing", function () {

That being said, the most common use case for event emitters in node
is when there's exactly one listener.  That's why we optimize the hell
out of that case.  It's perfectly fine to use them in cases where
there'll be one listener.  The relevant difference is if the event
setup and the method call are separate from one another conceptually.

A callback is "Do X and then when it's done, let me know".  An event
is "Any time X happens, let me know", which may be triggered by some
other thing somewhere else in teh program doing something.

On Sat, Apr 28, 2012 at 11:02, Mark Volkmann <r.mark.volkm...@gmail.com> wrote:
> On Sat, Apr 28, 2012 at 9:07 AM, Tim Caswell <t...@creationix.com> wrote:
>> A stream is some query that emits a finite stream of data.
> So do we have agreement that if a stream never emits an "end" event
> then something went wrong and this is never intentional? Maybe this is
> one criteria for choosing between EventEmitter and streams. Choose
> EventEmitter when there is no guaranteed end of data.

No.  Streams are not necessarily finite.  Consider the case where you
have a video camera streaming data.  You might have a stream of mpeg
chunks or something, but it won't necessarily end (unless someone
shuts off the video camera.)  Or even simpler, say you have a TCP
connection from one server to another.  Under normal circumstances,
that connection might not be severed for days, or years, or longer.

The finiteness of a Stream is determined by the thing it represents,
not by anything essential to the abstraction.

Choose Streams when it is a stream of buffers and strings that you
need to be able to pause and resume, as in the case of TCP
backpressure.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Oliver Leics  
View profile  
 More options Apr 28 2012, 3:27 pm
From: Oliver Leics <oliver.le...@gmail.com>
Date: Sat, 28 Apr 2012 21:27:08 +0200
Local: Sat, Apr 28 2012 3:27 pm
Subject: Re: [nodejs] Re: callbacks vs. streams
I really really like .pipe() of the stream-interface, not just for
netsockets or diskwrites/reads, but also for streams of data in
general.

A silly example to illustrate:
numberGenerator.pipe(fibonacciNumberDetector)
fibonacciNumberDetector.pipe(primeNumberDetector)
primeNumberDetector.pipe(aStreamThatNeedsPrimeFibonacciNumbers)

It is very easy to tell what is going on and what the result should look like.

Or, maybe, see this gist for a more practical example:
https://gist.github.com/2521280
Parse a stream of JSON data, join the streams of the previously parsed
objects into another stream and finally filter objects out of the
joining stream into another stream.

No joke, if you'd tell me right now that streams .pipe() will save the
world, I would believe it; At least for a couple of seconds.

--
Oliver Leics @ G+
https://plus.google.com/112912441146721682527

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Glenn Block  
View profile  
 More options Apr 28 2012, 5:51 pm
From: Glenn Block <glenn.bl...@gmail.com>
Date: Sat, 28 Apr 2012 14:51:41 -0700
Local: Sat, Apr 28 2012 5:51 pm
Subject: Re: [nodejs] Re: callbacks vs. streams

OK so really what you are saying is if you feel more comfortable using
callbacks no problem doing that, but event emitters are totally suitable
for this.

perf wise, is there a significant difference between the two approaches?
How much do you pay for emitting an event over just invoking a callback or
vice versa?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tim Caswell  
View profile  
 More options Apr 28 2012, 8:35 pm
From: Tim Caswell <t...@creationix.com>
Date: Sat, 28 Apr 2012 19:35:48 -0500
Local: Sat, Apr 28 2012 8:35 pm
Subject: Re: [nodejs] Re: callbacks vs. streams

Perhaps I should have been more clear. When I said streams are finite I
meant that you know when they are done.  If you haven't gotten an "end"
event, you can expect more data in the future.  Once there is an "end",
it's done.  This is important for things like cross-process resource
management.

Isaac explained everything very well so I won't repeat it. :)


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages < Older 
« Back to Discussions « Newer topic     Older topic »