callbacks vs. streams

[Mark Volkmann]

This is a question about recommended usage in Node.
As an example, let's say I want to write a function that returns the
first n prime numbers.
I could write a function that takes n and a callback.
It could invoke the callback once for each prime number to be returned
and then invoke it with zero to signal the end.

I could also write a function that takes n and returns a readable stream.
The caller could listen for data events (one per prime number to be returned)
and an end event.

I'm pretty sure the stream approach is preferred,
but would you say that the callback approach is wrong or at least discouraged?

--
R. Mark Volkmann
Object Computing, Inc.

[Steve Molitor]

Interesting question!  I'm looking forward to the answer. 

Both streams and callbacks usually involve IO, but calculating prime number would not.  So it might be a special case.  But still interesting.

Steve

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Tim Caswell]

Since it's a sync operation, I'm assuming no threading tricks, then the question is sync callback vs buffering.  Sync callback has more more overhead since it's many more function calls, but buffering has it's drawbacks if you're not going to need all the answers at once.

// buffering

var primes = getPrimes(n)

// iterator

getPrimes(n, function (prime) {

});

[Mark Volkmann]

Suppose though that it's a long-running operation that may or may not involve I/O. Both approaches would work. Is there a clear bias toward streams?

---

R. Mark Volkmann

Object Computing, Inc.

[Nathan Rajlich]

For times when a callback will be invoked multiple times, it's better to use an EventEmitter:

primeGenerator.on('prime', function (prime) {

  // do stuff with num

});

This allows other interested parties to be able to get the data as well.

[Elijah Insua]

In this case, I'd use callbacks.  Don't get me wrong, streams are great, but for every emission you are talking about 2+ function calls (depending on how many listeners you have). In a tight, synchronous, loop it doesn't make much sense to add in the extra overhead, unless you want to pipe that stream somewhere.

-- Elijah

[Mark Volkmann]

Now we're discussing three options. Perhaps there aren't generally agreed on guidelines for choosing. What criteria could be used to choose between EventEmitter and streams? Is it whether I/O is involved?

---

R. Mark Volkmann

Object Computing, Inc.

[Steve Molitor]

Streams are event emitters of course.  Change your example to '.on('data', function (prime)' and you have a stream.  In this case there is only one kind of event (and no 'end'!), so using a stream would let you pipe and other streamy stuff.

Steve

[Jorge]

I would say that in node calling a callback more than once is not only discouraged, but forbidden, as if it were part of an implied, non written contract.

It seems that every function of node's API that would need to call a cb more than once, has been implemented as an event emitter, apparently on purpose.

I for one don't know the reason why.

In my mind things like fs.readFile(..., cb) could better return the file in small chunks by calling cb(chunk) repeatedly, instead of buffering it all.
--
Jorge.

[Nathan Rajlich]

Jorge, at that point you're back to fs.ReadStream, which has added benefits like .pipe(). Functions like fs.readFile are sugar on top of the Stream API, hence the buffering.

[Mark Hahn]

>  I would say that in node calling a callback more than once is not only discouraged, but forbidden, as if it were part of an implied, non written contract. 

Then how do you explain ...

    http.createServer(function (req, res) {

I have always wondered if that was a kludge to make the example on the front page small.

[crypticswarm]

On Fri, Apr 27, 2012 at 10:40 PM, Mark Hahn <ma...@hahnca.com> wrote:

>  I would say that in node calling a callback more than once is not only discouraged, but forbidden, as if it were part of an implied, non written contract. 

Then how do you explain ...

    http.createServer(function (req, res) {

http.createServer(function (req, res) {})

is sugar for:

http.createServer().on('request', function (req, res) {})

[Mark Hahn]

How can it be sugar?  It isn't compiled into anything else.  Sugar is part of a syntax in a language.

You can call it whatever you want but it is a anonymous function being used as a callback and passed to a node function that is called many times.

--

[Lothar Pfeiler]

I understand the philosophy of using callbacks as "Please call me
back, when the job is done". Meaning, call me back after calculating
all prime numbers. (avoid roundtrips)

I understand the event emitter philosophy as "Please interrupt me all
the time you have news for me". Meaning, interrupt me for every single
prime number. (lots of roundtrips, because I want to show the most
current state all the time)

So, a PrimeNumberGenerator could offer both versions for every
purpose. And every single app might have a different understanding of
the job size (or chunk size) and what are the news. So, the question
for the app (which uses the PNG) would be, whether it wants to act on
every event or just act on a final result.

Lothar

On Apr 28, 6:20 am, Mark Hahn <m...@hahnca.com> wrote:
> How can it be sugar?  It isn't compiled into anything else.  Sugar is part
> of a syntax in a language.
>
> You can call it whatever you want but it is a anonymous function being used
> as a callback and passed to a node function that is called many times.
>

> On Fri, Apr 27, 2012 at 8:57 PM, crypticswarm <crypticsw...@gmail.com>wrote:

[Bruno Jouhier]

Neither!

I think that question should be callbacks vs. events vs. streams:

• Callback: called only once, when the function is done, to return result (optional) or error.
• Event: called repeatedly by the function to notify its listeners. Can be used to send intermediate results as they come.
• Stream: higher level concept based on events, with standardized event types (data, end, error, drain) and standardized API (pause, resume, write, ..).

So, assuming prime computation is asynchronous:

• If the function computes the N first primes and returns them all at once, it should use a callback.
• If the function returns the primes one by one, it should use events. It may also use a stream but that seems a bit over-engineered.

[Jorge]

Hi Bruno,

On Apr 28, 2012, at 11:14 AM, Bruno Jouhier wrote:

> Neither!
>
> I think that question should be callbacks vs. events vs. streams:
> • Callback: called only once, when the function is done, to return result (optional) or error.
> • Event: called repeatedly by the function to notify its listeners. Can be used to send intermediate results as they come.
> • Stream: higher level concept based on events, with standardized event types (data, end, error, drain) and standardized API (pause, resume, write, ..).
> So, assuming prime computation is asynchronous:
>
> • If the function computes the N first primes and returns them all at once, it should use a callback.
> • If the function returns the primes one by one, it should use events. It may also use a stream but that seems a bit over-engineered.

Good, yes, that's the way it is in node, but it does not explain *why* it is so.

.readFile() could as well deliver the chunks to the cb as they're read from disk, and to use it you'd need to write just one line:

fs.readFile(path, cb);

While to do the same with an evented interface there's a lot of boilerplate to write, and an extra object to create:

reader= new fileReaderConstructor(path);
reader.on('data', cb);
reader.on('end', endCB);
reader.on('error', errorCB);

It seems to me that the former is more functional and makes good use of closures (*), while the latter is the approach a classic OOP programmer would tend to write instead.

(*)In JavaScript we don't need to create objects to save state.
--
Jorge.

[Oliver Leics]

As i see it: We have a prime number generator that constantly *emits*
prime numbers. This is a *stream* of prime numbers. Thus: Use
stream.Stream.

IMHO implementing this as a stream it is not over-engineered. For me
it is the optimum way as streams provides a standard method for
everything emitting data. It makes it very easy to hook into that
stream of data and do whatever needs to be done with that prime
numbers. I do not have to read lots of API documentation. All i need
to know: It is a standard node stream of data.

BTW: I'm writing this as i recently added 'lazy' to my toolbelt of
indispensable node modules.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

--
Oliver Leics @ G+
https://plus.google.com/112912441146721682527

[Dominic Tarr]

There is one very good reason for using streams in a case like this,
that no one has mentioned yet.

First, lets just suppose that N is very large.
Also lets generate primes in nextTick so that it never blocks for very long.

And the thing that consume's the primes (data) only needs one at a time.

There could be a variety of reasons for this. one might be that it's
serializing them as a web service.

Now, streams have a cool feature that EventEmitters do not have.
pause()/resume()

Like streams, TCP also has pause. If the pipe is congested or packets
are being dropped
TCP actually slows down a bit.

So if you've made PrimeStream respect pause/resume then it will only
go as fast as the bottleneck (tcp) can handle them!

you can't do that will callbacks.

a callback might be fine if I only want N < ~500 but what if I want
billions and billions of primes?

what if I didn't know how many primes I want. I could just say give me
ALL the primes!
(N = Infinity) and then close the connection when I'm satisfied.

Another benefit of using streams is that you don't need to keep
everything in memory.
you can upload a file that is larger than your ram with streams (I
think you might need to keep a list of primes to calc the next prime,
so this benefit doesn't apply to this case)

in summary, use streams if:
* you can start using the data before you have received everything.
* you would like to be able to throttle the rate something is produced.

usually this is IO related. although, if you have I --> O it may make
sense to keep the _through_ part in the shape of a stream also.

use callbacks if:
* you can't do the next thing until after a thing is done.
* you need the whole thing.

example: create a driectory, delete a file. parse a JSON document
(note, if the document is LARGE and list like, it may make sense to
stream it)

use an EventEmitter if:
* many things will happen
* things will happen more than once.
* multiple listeners may be interested in an event.

note: createServer(function (req, res) {...
is not a callback, because callbacks ALWAYS have err as the first argument.
as pointed out above. it's an event listener. which has no error argument.

[Dominic Tarr]

oh, forgot something about event listeners:

unlike callbacks, you can decide when you are no longer interested

oh, I just reread the original message where you are talking about
calling the callback multiple times. that is what streams are for. a
callback should never callback twice. those bugs can be really hard to
find.

example, what is wrong with this code:

function parseJson (file, cb) {
fs.readFile(fileName, function (err, data) {
if(err) return cb(err);
try { cb(null, JSON.parse(data)); }
catch (err) { cb(err); }
});
}

?

[Oliver Leics]

On Sat, Apr 28, 2012 at 1:58 PM, Dominic Tarr <domini...@gmail.com> wrote:
> example, what is wrong with this code:
>
> function parseJson (file, cb) {
>  fs.readFile(fileName, function (err, data) {
>    if(err) return cb(err);
>    try { cb(null, JSON.parse(data)); }
>    catch (err) { cb(err); }
>  });
> }
>
> ?

Throws, as fileName will be undefined :-P

[Dominic Tarr]

oops,

s/file/fileName

... that is the easy to find bug, but there is another one lurking...

[Jorge]

If the cb() in the try {} throws it will be called again in the catch {} ?

--
Jorge.

[Lothar Pfeiler]

... just clicked the send button too early.

I use streams if the expected data is big and I want the memory
consumption to be low. Meaning, "Please interrupt me, and give me
small chunks, because I couldn't handle the big response".

Lothar

[Dominic Tarr]

@jorge

exactly. that means that you callback both succeed and failed.

[Bruno Jouhier]

Hi Jorge,

Yes, this is just the way things are in node.

I would probably do it slightly differently, as you suggest, with an API that lets you read the prime numbers one by one:

  var nextPrime = primeGenerator.create();
  nextPrime(cb);

State encapsulated in a closure rather than an object. I like that.

Bruno

[Tim Caswell]

If you want to follow node idioms there are three patterns as Bruno said.  They are:

Callback: I have an async, non-blocking operation that can take a while.  I want to be notified when it finishes.  If there was an error instead of throwing, the error will be passed to the first argument of the callback. The callback is always the last argument in the original call.  The callback will only fire one, but will always fire eventually. (Sometimes, depending on your coding style, you can even add a constraint that the callback never fires in the same tick the async function is called.  This is usually done with a nextTick)  Callbacks are the async equivalent to a normal function call that either returns a value or throws an error.

Event Emitter: I have some object that can emit events 0 or more times.  These events may have different names too.  Maybe I want to allow multiple consumers to pick and choose the data they are interested in.  If there is an error, the event emitter will emit an "error" event.  If there are no listeners to that event node will throw the error so that it doesn't go undetected.

Stream: A stream is closer to a callback than an event emitter even though it's implemented using Event Emitter.  A stream is some query that emits a finite stream of data.  Usually this is text or binary data.  Streams are meant to be used to pipe data between processes and sockets. where data can't be shared via references and you don't want to buffer everything at every point.  Streams have a very specific interface that involves .write(), .end(), .pause(), .resume(), "drain", "data", "end" events and more.  All the Stream prototype adds is a .pipe() function to readable streams that makes it easy to pipe data from a readable stream to a writable stream.  While a stream can also be used as a generic event emitter, the primary reason for using the stream interface it to transport a stream of homogeneous data that has a finite end between endpoints that can't see each other directly.

Now I said these are the three node idioms.  Obviously, there are also JavaScript idioms like the sync iterator callback in Array.prototype.forEach.

Also I've been known to use vanilla callbacks that get called repeatedly internally to some libraries for efficiency reasons.  When I write a parser, for example, I usually write it as a simple function:

    function makeParser(onData) {

      ...

      return write(chunk) {

        .. when something is found

        onData(data)

      }

    }

I don't need an event emitter when all that's ever going to come out are data events (this is a simple parser)  I just create a single closure.  But this isn't a stream, if I want my parser to pause it's upstream data source when whatever it's writing to is busy, I'll need to implement the stream interface and have a lot more code. (Or the code using this parser can just watch use the parser as a pure tool and create a stream itself that wraps the parser)

There are many options, but since the question was about node idioms, they are one-shot-callback last with err argument, event emitters, and finite data streams.

[Bruno Jouhier]

Just posted a git with an implementation: https://gist.github.com/2519472

[Oliver Leics]

On Sat, Apr 28, 2012 at 4:07 PM, Tim Caswell <t...@creationix.com> wrote:
> Stream: A stream is closer to a callback than an event emitter even though

> [..]

> Streams are
> meant to be used to pipe data between processes and sockets. where data
> can't be shared via references and you don't want to buffer everything at
> every point.

Is the following statement always true?

"A prime number generator that implements the stream-interface but
never pipes its data to another process or socket should better not
implement the stream-interface."

(I don't want to troll or something like that, I really want to know
your opinions about that)

[Tim Caswell]

In that case I would say that a stream interface is overkill.  But like most things, it depends on what you're interfacing with and your personal preference for abstraction.  If the code you're interfacing with expects a stream interface, even though it's in-process, there is something to be said for a consistent interface.

A prime number generator is a "generator".  That means you pull data from it when you want the data.  There is no outside force creating the numbers pushing them to you.  The generator Bruno wrote would be a good abstraction for an on-demand prime generator. (Or use harmony generators if you prefer it as a language feature)

If the generator uses threads and is generating primes as fast as possible in a worker, then maybe an event emitter or stream would make sense.  In that case there is a real event source.  Some other code running in another thread notifies you when it has data.  If the data is small enough to buffer and you don't need the primes till they are all generated, then use a single callback.  It's less times to synchronize the two threads.  If there is a ton of data and it's infeasible to buffer or wait till it's all done, then emit data as it happens (Event Emitter).  If you want to pipe the data to some other process or write to disk or something that had hard bandwidth constraints, then use a full stream and implement a way to pause the number generator when the target is busy.  That way you don't end up generating numbers faster than you can write them and blow up in memory.

[Isaac Schlueter]

This is actually much more simple than you're all making it. The
Node.js API is remarkably consistent on this point.

1. Callback

A callback is for cases where you have a single "question", which will
be answered by either an error, or optionally some data.

getSomeThing(arg, arg, arg, function (er, someThing) {
if (er) return ohnoes(er)
// .. now do something with the someThing
})

Callbacks should be called *exactly* once.

2. EventEmitter

EventEmitter : Callback :: Class : Function

An EventEmitter is used in cases where a single object may emit a
bunch of different messages, and you want to handle some of them. If
an EventEmitter encounters an error, it emits an 'error' event, which
throws if it is unhandled.

var ee = new EventEmitter
ee.on('someEvent', function (data) {
// some event happened, and wants to tell me about data
})
ee.on('error', function (er) {
ohnoes(er)
})

EventEmitters are highly optimized, and very cheap to create and use.
(They need to be, because node internally creates bazillions of them.)

You can easily have a class that extends EventEmitter to add
additional semantics:

function Foo() {}
util.inherits(Foo, EventEmitter)
Foo.protoype.getPrimes = function (n) {
if (this._gettingPrimes) {
this.emit('error', new Error('already getting some!'))
}
this._gettingPrimes = true
// go and get some primes.
// when you have one:
this.emit('prime', thePrime)
// maybe when you're done:
this._gettingPrimes = false
this.emit('end')
})


3. Streams

Streams : Time :: Array : Space

A Stream is a special EventEmitter subclass designed for dealing with
data flowing into or out of some underlying interface, such as a file
or socket. It's used throughout Node, though a bit clumsily in
places, and the interface and usage is scheduled to be cleaned up in
v0.9.

Readable streams have a .pipe() method that can be used to send the
bytes coming *out* of the readable stream *into* the writable stream.



For this prime generator thing, presumably each prime number takes
some relevant amount of time to generate, but it doesn't sound much
like a Stream, per se. I'd subclass EventEmitter, and just define
semantics for your events that makes sense for your specific use case.

[Glenn Block]

So given the case of a multi-step process where each step needs to be
performed async you would recommend callbacks/named callbacks right?

I ask because I was in a discussion with someone who argued that named
callbacks were discouraged for these scenarios and that event emitters
should be used. The other arg was using event emitters meant he could
clearly see in one place how everything is wired up.

My argument was that event emitters are more for pub/sub type scenarios
where there are 0 to n listeners. I felt it was overkill / not the
right abstraction as in this case there is one and only one listener.

Sent from my Windows Phone
From: Isaac Schlueter
Sent: 4/28/2012 9:55 AM
To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: callbacks vs. streams

[Mark Volkmann]

On Sat, Apr 28, 2012 at 9:07 AM, Tim Caswell <t...@creationix.com> wrote:

> A stream is some query that emits a finite stream of data.

So do we have agreement that if a stream never emits an "end" event
then something went wrong and this is never intentional? Maybe this is
one criteria for choosing between EventEmitter and streams. Choose
EventEmitter when there is no guaranteed end of data.

[Isaac Schlueter]

On Sat, Apr 28, 2012 at 10:42, Glenn Block <glenn...@gmail.com> wrote:
> So given the case of a multi-step process where each step needs to be
> performed async you would recommend callbacks/named callbacks right?

If each step is independent of each other step, then callbacks are fine.

obj.doThing(function (er, ok) {
// ok, done now
obj.doAnotherThing(function (er, ok) {
// done with the other thing now
})
})

is a bit more humane than:

obj.doThing()
obj.once("didthing", function () {
obj.doAnotherThing()
obj.once("didanotherthing", function () {


That being said, the most common use case for event emitters in node
is when there's exactly one listener. That's why we optimize the hell
out of that case. It's perfectly fine to use them in cases where
there'll be one listener. The relevant difference is if the event
setup and the method call are separate from one another conceptually.

A callback is "Do X and then when it's done, let me know". An event
is "Any time X happens, let me know", which may be triggered by some
other thing somewhere else in teh program doing something.

On Sat, Apr 28, 2012 at 11:02, Mark Volkmann <r.mark....@gmail.com> wrote:
> On Sat, Apr 28, 2012 at 9:07 AM, Tim Caswell <t...@creationix.com> wrote:
>> A stream is some query that emits a finite stream of data.
> So do we have agreement that if a stream never emits an "end" event
> then something went wrong and this is never intentional? Maybe this is
> one criteria for choosing between EventEmitter and streams. Choose
> EventEmitter when there is no guaranteed end of data.

No. Streams are not necessarily finite. Consider the case where you
have a video camera streaming data. You might have a stream of mpeg
chunks or something, but it won't necessarily end (unless someone
shuts off the video camera.) Or even simpler, say you have a TCP
connection from one server to another. Under normal circumstances,
that connection might not be severed for days, or years, or longer.

The finiteness of a Stream is determined by the thing it represents,
not by anything essential to the abstraction.

Choose Streams when it is a stream of buffers and strings that you
need to be able to pause and resume, as in the case of TCP
backpressure.

[Oliver Leics]

I really really like .pipe() of the stream-interface, not just for
netsockets or diskwrites/reads, but also for streams of data in
general.

A silly example to illustrate:
numberGenerator.pipe(fibonacciNumberDetector)
fibonacciNumberDetector.pipe(primeNumberDetector)
primeNumberDetector.pipe(aStreamThatNeedsPrimeFibonacciNumbers)

It is very easy to tell what is going on and what the result should look like.

Or, maybe, see this gist for a more practical example:
https://gist.github.com/2521280
Parse a stream of JSON data, join the streams of the previously parsed
objects into another stream and finally filter objects out of the
joining stream into another stream.

No joke, if you'd tell me right now that streams .pipe() will save the
world, I would believe it; At least for a couple of seconds.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en



--

[Glenn Block]

OK so really what you are saying is if you feel more comfortable using callbacks no problem doing that, but event emitters are totally suitable for this.

perf wise, is there a significant difference between the two approaches? How much do you pay for emitting an event over just invoking a callback or vice versa?

[Tim Caswell]

Perhaps I should have been more clear. When I said streams are finite I meant that you know when they are done.  If you haven't gotten an "end" event, you can expect more data in the future.  Once there is an "end", it's done.  This is important for things like cross-process resource management.

Isaac explained everything very well so I won't repeat it. :)