I love async, but I can't code like this

5,228 views
Skip to first unread message

Sam McCall

unread,
Jan 11, 2011, 11:47:21 AM1/11/11
to nodejs
I love the event loop, and I understand the execution model, and I
don't want to change it.
I don't feel like Javascript is desperately missing macros.
But I need someone to talk me down, because I'm coming to the
conclusion that the certain sorts of programs are unwritable without
language extensions.
Not technically unwritable, of course, they work really well. But the
code is so full of callback noise that you can't see the logic.

I have a server that speaks an HTTP/JSON based protocol, and client
libraries for Java + Node.JS. (It's for GUI automation, think Selenium
for windows apps. Open source soon, waiting on corp lawyers).
Client apps are highly sequential. They're easy to write in Java,
they're easy to write in synchronous JavaScript, and they'd be easy to
write using Bruno Jouhier's transformation library.
The consensus seems to be that threads are evil (I agree), source
transformation is evil (I agree), and that you can write anything you
want in vanilla JS today.
Well, show me how to write this in a way that's nearly as concise as
Java (surely not a high standard!).

I don't have an easy solution, using anything other than standard
JavaScript syntax + semantics is incredibly problematic.
But can we admit that there's a problem?

== Java: ==
mainWindow.menu("File").openMenu().item("Open").click();
Window dialog = mainWindow.getChild(type(Window.class));
...

== Evil synchronous JavaScript ==
mainWindow.menu('File').openMenu().item('Open').click();
var dialog = mainWindow.getChild(type('Window'));

== JavaScript: ==
mainWindow.menu("File", function(err, file) {
if(err) throw err;
file.openMenu(function(err, menu) {
if(err) throw err;
menu.item("Open", function(err, item) {
if(err) throw err;
item.click(function(err) {
if(err) throw err;
mainWindow.getChild(type('Window'), function(err, dialog) {
if(err) throw err;
...
});
});
});
});
});

== JavaScript + Seq: ==
Seq()
.seq(function() { mainWindow.menu("File", this); })
.seq(function(file) { file.openMenu(this); })
.seq(function(menu) { menu.item("Open", this); })
.seq(function(open) { open.click(); })
.seq(function(){ mainWindow.getChild(type('Window'), this); })
.seq(function(dialog){ ... });

---
Cheers,
Sam McCall
node.js newbie

Nick Husher

unread,
Jan 11, 2011, 1:36:08 PM1/11/11
to nod...@googlegroups.com
Is interacting with (what appears to be) a GUI menu actually a
blocking operation? The point of Node isn't to make everything
asynchronous, the point is to note clearly and loudly where you have
places in your code where you're communicating with a slow data
source. In other words, the file system or the network.

Let's say you're communicating with a GUI over the network: that's
cool, sometimes it's necessary. I recommend writing a wrapper that
abstracts all that asynchronous business into a set of commands that
can be queued up and run, since it's only the last callback you're
really interested in.

> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>

Guillermo Rauch

unread,
Jan 11, 2011, 1:43:05 PM1/11/11
to nod...@googlegroups.com
You're comparing apples to oranges.
You're not showing off an alternative concurrency model with your Java example. You're just using a blocking API. You can do the same thing with Node.JS

--
Guillermo Rauch
http://devthought.com

Sam McCall

unread,
Jan 11, 2011, 2:02:05 PM1/11/11
to nodejs
On Jan 11, 7:43 pm, Guillermo Rauch <rau...@gmail.com> wrote:
> You're just using a blocking API. You can do the same thing with Node.JS

Er, that would block the whole app, as node.js has no threads.

Guillermo Rauch

unread,
Jan 11, 2011, 2:18:20 PM1/11/11
to nod...@googlegroups.com
If you're writing a desktop GUI app, you have two options:
 - Blocking API + web workers
 - Non-blocking and some API that simplifies the nesting and error handling. See how we do it for Soda


browser
  .chain
  .session()
  .open('/')
  .assertTitle('Something')
  .and(login('foo', 'bar'))
  .assertTitle('Foobar')
  .and(login('someone', 'else'))
  .assertTitle('Someone else')
  .end(function(err){
    if (err) throw err;
  });

Sam McCall

unread,
Jan 11, 2011, 3:00:41 PM1/11/11
to nodejs
On Jan 11, 7:36 pm, Nick Husher <nhus...@gmail.com> wrote:
> Is interacting with (what appears to be) a GUI menu actually a
> blocking operation?
It's blocking per-session: suppose we're automating a notepad session
on machine A, if we call openMenu on a menu item then we can't do
anything[1] until we get the HTTP response. But of course we can do
work on another session targeted at another machine.

[1] Actually we can, but whether the menu is opened/closed/midway
isn't defined, so we can't usually do anything useful.

> Let's say you're communicating with a GUI over the network: that's
> cool, sometimes it's necessary.
That's exactly it, sorry if that wasn't clear.

> I recommend writing a wrapper that
> abstracts all that asynchronous business into a set of commands that
> can be queued up and run, since it's only the last callback you're
> really interested in.
Sure, this example is common enough to justify a library function like
window.clickMenuItem("File", "Open", cb);

Unfortunately, I chose it because it was universal/familiar, in
practice the code snippets tend to be application specific because the
UIAutomation tree (the Windows equivalent of the DOM) tends to vary in
unintuitive per-application ways.
Therefore the wrappers couldn't be part of the library, they'd be
written by whoever uses the library, for every app-specific task they
want to do. So you're writing the same noisy code, but in a function.

The JavaScript client is experimental, but we've used the Java client
to write automated tests for in-house software and in practice there's
little to no redundancy in the API - a higher level API would stop you
doing things you need to be able to do.

On Jan 11, 8:18 pm, Guillermo Rauch <rau...@gmail.com> wrote:
> If you're writing a desktop GUI app, you have two options:
>  - Blocking API + web workers
This is obviously an option, but it sucks having to have all the
overhead and sync problems of threads just to expose an API you can
write readable programs in. (Plus it doesn't work in Node yet,
right?)

> - Non-blocking and some API that simplifies the nesting and error handling.
That looks like a cool pattern, I'll check it out.
Our API is object-oriented (very similar to WebDriver) so we need to
deal with different object handles rather than just a single Selenium
connection, and there's sometimes complex flow-control mid-chain, but
I'll see if I can get something that works reasonably. (And ideally
that separates cleanly from both the app code and the library, so both
are readable). At this point i'm starting to disagree with people who
think everyone should write their own!

Cheers,
Sam

Bruno Jouhier

unread,
Jan 11, 2011, 3:29:59 PM1/11/11
to nodejs
Well Sam,

I wrote streamline.js for people like you and me, people who like
Javascript, who like node's event model and who are involved in
industrial projects, which means that they will, at one point or
another, need to involve "mainstream" programmers into it.

Mainstream programmers understand Javascript's flow control
statements. So I think that they'll understand streamline.js. I don't
think that they will be ever as productive and as comfortable with a
DSL however smart the DSL might be. Why would you need a DSL anyway
when all you're asking for is basic flow control statements that
actually exist in the language but got broken somehow. What you want
is that someone fixes the language so that you can use for loops, try/
catch, dot notation again, not that someone introduces new
abstractions that will never be as concise as the language's
primitives, that will have a learning curve and that will be obsolete
the next month because a newer abstraction will have taken the
spotlights.

I released streamline.js under MIT license which is not our standard
corporate license so that the community can feel relaxed about it (I'm
probably more fortunate than you because I have a kind CTO who
expedited things with the lawyers). So it is available and free and we
can start using it.

I am not particularly fond of preprocessors and I went this route
simply because I did not see any way to fix the problem in user-land
(I tried to write my own DSL like everyone else of course but I felt
like loosing my time). That's all. There is no ideological stand in
this, just pragmaticism.

So even if code transformation is evil, we may have to sympathize with
the devil, and write rock'n roll code. At least that's what I'm doing
from this point on. And maybe some day, the fix will get into the
language for real. After all, all it needs is a small operator or
keyword (to replace the underscore hack) and a relatively simple
algebraic code transfomation that can be pipelined into the other
stages of the compiler.

Bruno

Liam

unread,
Jan 11, 2011, 3:56:16 PM1/11/11
to nodejs
Sam, it sounds to me like you ought to be sending "scripts" to the UI
entities, and returning only the result you need for a particular
sequence...

Isn't there a serialization format for UI automation sequences that
allows them to be saved and played back?

Isaac Schlueter

unread,
Jan 11, 2011, 4:35:41 PM1/11/11
to nod...@googlegroups.com
Cognitive shenanigans. Strawman fallacy, more specifically, fallacy
of the excluded middle. No one actually writes code like this. The
options are not "73 layers of indentation, or threads."

The language includes named function declarations. Use them. That
alone would make your example way less terrible.

--i

Bradley Meck

unread,
Jan 11, 2011, 4:46:22 PM1/11/11
to nod...@googlegroups.com
name all your functions anyway for debugging purposes, lord knows seeing a stack trace of anonymous functions is painful on the eyes.

Sam McCall

unread,
Jan 11, 2011, 5:09:21 PM1/11/11
to nodejs
Bruno: I'm really tempted by streamline.js, and may well end up using
it.
I really don't like the idea of a preprocessor, with and the
associated deployment hassles and ambiguous semantics. So if it can be
done with something like a proxy object that records a chain of method
calls synchronously, then plays them back asynchronously, I'd prefer
that. But if it isn't manageable, then pragmatically I think it'll be
the best option. (Your implementation looks really good, it's a purely
philosophical objection).

Liam: we haven't defined such a format. The actions are functions with
complex object parameters and many different types of return values
that can raise errors at any step etc. This would basically mean
defining a new scripting language with synchronous semantics and
writing an asynchronous interpreter for it. It seems strange to have
to do this.

Isaac: there are 5 inline functions declared in the example.
'var x = function(){};' 5 times is a lot of code even before you do
anything in them, it can never be concise or really readable.
Besides, I'm not really sure what to name them as they don't really
have much independent behaviour, they're just glue.
But I'd be happy to be convinced, could you show me how you'd write it?

Liam

unread,
Jan 11, 2011, 6:25:17 PM1/11/11
to nodejs
Hmm, so fine-grained remote control scripts are a pain in callback-
style programming...

But if you don't need tons of concurrent script "threads", perhaps
write sync JS scripts and use a pool of Node processes consuming a
queue of scripts?

Isaac Schlueter

unread,
Jan 11, 2011, 9:33:37 PM1/11/11
to nod...@googlegroups.com
On Tue, Jan 11, 2011 at 14:09, Sam McCall <sam.m...@gmail.com> wrote:
> Isaac: there are 5 inline functions declared in the example.
> 'var x = function(){};' 5 times is a lot of code even before you do
> anything in them, it can never be concise or really readable.
> Besides, I'm not really sure what to name them as they don't really
> have much independent behaviour, they're just glue.
> But I'd be happy to be convinced, could you show me how you'd write it?

Cut out the "var" and the "=". But yeah, "function" is long and
klunky for how common it is. I'd love a shorter keyword for it.

Here's a very brief untested example of what I'm talking about:
https://gist.github.com/775591

I usually only care about success or failure, so I don't use any kind
of "passalong" util in my programs. But it's not hard to see how this
general technique could be applied in other ways.

--i

shaun etherton

unread,
Jan 11, 2011, 10:08:49 PM1/11/11
to nod...@googlegroups.com

IMO this is a great example; especially for new comers. It makes the
strange looking yet common function nesting look more familiar, then
progresses to show the awesomeness of JS in a simple, clear and
understandable way.
Sometimes i think those who are intimately familiar with the nested
callbacks/anon function style may not realize the amount of angst and
confusion it can cause for the uninitiated.

- shaun

mgutz

unread,
Jan 11, 2011, 10:45:27 PM1/11/11
to nodejs
And of course CoffeeScript (the more signal-to-noise ratio JavaScript)

# using named functions
doSomething = (done) ->
mainWindow.menu "File", onFile

onFile = (er, file) ->
return cb(er) if er
file.openMenu onMenu

onMenu = (er, menu) ->
return er if er
menu.item "Open", onOpen

onOpen = (er, item) ->
return er if er
item.click onClick

onClick = (er) ->
return er if er
mainWindow.getChild type("Window"), onGetWindow

onGetWindow = (er, dialog) ->
return er if er
dialog.doSomething done

mgutz

unread,
Jan 11, 2011, 10:49:22 PM1/11/11
to nodejs
oops, pasted wrong one

# using named functions
function doSomething (cb) ->
mainWindow.menu "File", onFile

onFile = (er, file) ->
return cb(er) if er
file.openMenu onMenu

onMenu = (er, menu) ->
return cb(er) if er
menu.item "Open", onOpen

onOpen = (er, item) ->
return cb(er) if er
item.click onClick

onClick = (er) ->
return cb(er) if er
mainWindow.getChild type("Window"), onGetWindow

onGetWindow = (er, dialog) ->
return cb(er) if er
dialog.doSomething cb(null)

aaronblohowiak

unread,
Jan 12, 2011, 4:40:15 AM1/12/11
to nodejs
Everyone: repeat this at every occasion, please. I am going to go
back and edit some of my source to do this.

Floby

unread,
Jan 12, 2011, 5:49:42 AM1/12/11
to nodejs
No, coffeescript has no named functions. It functions affected to
variables.
all your coffeescript function will come up as anonymous in a stacj
trace.

Preston Guillory

unread,
Jan 12, 2011, 8:54:04 PM1/12/11
to nod...@googlegroups.com
Sam has a point.  Asynchronous code is much larger and has a lot of syntactic noise.  Look at his example: synchronous Java does in 2 lines what takes 7 lines of asynch code, even using Seq.  Named intermediate functions don't help -- see how Isaac's example stretched to 14 lines.

I run into the same situation frequently, so I feel his pain.  My code has lots of nesting and "if (err) return callback(err)".  Any time I touch the file system, or call exec(), or query a database, or interact with the network.  And hey, that describes basically everything the code does!

It's not a problem in the sense that the approach doesn't work -- the code functions and it's fast.  But for a lot of common programming tasks, Node.js is verbose, for sure.

John Taber

unread,
Jan 12, 2011, 10:46:27 PM1/12/11
to nod...@googlegroups.com
yes, more LOC, more chance of errors and more testing required. Async
code w/ assoc. callbacks is great when apps need non-blocking, but for
many "normal" CRUD type apps where blocking isn't that much an issue,
either a simpler approach to using Node or a synchronous js server side
package (w/o the huge JVM) would make programming simpler.

Sam McCall

unread,
Jan 13, 2011, 2:39:13 AM1/13/11
to nodejs
This is by no means production ready, and so far only addresses this
very narrow case (calling methods on the 'return value' of an async
result), but I got this working:

var Plate = require('plate'); // [1]


Plate(mainWindow).menu('File').openMenu().item('Open').click().end(function(err)
{
if(err) throw err;
});

Where all the library functions are still written in callback-as-last-
parameter style.
Plate returns a proxy (hoxy?) object that records the method calls and
starts the chain when you call end().

I'll code using this for a while and see if there are other common
patterns I can't live without.

[1] https://github.com/sam-mccall/node-plate

Mihai Călin Bazon

unread,
Jan 13, 2011, 2:43:33 AM1/13/11
to nod...@googlegroups.com
Also, may I add, the biggest issue with continuation-passing style is
that it turns your code inside-out, almost like disassembling a piece
of code from a high-level language into a lower-level one. Async:

get_val("foo", function(error, foo){
if (error) print("there was an error");
else get_val("bar", function(error, bar){
if (error) print("there was an error");
else get_val("baz", function(error, baz){
if (error) print("there was an error");
else print(foo + bar + baz);
});
});
});

Sync:

try {
print(get_val("foo") + get_val("bar") + get_val("baz"));
} catch(error) {
print("there was an error");
}

The sync version is cleaner: you can see at a glance what it does. It
prints something -- the sum of 3 values returned by get_val. In case
of an error, you get "there was an error". You get the luxury of
handling the error in a single place.

The async looks like the low-level version of the sync one. If we
could disassemble a JavaScript function, it would look pretty much
like the async version (in terms of program flow). What happens is
that get_val("foo") is executed first, then get_val("bar"), then
get_val("baz"), then their sum is computed and printed -- that's
exactly what both versions do. But which one do you prefer to write?

I don't want to hurt anyone's feelings, but to me at least, Node is
unusable for serious work because of this. I'm prepared to lobby
against do-everything-asynchronously for a long time. :-)

Cheers,
-Mihai

--
Mihai Bazon,
http://mihai.bazon.net/blog

Guillermo Rauch

unread,
Jan 13, 2011, 3:00:56 AM1/13/11
to nod...@googlegroups.com
Mihai,

You can create an api that looks like

print(a,b,c)

where a b c are functions that take callbacks, or they're promises. And it's still async.

Sam McCall

unread,
Jan 13, 2011, 5:00:03 AM1/13/11
to nodejs
Mihai: a single-threaded, event-loop based execution model using non-
blocking APIs is the essential premise behind node.js.

Sure, it has issues that could be addressed nicely by using threads
instead, and I agree that with the current conventions, code can be
hard to read and debug.

But fixing this by using threads would be a solution that had nothing
in common with node, so it's not really on-topic for the node mailing
list.

Isaac Schlueter

unread,
Jan 13, 2011, 5:04:26 AM1/13/11
to nod...@googlegroups.com
On Wed, Jan 12, 2011 at 19:46, John Taber <john.tig...@gmail.com> wrote:
> a synchronous js server side package would make programming simpler.

There are many of those. Have fun. (Not being facetious.
Programmers write better programs while enjoying the process, so if
that means using Rhino or v8cgi or Ruby, more power to you.)

This conversation is a bit silly. I mean, conceptually, asynchronous
code *is* a bit more complicated.

For many problems, threads do not scale effectively. But for many
problems, they're perfectly adequate, and if you're using an
implementation that's well understood, you may even be able to work
around the Hard Problems that they introduce.

Node is cool and hot and all the tech blorgs in the blagodome are
weblooging about how awesome and fast and super insanely stupid sexy
it is. But that doesn't mean it's best for every problem, or that
JavaScript is everyone's favorite language, or that you have to love
callbacks, or that you should use it.

When you're writing network programs, your program is a proxy the vast
majority of the time. It is a glue that wires together a few
different things that *by their very nature* are most efficiently
handled with asynchronous evented message-passing. XMPP, HTTP, AMQP,
child processes, etc. Those problems are well served by an event
loop.

You can run an event loop in C. You know what? After juggling
structs and function pointers for a bit, all this "verbose" JavaScript
starts to look pretty nice.


> for many "normal" CRUD type apps where blocking isn't that much an issue...

Are you joking? That's *exactly* the sort of app where blocking very
quickly becomes a *huge* issue.

Or do you mean the kind of app where only one user is ever going to be
creating, reading, updating, and deleting data at once? Because
that's what blocking IO means.

All websites are fast with one user on localhost.


> On 01/12/2011 06:54 PM, Preston Guillory wrote:
>> Sam has a point.  Asynchronous code is much larger and has a lot of
>> syntactic noise.  Look at his example: synchronous Java does in 2 lines what
>> takes 7 lines of asynch code

Except that the 7 lines of async code *can handle other events while
waiting*, without the overhead of coroutines or threads.

And, with the Java version, there's a *ton* of stuff going on below
the surface. Actually doing real work in a higher-level-than-c
language with an event loop is kind of a newish thing. Granted, those
of us who cut our teeth on web browsers have been doing it forever,
which is part of why JS is so right for this task. But the tooling is
not super mature, and node is a lowlevel library by design.


>> Named intermediate functions
>> don't help -- see how Isaac's example stretched to 14 lines.

They reduce indentation, label blocks, and improve stack traces. They
won't improve your golf game.


>> But for a lot of common programming tasks, Node.js
>> is verbose, for sure.

The advantages are that you can do a bunch of things in parallel, and
handle "local" stuff in the pattern as "remote" stuff.

I write most of my shell scripts in Bash.


On Wed, Jan 12, 2011 at 23:39, Sam McCall <sam.m...@gmail.com> wrote:
> Plate(mainWindow).menu('File').openMenu().item('Open').click().end(function(err)
> {
>       if(err) throw err;
>    });
>
> Where all the library functions are still written in callback-as-last-
> parameter style.

See? I *told you* you could do better than my example ;D


If you can come up with a terser way to express callbacks, that
doesn't lose anything in translation, then by all means, implement it.
I think that this problem, such as it is, will be solved by either
writing a new language no one uses, or just sucking it up and using
the JavaScript we have instead of the JavaScript we wish we had.


--i

Bruno Jouhier

unread,
Jan 13, 2011, 5:22:18 AM1/13/11
to nodejs
Exactly.

This is why I wrote the little streamline.js transformation engine. I
was getting sick of turning the code inside out, not being able to use
the language's native constructs, etc. I felt like programming with a
low-level language (see http://bjouhier.wordpress.com/2011/01/09/asynchronous-javascript-the-tale-of-harry/)

Maybe some people find it "fun" to write such disassembled code. I
don't. This is just a waste of energy.

There seems to be a golden rule out there saying: "You have to solve
this problem in user-land, through DSLs, libraries, whatever, ...".
But this is just not working, or rather it always only works "up to a
point". So maybe someone has to break the rule some day, abandon this
dogma that the solution must lie in user land, write a transformation
engine, design it so that it fits the node model (you should be able
to consume node functions, and the functions that you write should in
turn be regular node functions) and hook it up into node. See
https://github.com/Sage/streamlinejs

I used a "naming" hack because I want to be able to use this today,
with today's Javascript editors (I cannot write JS code if the
"reformat entire file" function is broken). But this is not how it
should be done ultimately. What is needed is a special operator to
distinguish async function definitions and calls from sync ones. If we
had such an operator in the language, we could happily use Javascript
to mix sync and async calls.

There are probably some other solutions, like introducing true
continuations in the language (my CPS transform is probably only half-
baked) but we have to also be careful to not lose sight of what makes
Javascript so successful: its simplicity and its "mainstream-ness". So
it is probably more important to "repair" the built-in keywords of the
language so that they work in async-land the same way they have always
worked in sync-land, than to introduce more abstract features into the
language.

Also, I have the weird feeling that continuations are somewhat like
gotos (they really allow you to jump anywhere) while the
"streamlining" that I am proposing is more "structured". It will not
put the full power of continuations in the hands of programmers (so
that they don't shoot themselves in the foot) but it will give them
enough to solve their main problem (writing async code).

Bruno

Sam McCall

unread,
Jan 13, 2011, 5:35:00 AM1/13/11
to nodejs
On Jan 13, 11:04 am, Isaac Schlueter <i...@izs.me> wrote:
> See? I *told you* you could do better than my example ;D

:-) And a good learning exercise.
I'm not convinced that "write your own, it's easy" is a the right long
term answer though.
You learn a lot by implementing your own hashtable, but most of the
time you use a library.

> sucking it up and using the JavaScript we have instead of the JavaScript we wish we had

First, there's a difference between syntax and idiom.
'Functions take parameters, perform actions, and return values' is
syntax.
'Functions take a callback parameter, and call it on completion with
the error as the first argument' is idiom.
And I think it's entirely appropriate to try to find the most
expressive idioms, given the syntax we have. In public, both because
otherwise we might give up (see first post) and because having idioms
in common mean we can read each others' code.

Secondly, node.js is 'cool and hot' and could well be influential in
JS's future development.
That doesn't mean wishing for a language feature will magically make
it appear, but if people are aware of the friction points then maybe
there'll be a proposal or two when ES6 comes around.

Sam

billywhizz

unread,
Jan 13, 2011, 5:56:14 AM1/13/11
to nodejs
+100 for Isaac's excellent response.

i would also recommend anyone who is advocating adding threads/
blocking calls to node.js to have a look at this presentation by the
godfather of javascript:

http://www.yuiblog.com/blog/2010/08/30/yui-theater-douglas-crockford-crockford-on-javascript-scene-6-loopage-52-min/

if you're pressed for time, skip to approx 20 mins in where he starts
to get into the nitty-gritty of event loops v threading. mr crockford
explains it a lot better than i would be able to...

yes, async coding in an event loop is different and does have some
pain associated with it, but from where i am standing the pain is a
lot less than dealing with all the synchronization and locking issues
that threading and shared memory bring to the table...

On Jan 13, 10:04 am, Isaac Schlueter <i...@izs.me> wrote:

Preston Guillory

unread,
Jan 13, 2011, 9:25:59 AM1/13/11
to nod...@googlegroups.com
Generally every design pattern represents a missing language feature.  For instance in Java, in order to enforce constraints on an object's properties, you create setters and getters and then only access the properties through those methods.  It's extra code and not as clean, but a Java programmer only sees the necessity of it.  But C# has shown us that it was really just a missing language feature all along.

I haven't tried Bruno's solution yet, but I like direction it's taking.  Programming languages trend toward compact, expressive syntaxes.  See Ruby and Python.

Bradley Meck

unread,
Jan 13, 2011, 9:27:17 AM1/13/11
to nod...@googlegroups.com
I would be interested to know how the code bruno generates prevents concurrent editting between the synchronous looking chain or does it require semaphores?

Bruno Jouhier

unread,
Jan 13, 2011, 10:03:12 AM1/13/11
to nodejs
I don't actually generate the code myself, streamline.js does it for
me :-)

What do you mean by concurrent editting? Can you give an example?

The generated code corresponds to what you would have to write with
callbacks to reproduce the sequential chaining that is expressed in
the sequential version of the code (if this mumbo jumbo makes any
sense to you). If you want to see examples of what is generated, you
can take a look at the unit test: https://github.com/Sage/streamlinejs/blob/master/test/transform-test.js

I'm actually thinking of revisiting this a bit. The idea is that when
I write something like
var x = foo_().bar_().x + zoo_().y
it is very important to have a sequential chain for foo_().bar_().x on
one side and another one for zoo_().y but it rather counterproductive
to chain the two together to apply the plus. Instead, we should let
them execute in parallel and join the two chains before the plus.

My current transform doesn't do that but it should be relatively easy
to modify it so that it does that. With this, the programmer would get
transparent parallelization of the evaluation of sub-expressions. I
find the idea pretty cool. Actually, if I apply the same principle to
array and object literals, I get a pretty neat way to parallelize
function calls and put their results into an array (or an object):

var results = [ f1_(), f2_() ]; // this is neater than the spray/
colllectAll API that I proposed.

But then, you'll need a mechanism to control the level of parallelism.
There are places where you really want to serialize execution. This is
where the little "funnel" utility comes into play. The diskUsage2.js
shows how I use it to avoid exhaustion of file descriptors when
parallelizing a recursive traversal of directories. (https://
github.com/Sage/streamlinejs/blob/master/examples/diskUsage2.js).

So maybe the funnel is what you're asking for. If you set its max to
1, you get some kind of monitor that queues incoming calls and only
lets them go through one at a time.

Bruno.

Isaac Schlueter

unread,
Jan 13, 2011, 1:35:02 PM1/13/11
to nod...@googlegroups.com
On Thu, Jan 13, 2011 at 02:22, Bruno Jouhier <bjou...@gmail.com> wrote:
> There seems to be a golden rule out there saying: "You have to solve
> this problem in user-land, through DSLs, libraries, whatever, ...".
> But this is just not working, or rather it always only works "up to a
> point".

Really? Seems to me like it's working. Qv: this thread.

> What is needed is a special operator to
> distinguish async function definitions and calls from sync ones. If we
> had such an operator in the language, we could happily use Javascript
> to mix sync and async calls.

Sure. That's a fair point. So... go do that.

But it won't be JavaScript. It'll be some other thing. If you want
to change JavaScript-the-language, you're posting on the wrong mailing
list.

If you wanna design a new language that compiles to node-friendly JS,
you can clearly do that. Some people will love it, others will hate
it, sky's still blue, pope's still catholic, etc.

> It will not
> put the full power of continuations in the hands of programmers

You may find that programmers are more power-hungry than you can
possibly imagine. ;)


On Thu, Jan 13, 2011 at 02:35, Sam McCall <sam.m...@gmail.com> wrote:
> First, there's a difference between syntax and idiom.

Fair point.

Node's idiom is the most simple idiom possible to *enable* more
creative things in userland. It's the base. Think of the
continuation-callback style as the bytecode that your flow control
library interfaces with.

When you're interfacing with the low-level node libraries a lot,
callbacks are actually a very nice abstraction that gives you a lot of
control over what's going on. If your program is going to deal with
streams more often than single fs operations, then perhaps it makes
more sense to structure your program differently. No one's trying to
say that callbacks are best for everything.

In npm, I *have* to do a lot of lowlevel file system operations.
That's what npm is for. So, I need some abstraction around it so that
I can do a list of N things sequentially or in parallel, and compose
multiple callback-taking actors into one.

Since I have those abstractions, it makes sense to kind of shove the
stream-ish operations (writing or uploading a file, for instance) into
a function that takes a callback, so that I can say "Write this file,
create a symlink, make this directory, upload this thing, and then
signal success or failure" and wrap that up as one "operation" to be
composed in another area of my program.

The thing about closures and continuations is that, while they look
obnoxious in these little examples, once embraced fully, they allow
for a huge degree of DRYness and consistency.

> And I think it's entirely appropriate to try to find the most
> expressive idioms, given the syntax we have. In public, both because
> otherwise we might give up (see first post) and because having idioms
> in common mean we can read each others' code.

Rewind the NodeJS clock about 10 months, to when everyone was debating
how to do Promises in the most powerful and correct way.

Shooting that horse was the only way to get everyone to stop beating
it. Now we have 3 promise libraries that are all better than the one
that was in node-core.

It is important for a platform to be very careful about what it lets
in, lest it prevent innovation by the community. Ryan's extreme
aesthetic in that regard is a big part of what has made node
successful.

> if people are aware of the friction points then maybe
> there'll be a proposal or two when ES6 comes around.

I don't disagree. But I've got programs to write in the thousand
years or so before ES6 will ever be a reality.

--i

Sam McCall

unread,
Jan 13, 2011, 2:01:49 PM1/13/11
to nodejs
I've rewritten my lib so it's a bit more flexible. And I renamed it
Ship because Plate was taken (I really suck at names!)
It's not as powerful as Bruno's stuff, but it doesn't do source
parsing/transformation, it uses proxy objects instead which are
marginally less evil.

So in case it's of interest to anyone:

https://github.com/sam-mccall/ship || npm install ship

Examples:
(the APIs called are all async except where the names indicate)

function handler(err, result) { ... };

// Sequential execution:
var ps = plate(something);
ps.foo();
ps.bar();
// is the same as
something.foo(function(err) {
if(err)
handler(err);
else
something.bar(handler);
};

// Chaining:
plate(something).foo(1).bar(2).end(handler);
// is the same as
something.foo(1, function(err, data) {
if(err)
handler(err);
else
data.bar(2, handler);
});

// Inside-out evaluation:
var ps = plate(something);
ps.foo(2, ps.bar()).end(handler);
// is the same as
something.bar(function(err, data) {
if(err)
handler(err);
else
something.foo(2, data, handler);
}

// Call sync functions with $, and access attributes
plate(something).asyncOne().sync$().attrib().asyncTwo().end(handler);
// is the same as
something.asyncOne(function(err, data) {
if(err)
handler(err);
else
data.sync().attrib.asyncTwo(handler);
});

Sam McCall

unread,
Jan 13, 2011, 2:09:34 PM1/13/11
to nodejs
Isaac: all true. My only quibble is with:

> The thing about closures and continuations is that, while they look
> obnoxious in these little examples, once embraced fully, they allow
> for a huge degree of DRYness and consistency.

For some tasks it will always look obnoxious, and you don't need the
power.
I really wish we had continuations in node, this stuff would be cake.

Thanks for the context w.r.t. promises, I wasn't around then but that
does explain a lot.

Isaac Schlueter

unread,
Jan 13, 2011, 2:37:47 PM1/13/11
to nod...@googlegroups.com
On Thu, Jan 13, 2011 at 11:09, Sam McCall <sam.m...@gmail.com> wrote:
> For some tasks it will always look obnoxious, and you don't need the
> power.

Yep.

> I really wish we had continuations in node, this stuff would be cake.

The actor/callback pattern basically is CPS, though of course it's
much uglier in JS than in Scheme. There's no TCO, of course, but the
event loop's stack-destruction performs a similar effect as TCO.

The thing is that, like in Scheme, JS's continuations trap their state
in a closure, rather than a stack-saving Continuation object that
implements coroutines. It's very similar in practice to the Promise
pattern, except done in FP style rather than OOP.

--i

Sam McCall

unread,
Jan 13, 2011, 3:08:51 PM1/13/11
to nodejs
Yeah, I guess with a continuation you capture the whole stack, so you
can easily end up with similar overhead to threads. Mostly I just
meant the syntax is better - pass in a callback function when
suitable, pass in the current continuation where it makes more sense
to write sequential (but asynchronous) code.

Hmm, I don't think you need full continuations for this case, as you
would always be jumping up the stack... just the equivalent of setjmp/
longjmp. But I'll save it for Ecma3000 :-)

Sam McCall

unread,
Jan 13, 2011, 3:12:09 PM1/13/11
to nodejs
On Jan 13, 9:08 pm, Sam McCall <sam.mcc...@gmail.com> wrote:
> Hmm, I don't think you need full continuations for this case, as you
> would always be jumping up the stack... just the equivalent of setjmp/
> longjmp. But I'll save it for Ecma3000 :-)

Er, disregard that, only in the synchronous case!

Isaac

unread,
Jan 13, 2011, 9:55:29 PM1/13/11