I love async, but I can't code like this
Sam McCall
don't want to change it.
I don't feel like Javascript is desperately missing macros.
But I need someone to talk me down, because I'm coming to the
conclusion that the certain sorts of programs are unwritable without
language extensions.
Not technically unwritable, of course, they work really well. But the
code is so full of callback noise that you can't see the logic.
I have a server that speaks an HTTP/JSON based protocol, and client
libraries for Java + Node.JS. (It's for GUI automation, think Selenium
for windows apps. Open source soon, waiting on corp lawyers).
Client apps are highly sequential. They're easy to write in Java,
they're easy to write in synchronous JavaScript, and they'd be easy to
write using Bruno Jouhier's transformation library.
The consensus seems to be that threads are evil (I agree), source
transformation is evil (I agree), and that you can write anything you
want in vanilla JS today.
Well, show me how to write this in a way that's nearly as concise as
Java (surely not a high standard!).
I don't have an easy solution, using anything other than standard
JavaScript syntax + semantics is incredibly problematic.
But can we admit that there's a problem?
== Java: ==
mainWindow.menu("File").openMenu().item("Open").click();
Window dialog = mainWindow.getChild(type(Window.class));
...
== Evil synchronous JavaScript ==
mainWindow.menu('File').openMenu().item('Open').click();
var dialog = mainWindow.getChild(type('Window'));
== JavaScript: ==
mainWindow.menu("File", function(err, file) {
if(err) throw err;
file.openMenu(function(err, menu) {
if(err) throw err;
menu.item("Open", function(err, item) {
if(err) throw err;
item.click(function(err) {
if(err) throw err;
mainWindow.getChild(type('Window'), function(err, dialog) {
if(err) throw err;
...
});
});
});
});
});
== JavaScript + Seq: ==
Seq()
.seq(function() { mainWindow.menu("File", this); })
.seq(function(file) { file.openMenu(this); })
.seq(function(menu) { menu.item("Open", this); })
.seq(function(open) { open.click(); })
.seq(function(){ mainWindow.getChild(type('Window'), this); })
.seq(function(dialog){ ... });
---
Cheers,
Sam McCall
node.js newbie
Nick Husher
blocking operation? The point of Node isn't to make everything
asynchronous, the point is to note clearly and loudly where you have
places in your code where you're communicating with a slow data
source. In other words, the file system or the network.
Let's say you're communicating with a GUI over the network: that's
cool, sometimes it's necessary. I recommend writing a wrapper that
abstracts all that asynchronous business into a set of commands that
can be queued up and run, since it's only the last callback you're
really interested in.
> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>
Guillermo Rauch
Sam McCall
> You're just using a blocking API. You can do the same thing with Node.JS
Guillermo Rauch
browser
.chain
.session()
.open('/')
.assertTitle('Something')
.and(login('foo', 'bar'))
.assertTitle('Foobar')
.and(login('someone', 'else'))
.assertTitle('Someone else')
.end(function(err){
if (err) throw err;
});Sam McCall
> Is interacting with (what appears to be) a GUI menu actually a
> blocking operation?
on machine A, if we call openMenu on a menu item then we can't do
anything[1] until we get the HTTP response. But of course we can do
work on another session targeted at another machine.
[1] Actually we can, but whether the menu is opened/closed/midway
isn't defined, so we can't usually do anything useful.
> Let's say you're communicating with a GUI over the network: that's
> cool, sometimes it's necessary.
> I recommend writing a wrapper that
> abstracts all that asynchronous business into a set of commands that
> can be queued up and run, since it's only the last callback you're
> really interested in.
window.clickMenuItem("File", "Open", cb);
Unfortunately, I chose it because it was universal/familiar, in
practice the code snippets tend to be application specific because the
UIAutomation tree (the Windows equivalent of the DOM) tends to vary in
unintuitive per-application ways.
Therefore the wrappers couldn't be part of the library, they'd be
written by whoever uses the library, for every app-specific task they
want to do. So you're writing the same noisy code, but in a function.
The JavaScript client is experimental, but we've used the Java client
to write automated tests for in-house software and in practice there's
little to no redundancy in the API - a higher level API would stop you
doing things you need to be able to do.
On Jan 11, 8:18 pm, Guillermo Rauch <rau...@gmail.com> wrote:
> If you're writing a desktop GUI app, you have two options:
> - Blocking API + web workers
overhead and sync problems of threads just to expose an API you can
write readable programs in. (Plus it doesn't work in Node yet,
right?)
> - Non-blocking and some API that simplifies the nesting and error handling.
Our API is object-oriented (very similar to WebDriver) so we need to
deal with different object handles rather than just a single Selenium
connection, and there's sometimes complex flow-control mid-chain, but
I'll see if I can get something that works reasonably. (And ideally
that separates cleanly from both the app code and the library, so both
are readable). At this point i'm starting to disagree with people who
think everyone should write their own!
Cheers,
Sam
Bruno Jouhier
I wrote streamline.js for people like you and me, people who like
Javascript, who like node's event model and who are involved in
industrial projects, which means that they will, at one point or
another, need to involve "mainstream" programmers into it.
Mainstream programmers understand Javascript's flow control
statements. So I think that they'll understand streamline.js. I don't
think that they will be ever as productive and as comfortable with a
DSL however smart the DSL might be. Why would you need a DSL anyway
when all you're asking for is basic flow control statements that
actually exist in the language but got broken somehow. What you want
is that someone fixes the language so that you can use for loops, try/
catch, dot notation again, not that someone introduces new
abstractions that will never be as concise as the language's
primitives, that will have a learning curve and that will be obsolete
the next month because a newer abstraction will have taken the
spotlights.
I released streamline.js under MIT license which is not our standard
corporate license so that the community can feel relaxed about it (I'm
probably more fortunate than you because I have a kind CTO who
expedited things with the lawyers). So it is available and free and we
can start using it.
I am not particularly fond of preprocessors and I went this route
simply because I did not see any way to fix the problem in user-land
(I tried to write my own DSL like everyone else of course but I felt
like loosing my time). That's all. There is no ideological stand in
this, just pragmaticism.
So even if code transformation is evil, we may have to sympathize with
the devil, and write rock'n roll code. At least that's what I'm doing
from this point on. And maybe some day, the fix will get into the
language for real. After all, all it needs is a small operator or
keyword (to replace the underscore hack) and a relatively simple
algebraic code transfomation that can be pipelined into the other
stages of the compiler.
Bruno
Liam
entities, and returning only the result you need for a particular
sequence...
Isn't there a serialization format for UI automation sequences that
allows them to be saved and played back?
Isaac Schlueter
of the excluded middle. No one actually writes code like this. The
options are not "73 layers of indentation, or threads."
The language includes named function declarations. Use them. That
alone would make your example way less terrible.
--i
Bradley Meck
Sam McCall
it.
I really don't like the idea of a preprocessor, with and the
associated deployment hassles and ambiguous semantics. So if it can be
done with something like a proxy object that records a chain of method
calls synchronously, then plays them back asynchronously, I'd prefer
that. But if it isn't manageable, then pragmatically I think it'll be
the best option. (Your implementation looks really good, it's a purely
philosophical objection).
Liam: we haven't defined such a format. The actions are functions with
complex object parameters and many different types of return values
that can raise errors at any step etc. This would basically mean
defining a new scripting language with synchronous semantics and
writing an asynchronous interpreter for it. It seems strange to have
to do this.
Isaac: there are 5 inline functions declared in the example.
'var x = function(){};' 5 times is a lot of code even before you do
anything in them, it can never be concise or really readable.
Besides, I'm not really sure what to name them as they don't really
have much independent behaviour, they're just glue.
But I'd be happy to be convinced, could you show me how you'd write it?
Liam
style programming...
But if you don't need tons of concurrent script "threads", perhaps
write sync JS scripts and use a pool of Node processes consuming a
queue of scripts?
Isaac Schlueter
> Isaac: there are 5 inline functions declared in the example.
> 'var x = function(){};' 5 times is a lot of code even before you do
> anything in them, it can never be concise or really readable.
> Besides, I'm not really sure what to name them as they don't really
> have much independent behaviour, they're just glue.
> But I'd be happy to be convinced, could you show me how you'd write it?
Cut out the "var" and the "=". But yeah, "function" is long and
klunky for how common it is. I'd love a shorter keyword for it.
Here's a very brief untested example of what I'm talking about:
https://gist.github.com/775591
I usually only care about success or failure, so I don't use any kind
of "passalong" util in my programs. But it's not hard to see how this
general technique could be applied in other ways.
--i
shaun etherton
IMO this is a great example; especially for new comers. It makes the
strange looking yet common function nesting look more familiar, then
progresses to show the awesomeness of JS in a simple, clear and
understandable way.
Sometimes i think those who are intimately familiar with the nested
callbacks/anon function style may not realize the amount of angst and
confusion it can cause for the uninitiated.
- shaun
mgutz
# using named functions
doSomething = (done) ->
mainWindow.menu "File", onFile
onFile = (er, file) ->
return cb(er) if er
file.openMenu onMenu
onMenu = (er, menu) ->
return er if er
menu.item "Open", onOpen
onOpen = (er, item) ->
return er if er
item.click onClick
onClick = (er) ->
return er if er
mainWindow.getChild type("Window"), onGetWindow
onGetWindow = (er, dialog) ->
return er if er
dialog.doSomething done
mgutz
# using named functions
function doSomething (cb) ->
onFile = (er, file) ->
return cb(er) if er
file.openMenu onMenu
onMenu = (er, menu) ->
onOpen = (er, item) ->
onClick = (er) ->
onGetWindow = (er, dialog) ->
dialog.doSomething cb(null)
aaronblohowiak
back and edit some of my source to do this.
Floby
variables.
all your coffeescript function will come up as anonymous in a stacj
trace.
Preston Guillory
I run into the same situation frequently, so I feel his pain. My code has lots of nesting and "if (err) return callback(err)". Any time I touch the file system, or call exec(), or query a database, or interact with the network. And hey, that describes basically everything the code does!
It's not a problem in the sense that the approach doesn't work -- the code functions and it's fast. But for a lot of common programming tasks, Node.js is verbose, for sure.
John Taber
code w/ assoc. callbacks is great when apps need non-blocking, but for
many "normal" CRUD type apps where blocking isn't that much an issue,
either a simpler approach to using Node or a synchronous js server side
package (w/o the huge JVM) would make programming simpler.
Sam McCall
very narrow case (calling methods on the 'return value' of an async
result), but I got this working:
var Plate = require('plate'); // [1]
Plate(mainWindow).menu('File').openMenu().item('Open').click().end(function(err)
{
if(err) throw err;
});
Where all the library functions are still written in callback-as-last-
parameter style.
Plate returns a proxy (hoxy?) object that records the method calls and
starts the chain when you call end().
I'll code using this for a while and see if there are other common
patterns I can't live without.
[1] https://github.com/sam-mccall/node-plate
Mihai Călin Bazon
that it turns your code inside-out, almost like disassembling a piece
of code from a high-level language into a lower-level one. Async:
get_val("foo", function(error, foo){
if (error) print("there was an error");
else get_val("bar", function(error, bar){
if (error) print("there was an error");
else get_val("baz", function(error, baz){
if (error) print("there was an error");
else print(foo + bar + baz);
});
});
});
Sync:
try {
print(get_val("foo") + get_val("bar") + get_val("baz"));
} catch(error) {
print("there was an error");
}
The sync version is cleaner: you can see at a glance what it does. It
prints something -- the sum of 3 values returned by get_val. In case
of an error, you get "there was an error". You get the luxury of
handling the error in a single place.
The async looks like the low-level version of the sync one. If we
could disassemble a JavaScript function, it would look pretty much
like the async version (in terms of program flow). What happens is
that get_val("foo") is executed first, then get_val("bar"), then
get_val("baz"), then their sum is computed and printed -- that's
exactly what both versions do. But which one do you prefer to write?
I don't want to hurt anyone's feelings, but to me at least, Node is
unusable for serious work because of this. I'm prepared to lobby
against do-everything-asynchronously for a long time. :-)
Cheers,
-Mihai
--
Mihai Bazon,
http://mihai.bazon.net/blog
Guillermo Rauch
print(a,b,c)
Sam McCall
blocking APIs is the essential premise behind node.js.
Sure, it has issues that could be addressed nicely by using threads
instead, and I agree that with the current conventions, code can be
hard to read and debug.
But fixing this by using threads would be a solution that had nothing
in common with node, so it's not really on-topic for the node mailing
list.
Isaac Schlueter
> a synchronous js server side package would make programming simpler.
There are many of those. Have fun. (Not being facetious.
Programmers write better programs while enjoying the process, so if
that means using Rhino or v8cgi or Ruby, more power to you.)
This conversation is a bit silly. I mean, conceptually, asynchronous
code *is* a bit more complicated.
For many problems, threads do not scale effectively. But for many
problems, they're perfectly adequate, and if you're using an
implementation that's well understood, you may even be able to work
around the Hard Problems that they introduce.
Node is cool and hot and all the tech blorgs in the blagodome are
weblooging about how awesome and fast and super insanely stupid sexy
it is. But that doesn't mean it's best for every problem, or that
JavaScript is everyone's favorite language, or that you have to love
callbacks, or that you should use it.
When you're writing network programs, your program is a proxy the vast
majority of the time. It is a glue that wires together a few
different things that *by their very nature* are most efficiently
handled with asynchronous evented message-passing. XMPP, HTTP, AMQP,
child processes, etc. Those problems are well served by an event
loop.
You can run an event loop in C. You know what? After juggling
structs and function pointers for a bit, all this "verbose" JavaScript
starts to look pretty nice.
> for many "normal" CRUD type apps where blocking isn't that much an issue...
Are you joking? That's *exactly* the sort of app where blocking very
quickly becomes a *huge* issue.
Or do you mean the kind of app where only one user is ever going to be
creating, reading, updating, and deleting data at once? Because
that's what blocking IO means.
All websites are fast with one user on localhost.
> On 01/12/2011 06:54 PM, Preston Guillory wrote:
>> Sam has a point. Asynchronous code is much larger and has a lot of
>> syntactic noise. Look at his example: synchronous Java does in 2 lines what
>> takes 7 lines of asynch code
Except that the 7 lines of async code *can handle other events while
waiting*, without the overhead of coroutines or threads.
And, with the Java version, there's a *ton* of stuff going on below
the surface. Actually doing real work in a higher-level-than-c
language with an event loop is kind of a newish thing. Granted, those
of us who cut our teeth on web browsers have been doing it forever,
which is part of why JS is so right for this task. But the tooling is
not super mature, and node is a lowlevel library by design.
>> Named intermediate functions
>> don't help -- see how Isaac's example stretched to 14 lines.
They reduce indentation, label blocks, and improve stack traces. They
won't improve your golf game.
>> But for a lot of common programming tasks, Node.js
>> is verbose, for sure.
The advantages are that you can do a bunch of things in parallel, and
handle "local" stuff in the pattern as "remote" stuff.
I write most of my shell scripts in Bash.
On Wed, Jan 12, 2011 at 23:39, Sam McCall <sam.m...@gmail.com> wrote:
> Plate(mainWindow).menu('File').openMenu().item('Open').click().end(function(err)
> {
> if(err) throw err;
> });
>
> Where all the library functions are still written in callback-as-last-
> parameter style.
See? I *told you* you could do better than my example ;D
If you can come up with a terser way to express callbacks, that
doesn't lose anything in translation, then by all means, implement it.
I think that this problem, such as it is, will be solved by either
writing a new language no one uses, or just sucking it up and using
the JavaScript we have instead of the JavaScript we wish we had.
--i
Bruno Jouhier
This is why I wrote the little streamline.js transformation engine. I
was getting sick of turning the code inside out, not being able to use
the language's native constructs, etc. I felt like programming with a
low-level language (see http://bjouhier.wordpress.com/2011/01/09/asynchronous-javascript-the-tale-of-harry/)
Maybe some people find it "fun" to write such disassembled code. I
don't. This is just a waste of energy.
There seems to be a golden rule out there saying: "You have to solve
this problem in user-land, through DSLs, libraries, whatever, ...".
But this is just not working, or rather it always only works "up to a
point". So maybe someone has to break the rule some day, abandon this
dogma that the solution must lie in user land, write a transformation
engine, design it so that it fits the node model (you should be able
to consume node functions, and the functions that you write should in
turn be regular node functions) and hook it up into node. See
https://github.com/Sage/streamlinejs
I used a "naming" hack because I want to be able to use this today,
with today's Javascript editors (I cannot write JS code if the
"reformat entire file" function is broken). But this is not how it
should be done ultimately. What is needed is a special operator to
distinguish async function definitions and calls from sync ones. If we
had such an operator in the language, we could happily use Javascript
to mix sync and async calls.
There are probably some other solutions, like introducing true
continuations in the language (my CPS transform is probably only half-
baked) but we have to also be careful to not lose sight of what makes
Javascript so successful: its simplicity and its "mainstream-ness". So
it is probably more important to "repair" the built-in keywords of the
language so that they work in async-land the same way they have always
worked in sync-land, than to introduce more abstract features into the
language.
Also, I have the weird feeling that continuations are somewhat like
gotos (they really allow you to jump anywhere) while the
"streamlining" that I am proposing is more "structured". It will not
put the full power of continuations in the hands of programmers (so
that they don't shoot themselves in the foot) but it will give them
enough to solve their main problem (writing async code).
Bruno
Sam McCall
> See? I *told you* you could do better than my example ;D
I'm not convinced that "write your own, it's easy" is a the right long
term answer though.
You learn a lot by implementing your own hashtable, but most of the
time you use a library.
> sucking it up and using the JavaScript we have instead of the JavaScript we wish we had
'Functions take parameters, perform actions, and return values' is
syntax.
'Functions take a callback parameter, and call it on completion with
the error as the first argument' is idiom.
And I think it's entirely appropriate to try to find the most
expressive idioms, given the syntax we have. In public, both because
otherwise we might give up (see first post) and because having idioms
in common mean we can read each others' code.
Secondly, node.js is 'cool and hot' and could well be influential in
JS's future development.
That doesn't mean wishing for a language feature will magically make
it appear, but if people are aware of the friction points then maybe
there'll be a proposal or two when ES6 comes around.
Sam
billywhizz
i would also recommend anyone who is advocating adding threads/
blocking calls to node.js to have a look at this presentation by the
godfather of javascript:
http://www.yuiblog.com/blog/2010/08/30/yui-theater-douglas-crockford-crockford-on-javascript-scene-6-loopage-52-min/
if you're pressed for time, skip to approx 20 mins in where he starts
to get into the nitty-gritty of event loops v threading. mr crockford
explains it a lot better than i would be able to...
yes, async coding in an event loop is different and does have some
pain associated with it, but from where i am standing the pain is a
lot less than dealing with all the synchronization and locking issues
that threading and shared memory bring to the table...
On Jan 13, 10:04 am, Isaac Schlueter <i...@izs.me> wrote:
Preston Guillory
I haven't tried Bruno's solution yet, but I like direction it's taking. Programming languages trend toward compact, expressive syntaxes. See Ruby and Python.
Bradley Meck
Bruno Jouhier
me :-)
What do you mean by concurrent editting? Can you give an example?
The generated code corresponds to what you would have to write with
callbacks to reproduce the sequential chaining that is expressed in
the sequential version of the code (if this mumbo jumbo makes any
sense to you). If you want to see examples of what is generated, you
can take a look at the unit test: https://github.com/Sage/streamlinejs/blob/master/test/transform-test.js
I'm actually thinking of revisiting this a bit. The idea is that when
I write something like
var x = foo_().bar_().x + zoo_().y
it is very important to have a sequential chain for foo_().bar_().x on
one side and another one for zoo_().y but it rather counterproductive
to chain the two together to apply the plus. Instead, we should let
them execute in parallel and join the two chains before the plus.
My current transform doesn't do that but it should be relatively easy
to modify it so that it does that. With this, the programmer would get
transparent parallelization of the evaluation of sub-expressions. I
find the idea pretty cool. Actually, if I apply the same principle to
array and object literals, I get a pretty neat way to parallelize
function calls and put their results into an array (or an object):
var results = [ f1_(), f2_() ]; // this is neater than the spray/
colllectAll API that I proposed.
But then, you'll need a mechanism to control the level of parallelism.
There are places where you really want to serialize execution. This is
where the little "funnel" utility comes into play. The diskUsage2.js
shows how I use it to avoid exhaustion of file descriptors when
parallelizing a recursive traversal of directories. (https://
github.com/Sage/streamlinejs/blob/master/examples/diskUsage2.js).
So maybe the funnel is what you're asking for. If you set its max to
1, you get some kind of monitor that queues incoming calls and only
lets them go through one at a time.
Bruno.
Isaac Schlueter
> There seems to be a golden rule out there saying: "You have to solve
> this problem in user-land, through DSLs, libraries, whatever, ...".
> But this is just not working, or rather it always only works "up to a
> point".
Really? Seems to me like it's working. Qv: this thread.
> What is needed is a special operator to
> distinguish async function definitions and calls from sync ones. If we
> had such an operator in the language, we could happily use Javascript
> to mix sync and async calls.
Sure. That's a fair point. So... go do that.
But it won't be JavaScript. It'll be some other thing. If you want
to change JavaScript-the-language, you're posting on the wrong mailing
list.
If you wanna design a new language that compiles to node-friendly JS,
you can clearly do that. Some people will love it, others will hate
it, sky's still blue, pope's still catholic, etc.
> It will not
> put the full power of continuations in the hands of programmers
You may find that programmers are more power-hungry than you can
possibly imagine. ;)
On Thu, Jan 13, 2011 at 02:35, Sam McCall <sam.m...@gmail.com> wrote:
> First, there's a difference between syntax and idiom.
Fair point.
Node's idiom is the most simple idiom possible to *enable* more
creative things in userland. It's the base. Think of the
continuation-callback style as the bytecode that your flow control
library interfaces with.
When you're interfacing with the low-level node libraries a lot,
callbacks are actually a very nice abstraction that gives you a lot of
control over what's going on. If your program is going to deal with
streams more often than single fs operations, then perhaps it makes
more sense to structure your program differently. No one's trying to
say that callbacks are best for everything.
In npm, I *have* to do a lot of lowlevel file system operations.
That's what npm is for. So, I need some abstraction around it so that
I can do a list of N things sequentially or in parallel, and compose
multiple callback-taking actors into one.
Since I have those abstractions, it makes sense to kind of shove the
stream-ish operations (writing or uploading a file, for instance) into
a function that takes a callback, so that I can say "Write this file,
create a symlink, make this directory, upload this thing, and then
signal success or failure" and wrap that up as one "operation" to be
composed in another area of my program.
The thing about closures and continuations is that, while they look
obnoxious in these little examples, once embraced fully, they allow
for a huge degree of DRYness and consistency.
> And I think it's entirely appropriate to try to find the most
> expressive idioms, given the syntax we have. In public, both because
> otherwise we might give up (see first post) and because having idioms
> in common mean we can read each others' code.
Rewind the NodeJS clock about 10 months, to when everyone was debating
how to do Promises in the most powerful and correct way.
Shooting that horse was the only way to get everyone to stop beating
it. Now we have 3 promise libraries that are all better than the one
that was in node-core.
It is important for a platform to be very careful about what it lets
in, lest it prevent innovation by the community. Ryan's extreme
aesthetic in that regard is a big part of what has made node
successful.
> if people are aware of the friction points then maybe
> there'll be a proposal or two when ES6 comes around.
I don't disagree. But I've got programs to write in the thousand
years or so before ES6 will ever be a reality.
--i
Sam McCall
Ship because Plate was taken (I really suck at names!)
It's not as powerful as Bruno's stuff, but it doesn't do source
parsing/transformation, it uses proxy objects instead which are
marginally less evil.
So in case it's of interest to anyone:
https://github.com/sam-mccall/ship || npm install ship
Examples:
(the APIs called are all async except where the names indicate)
function handler(err, result) { ... };
// Sequential execution:
var ps = plate(something);
ps.foo();
ps.bar();
// is the same as
something.foo(function(err) {
if(err)
handler(err);
else
something.bar(handler);
};
// Chaining:
plate(something).foo(1).bar(2).end(handler);
// is the same as
something.foo(1, function(err, data) {
if(err)
handler(err);
else
data.bar(2, handler);
});
// Inside-out evaluation:
var ps = plate(something);
ps.foo(2, ps.bar()).end(handler);
// is the same as
something.bar(function(err, data) {
if(err)
handler(err);
else
something.foo(2, data, handler);
}
// Call sync functions with $, and access attributes
plate(something).asyncOne().sync$().attrib().asyncTwo().end(handler);
// is the same as
something.asyncOne(function(err, data) {
if(err)
handler(err);
else
data.sync().attrib.asyncTwo(handler);
});
Sam McCall
> The thing about closures and continuations is that, while they look
> obnoxious in these little examples, once embraced fully, they allow
> for a huge degree of DRYness and consistency.
power.
I really wish we had continuations in node, this stuff would be cake.
Thanks for the context w.r.t. promises, I wasn't around then but that
does explain a lot.
Isaac Schlueter
> For some tasks it will always look obnoxious, and you don't need the
> power.
Yep.
> I really wish we had continuations in node, this stuff would be cake.
The actor/callback pattern basically is CPS, though of course it's
much uglier in JS than in Scheme. There's no TCO, of course, but the
event loop's stack-destruction performs a similar effect as TCO.
The thing is that, like in Scheme, JS's continuations trap their state
in a closure, rather than a stack-saving Continuation object that
implements coroutines. It's very similar in practice to the Promise
pattern, except done in FP style rather than OOP.
--i
Sam McCall
can easily end up with similar overhead to threads. Mostly I just
meant the syntax is better - pass in a callback function when
suitable, pass in the current continuation where it makes more sense
to write sequential (but asynchronous) code.
Hmm, I don't think you need full continuations for this case, as you
would always be jumping up the stack... just the equivalent of setjmp/
longjmp. But I'll save it for Ecma3000 :-)
Sam McCall
> Hmm, I don't think you need full continuations for this case, as you
> would always be jumping up the stack... just the equivalent of setjmp/
> longjmp. But I'll save it for Ecma3000 :-)
Isaac
I'm sure we're all well aware how GOTO turned out...
Anyway user land is exactly the right place for this sort of thing.
Why ? Because there are a whole bunch of people, including myself, who
like Node.js just the way it is in this regard.
Its great to be discussing this stuff and trying out solutions =)
Just please don't try to mess up the core JavaScript language by
adding an operator or whatever else because of a personal style
preference. Keep it in user land so other people at least have a
choice to continue happily passing around anonymous functions in async
styles.
Another reason to stay in user land is for code portability esp with
browsers for client-server code sharing. Unless your planning to have
an operator added to the EMCAScript standard for this and even then it
would take years for browser vendors to implement it.
Cheers.
Mikeal Rogers
The place for that discussion is the es-discuss list, the public facing part of TC-39 (the ECMAScript committee).
Bruno Jouhier
> adding an operator or whatever else because of a personal style
> preference. Keep it in user land so other people at least have a
> choice to continue happily passing around anonymous functions in async
> styles.
pass around anonymous functions in async style, as you say, or they
can use the new way. They can actually mix and match between the old
and new way at their will . The precompiler will *not* do anything to
their "old way" code. It only transforms functions that are defined
the new way and it will transform them to create a function that has
the classical (old way) signature (so it can be called seamlessly by
old style code).
The only problem is that you cannot do this if you stay in user land,
you have to transform the code. The key is to have a transformation
that works locally and stops at function boundary, not something that
is going to turn the entire code upside down.
Alen Mujezinovic
Havoc Pennington
http://blog.ometer.com/2010/11/28/a-sequential-actor-like-api-for-server-side-javascript/
but it requires yield (generators) in the JS implementation.
Havoc
Nick
On Jan 14, 2:58 am, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
> FYI, for those interested. Brendan is proposing that ECMA Harmony's new function keyword, #, actually return a new implementation of a function object that, among other things, will do proper tail calls, which would allow for some more of these ideas.
> The place for that discussion is the es-discuss list, the public facing part of TC-39 (the ECMAScript committee).
think this qualifies as a very good thing. Personally I like the "Ruby-
style" of [].forEach({|x| x+1 }) more.
Although it's worth noting that there's also some noise advocating an
AST API, which would allow you to solve a lot of the ugly christmas-
tree callback stuff by writing a simple translator to do what you need.
Mikeal Rogers
Isaac Schlueter
Node is the present of JavaScript.
--i
On Fri, Jan 14, 2011 at 15:52, Mikeal Rogers <mikeal...@gmail.com> wrote:
> you should tell that the es-discuss list :)
>
Felix Geisend?rfer
== Java: ==
mainWindow.menu("File").openMenu().item("Open").click();
Window dialog = mainWindow.getChild(type(Window.class));
// test goes here
Sam McCall
Re: combinatoric explosion - yeah, it's not a method that will ever
give you exhaustive testing of every codepath.
But it does let you smoke test the common paths through the app,
generate screenshots for 10 languages, on a two week release cycle,
without access to the source, automatically. (i.e. it may not be
perfect but it's a hell of a lot better than what we had before).
Re: law of demeter - I've heard of it, but its correlation with useful
code is zero, to be generous. It assumes the OO metaphor is the most
important part of your code, rather than just a language feature that
makes certain patterns clearer.
Especially in this case - the intention of a GUI automation library is
not to abstract the GUI away from the client code. A 'menu item'
object represents an OS MenuItem widget, nothing more or less, and the
client library should provide low-level, not high-level operations.
Even openMenu() is debatable, in Notepad++ the 'close document' button
is actually a MenuItem labeled 'X'.
Sure, the client code will probably want some wrappers for things that
are common in their particular target application, but they'll be
implemented as above.
In any case, if
a.b().c()
is a violation, then
a.b(function(x) {
x.c();
});
certainly is also. (The rule claims to be about encapsulation and
coupling, not syntax, and the two are clearly the equivalent sync/
async code).
Granted, the short version of demeter is 'two dots' and node makes it
hard to use 'two dots' for unrelated reasons.
However:
var Thing = require('thing');
Thing.whatever();
also violates the law of demeter.
Cheers,
Sam
Ryan Dahl
Coroutines are possible with V8. That is, you can longjmp into another
stack without disrupting the VM - there are several implementations on
the internet. Node is not going to do this because multiple call
stacks dramatically complicate the mental model of how the system
works. Furthermore it's a hard deviation from JavaScript - browser
programmers have never had to consider multiple execution stacks - it
would confuse them to subtly yet significantly change the language.
Adding multiple stacks is not a seamless abstraction. Users must know
worry that at any function call they may be yielded out and end up in
an entirely different state. Locking is now required in many
situations. Knowing if a function is reentrant or not is required.
Documentation about the "coroutine safety" of methods is required.
Node offers a powerful guarantee: when you call a function the state
of the program will only be modified by that function and its
descendants.
Havoc Pennington
On Sun, Jan 16, 2011 at 4:07 AM, Ryan Dahl <r...@tinyclouds.org> wrote:
> Adding multiple stacks is not a seamless abstraction. Users must know
> worry that at any function call they may be yielded out and end up in
> an entirely different state. Locking is now required in many
> situations. Knowing if a function is reentrant or not is required.
I sympathize with what you're saying and with node's design. There are
real tradeoffs in simplicity and also in overhead. My first reaction
to this was also, "it's better to be explicit and type in your
callbacks."
That said:
- I'm not sure all the points you mentioned about coroutines apply.
JavaScript generators are more controlled.
- I'd love to try something built on node or like node that goes
slightly differently, even if node itself remains with the tradeoffs
it has. I think the way node works is good, but it's not the only
useful way.
It sort of depends on what kind of developer and what kind of task.
- Raw-callback code is truly rocky in many instances. Chains of
callbacks where async result C depends on async result B depends on
async result A, are the worst. And trying to ensure that the outermost
callback always gets called, exactly once, with only one of either an
error or a valid result. It's not writable or readable code.
Some details that I think matter:
1. Generators here are just an implementation detail of promises. They
don't "leak" upward ... i.e. I don't think a caller would ever care
that somewhere below it in the call stack, someone used "yield" - it
should not be distinguishable from someone creating a promise in
another way.
See fromGenerator() here, which is just a promise factory:
http://git.gnome.org/browse/gjs/tree/modules/promise.js#n317
you mentioned:
> Node offers a powerful guarantee: when you call a function the state
> of the program will only be modified by that function and its
> descendants.
fwiw, I don't think generators and yield modify this guarantee. From
"inside" the generator during a yield, yes program state can be
changed by the caller; but from "outside" the generator, the generator
is just a function. Anytime program state can change "under you,"
you're going to be seeing the yield keyword, and you're going to be
implementing a generator yourself.
2. Promises in this code are just callback-holders. That is, if you
have an API that takes a callback(result, error) then you can
trivially wrap it to return a promise:
function asyncThingWithPromise() {
var p = new Promise();
asyncThingWithCallback(function(result, error) { if (error)
p.putError(error); else p.putReturn(result); })
return p;
}
And vice versa:
function asyncThingWithCallback(cb) {
asyncThingWithPromise.get(cb);
}
3. The way I'd see APIs and libraries in this world would be
promise-based, i.e. an async routine returns a promise. Given
asyncThing() that returns a promise, you can set up a callback:
asyncThing().get(function(result, error) { });
Or if you're a generator, you can kick the promise up to your caller,
save your continuation, and either resume or throw when the result
arrives:
result = yield asyncThing();
"yield" only goes up one frame though, it just makes an iterator from
the continuation and returns it, it doesn't unwind the stack or allow
arbitrary code to run. The caller (or some caller of that caller)
still has to set up a callback on the promise. The callback would
resume the continuation.
4. I could be wrong but I don't think JS generators in SpiderMonkey
get longjmp() involved. The yield keyword saves a continuation, but
the generator function then returns "normally" returning an iterator
object. The calling function has to take that iterator object (which
is implemented with a saved continuation) and if it chooses, it can
further unwind the stack by itself yielding or returning, or it
doesn't have to. Anyway there's not really much magic. Continuations
are an implementation detail of iterators.
Another way to put it, is that at each re-entrancy point you're going
to see either the "yield" keyword (push the promise to the caller, by
returning a promise-generating iterator) or you're going to see a
callback (add a callback to the promise).
It isn't like exceptions or threads where you can be interrupted at
any time. Interruptions only happen when you unwind all the way back
to the event loop which then invokes a callback.
5. The flip side of this is that "yield" isn't magic; people have to
understand how it relates to promises, what a generator function is,
etc. Pasting yield-using code from one function to another might not
work if the destination function isn't a generator.
6. I piled several potentially separable but kinda-related things into
the blog post I linked to. One is the "yield" convention. Others
include:
- have "actors" which have their own isolated JS global object, so no
state shared with other actors.
- share the event loop among actors such that callbacks from
different actors can run concurrently but those from the same actor
can't
(i.e. from perspective of each distinct JS global object, there's
only one thread, from perspective of the overall runtime, there are
multiple - but not thread-per-request! the thread pool runs libev
callbacks)
(actors think they each have their own event loop, but it's
implemented with a shared one for efficiency)
- have a framework where each http request gets its own actor
(therefore global object) ... (something that's only efficient with
SpiderMonkey in their most bleeding edge stuff, and I have no idea
about any other JS implementation. and for sure it would always have
_some_ overhead. I haven't measured it.)
- I'd expect state shared among http requests to be handled as in
other actor frameworks, for example you might start an actor which
holds your shared state, and then http request handlers could use
async message passing to ask to set/get the state, just as if they
were talking to another process.
7. In addition to using all cores without shared state or locking, an
actors approach could really help with the re-entrancy dangers of both
callbacks AND a "yield promise" syntax. The reason is that each actor
is isolated, so you don't care that the event loop is running handlers
in *other* actors. The only dangerous re-entrancy (unexpected callback
mixing) comes from callbacks in the actor you're writing right now.
One way to think about actors is that each one works like its own
little server process, even though implementation-wise they are much
lighter-weight.
8. So the idea is Akka-style functionality, and ALSO libev, and ALSO
ability to write in JavaScript, and ALSO wire it up nicely to http
with actor-per-request ;-) and a pony of course
Anyhow, I realize there are lots of practical problems with this from
a node project perspective. And that it's harder to understand than
the browser-like model.
At the same time, perhaps it remains simpler than using actors with
Java, Scala, or Erlang and preserves other advantages of node. Once
you're doing complex things, sometimes it can be simpler to have a
little more power in the framework taking care of hard problems on
your behalf.
Just some food for thought is all since it was sort of an in-the-weeds
thread to begin with. I don't intend it as a highly-unrealistic
feature request ;-) don't worry.
I am hoping there are people out there who explore these kind of
directions sometime.
Havoc
Mikeal Rogers
Furthermore, those implementations introduce a new keyword and are breaking changes to the language so considering them for use in node is incredibly premature.
v8 has no publicly stated plans to support generators.
I've worked with generators when I worked in the Mozilla platform. They suck. They tend to make code more complicated and less obvious, all for a perceiving performance improvement that isn't true with modern function optimizations. Python needs generators because function overhead is absurdly high, JavaScript doesn't.
There isn't a task you can't accomplish because you don't have generators except sugar.
Pau
Bruno Jouhier
There are a lot of proposals to improve the situation but most of them
rely on the introduction of new mechanisms into the language
(generators, coroutines, continuations). While these mechanisms may
have their own virtues (and also their own pitfalls because they give
the programmer new ways to "shoot himself in the foot"), I think that
the problem can be solved without adding such extra power to the
language.
Programming with node.js feels a bit like programming with a
schizophrenic language. Your code has two territories: sync-land and
async-land.
In sync-land you can use all the basic features of the language
(nested function calls, subexpressions that call functions, if/else
followed by common code, while loops, try/catch/finally, etc.). The
code is aligned on the way you naturally express your algorithms (at
least if you are a mainstream programmer) and there is no extra
"noise".
In async-land, things are very different because you have to deal
explicitly with the callbacks. They impose a new structure on your
code. Often it feels like you have to turn the code inside out: you
have to move all the async calls that would naturally be nested into
subexpressions before the statement that contains them. You also have
to rethink your flow control: turn iterative constructs into recursive
ones, find tricks to re-package common code after if/then branches,
etc. The simple patterns that worked so well in sync-land don't work
any more; you have to learn new, much heavier, patterns. You feel (or
at least I felt) like you are not working with a high-level language
any more but instead with some kind of strange "intermediate language"
that forces you into a complex translation effort: your still think
about your algorithms in terms of nested function calls, if/else
branches, while loops, etc. but you cannot write them this way; you
have to transform the code to cope with what callbacks impose on you.
What I am proposing with streamline.js is to fix this so that you
don't have two languages but one, and so that you can use the same
"mental model" in sync-land and async-land. I had to introduce a new
operator for this (the underscore prefix - I actually hacked it with a
naming convention to avoid problems with today's tools) and I can
understand that the community is worried to see a new operator
introduced in the language. But this operator is only a catalyst. It
does not do much by itself. It does something much more interesting:
it re-enables all the existing language features (nested calls, if/
else branches, while loops, try/catch/finally) in async-land.
The idea is that the programmer should not be required to learn new
libraries or abstractions in order to write async code. He should be
able to use the simple mechanisms that he has been using all along in
sync-land and the compiler should take care of translating his code to
the new intermediate language. He won't feel like programming with a
schizophrenic language any more because async-land programming will
feel again like sync-land programming: simple algorithms expressed
with core language constructs that you can directly relate to the
problem you're trying to solve, not disassembled code with lots of
extra noise.
What I like in this approach is its simplicity: it should allow
mainstream Javascript developers to write code for node, without
having to learn anything radically new.
Bruno
On Jan 16, 10:07 am, Ryan Dahl <r...@tinyclouds.org> wrote:
> On Fri, Jan 14, 2011 at 11:58 AM, Havoc Pennington <h...@pobox.com> wrote:
> > Here's what I think would be nice, I wrote down a couple months ago:
billywhizz
argument. developers should be constanstly learning new things, or by
"mainstream" developers do you mean "bad" developers. please stop
flogging this dead horse....
Liam
> just what is wrong with learning something new? i don't understand the
> argument. developers should be constanstly learning new things, or by
> "mainstream" developers do you mean "bad" developers. please stop
> flogging this dead horse....
One of Ryan's stated goals is to replace PHP. When average PHP
developers start checking out Node, many are likely to be frustrated
by the issues Bruno's raising. If the Node community doesn't have an
answer for them, they won't hang around.
Havoc Pennington
On Sun, Jan 16, 2011 at 10:01 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:
> Furthermore, those implementations introduce a new keyword and are breaking
> changes to the language so considering them for use in node is incredibly
> premature.
Sure. Still, in my opinion there's a pretty interesting problem about
nicer coding models, and I think "callback spaghetti" is a real
problem in code that contains chains of async calls, especially in
more complex cases (making a set of parallel calls then collecting
results, or trying to handle errors robustly, are examples of
complexity).
There are various third-party attempts to address this for node as
mentioned earlier in the thread, showing the demand.
It's sort of like raw threads vs. higher-level tools such as
java.util.concurrent or actors. Those higher-level tools are useful.
> There isn't a task you can't accomplish because you don't have generators
> except sugar.
I've also used them pretty heavily (exactly as described) doing
client-side async code with gjs, and I think they work well for this.
The thing you can uniquely do is write sequential code (not
"inverted"/"nested") that waits on async events [1], because
generators allow you to save a continuation. Error handling also
happens more naturally since the "yield" statement can throw.
I agree generators aren't that exciting just as a way to write an
iterator, and that it's a bit odd that continuation-saving appears as
kind of a side effect of an iterator mechanism.
The usage of "yield" here is almost exactly like C#'s new "await"
keyword, I think. The "await" language feature is done purely for this
write-my-callbacks-sequentially purpose instead of mixing in the
iterator stuff. A JS feature like that might be nicer.
Kilim for Java does a very similar thing as well.
This stuff really is just syntactic sugar for typing your callbacks in
a nicer way, it's true. "Make the rest of this function a callback"
rather than typing the callback as a function literal.
However, sequential code is a lot easier for human brains to work with.
I think that's why multiple frameworks/languages/whatever are
inventing this kind of thing.
Havoc
[1] another way to write sequential async code, that's been used some
e.g. in GTK+ in the past, is the "recursive main loop" which is
allowed by GLib. In this case you don't return to the calling function
but just do a mainloop poll() while inside a callback from the
original mainloop poll(), then continue with your original callback.
There are some worrisome side effects and complexities caused by this
though so the yield/await type thing has more appeal imo.
Havoc Pennington
http://blogs.msdn.com/b/ericlippert/archive/2010/10/29/asynchronous-programming-in-c-5-0-part-two-whence-await.aspx
Havoc
Kyle Simpson
Chris Austin-Lane
> What if I don't control A? What if I don't really trust A in that sense, and only want to "subscribe" to be notified that A
> has finished, and then let *my* code that I control be responsible for executing (and parameter passing) of B?
> And that line of questioning obviously gets much murkier if there's a chain of A, B, C, and D, etc.
>
This question is very important for programming in single-threaded
call back environment when you have various libraries of various
degrees of quality.
The canonical answer is to make your interface to the bad libs you
call be intermediates with a dict of strings of pending calls. You
get called back from that lib, you take the string handed back, look
it up in the dict, log error and return if not found, and delete the
string from the dict and proceed if found (using data from the dict
values if your caller is especially bad).
If there is some chance the cb will never be invoked, then you set a
timer in what you control and delete the pending call from your dict
there and do some time out handling.
Then you can explicitly handle the suckiness of your API.
On a broader note, the stark reality of network calls is that any
given call have have at least six results:
Request receieved, succeeded, reply received.
Request receieved, succeeded, reply lost.
Request receieved, failed, reply received.
Request receieved, failed, reply lost.
Request fatal to receiver, no reply, endpoint killed. (retries do not
fix all problems).
Request lost.
People that think about these outcomes and see how different from
in-stack function calls it is will write more robust networking code,
that will be faster and have fewer weird failures than those that use
an RPC type model.
The amazing power that the one-thread, async framework gets you is an
ability to reason much more strongly about your code. This reasoning
power saves you more programmer time than the async syntax is costing
you, at least once you have any trouble or surprises.
--Chris
billywhizz
of thing in core. ryan has stated many times that core will remain as
small and low level as possible. i have no problem with anyone
building and publicising a library to do this kind of thing, but
please stop pushing for it to be in core. if i'm misreading bruno i
apologise, but it seems to me that is what he keeps arguing for...
Kyle Simpson
> of thing in core. ryan has stated many times that core will remain as
> small and low level as possible.
That's exactly what I'm proposing with the @ operator, a very low-level
basic building block *built into JavaScript* for negotiating async
continuations, localized to a single statement. Many of the common use-cases
would be composable by simple use of the @ operator, and the libraries would
then be more for the advanced use-cases and for providing additional
syntactic sugar as desired.
I'm thinking about implementing this idea as an extension to V8 to make it
available to node. Even if the idea never caught on for mainstream
JavaScript engines, that doesn't mean something low-level like this (or
similar) couldn't be useful to the V8 that powers node.
--Kyle