Let's say you're communicating with a GUI over the network: that's
cool, sometimes it's necessary. I recommend writing a wrapper that
abstracts all that asynchronous business into a set of commands that
can be queued up and run, since it's only the last callback you're
really interested in.
> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>
browser
.chain
.session()
.open('/')
.assertTitle('Something')
.and(login('foo', 'bar'))
.assertTitle('Foobar')
.and(login('someone', 'else'))
.assertTitle('Someone else')
.end(function(err){
if (err) throw err;
});
The language includes named function declarations. Use them. That
alone would make your example way less terrible.
--i
Cut out the "var" and the "=". But yeah, "function" is long and
klunky for how common it is. I'd love a shorter keyword for it.
Here's a very brief untested example of what I'm talking about:
https://gist.github.com/775591
I usually only care about success or failure, so I don't use any kind
of "passalong" util in my programs. But it's not hard to see how this
general technique could be applied in other ways.
--i
IMO this is a great example; especially for new comers. It makes the
strange looking yet common function nesting look more familiar, then
progresses to show the awesomeness of JS in a simple, clear and
understandable way.
Sometimes i think those who are intimately familiar with the nested
callbacks/anon function style may not realize the amount of angst and
confusion it can cause for the uninitiated.
- shaun
get_val("foo", function(error, foo){
if (error) print("there was an error");
else get_val("bar", function(error, bar){
if (error) print("there was an error");
else get_val("baz", function(error, baz){
if (error) print("there was an error");
else print(foo + bar + baz);
});
});
});
Sync:
try {
print(get_val("foo") + get_val("bar") + get_val("baz"));
} catch(error) {
print("there was an error");
}
The sync version is cleaner: you can see at a glance what it does. It
prints something -- the sum of 3 values returned by get_val. In case
of an error, you get "there was an error". You get the luxury of
handling the error in a single place.
The async looks like the low-level version of the sync one. If we
could disassemble a JavaScript function, it would look pretty much
like the async version (in terms of program flow). What happens is
that get_val("foo") is executed first, then get_val("bar"), then
get_val("baz"), then their sum is computed and printed -- that's
exactly what both versions do. But which one do you prefer to write?
I don't want to hurt anyone's feelings, but to me at least, Node is
unusable for serious work because of this. I'm prepared to lobby
against do-everything-asynchronously for a long time. :-)
Cheers,
-Mihai
--
Mihai Bazon,
http://mihai.bazon.net/blog
print(a,b,c)
There are many of those. Have fun. (Not being facetious.
Programmers write better programs while enjoying the process, so if
that means using Rhino or v8cgi or Ruby, more power to you.)
This conversation is a bit silly. I mean, conceptually, asynchronous
code *is* a bit more complicated.
For many problems, threads do not scale effectively. But for many
problems, they're perfectly adequate, and if you're using an
implementation that's well understood, you may even be able to work
around the Hard Problems that they introduce.
Node is cool and hot and all the tech blorgs in the blagodome are
weblooging about how awesome and fast and super insanely stupid sexy
it is. But that doesn't mean it's best for every problem, or that
JavaScript is everyone's favorite language, or that you have to love
callbacks, or that you should use it.
When you're writing network programs, your program is a proxy the vast
majority of the time. It is a glue that wires together a few
different things that *by their very nature* are most efficiently
handled with asynchronous evented message-passing. XMPP, HTTP, AMQP,
child processes, etc. Those problems are well served by an event
loop.
You can run an event loop in C. You know what? After juggling
structs and function pointers for a bit, all this "verbose" JavaScript
starts to look pretty nice.
> for many "normal" CRUD type apps where blocking isn't that much an issue...
Are you joking? That's *exactly* the sort of app where blocking very
quickly becomes a *huge* issue.
Or do you mean the kind of app where only one user is ever going to be
creating, reading, updating, and deleting data at once? Because
that's what blocking IO means.
All websites are fast with one user on localhost.
> On 01/12/2011 06:54 PM, Preston Guillory wrote:
>> Sam has a point. Asynchronous code is much larger and has a lot of
>> syntactic noise. Look at his example: synchronous Java does in 2 lines what
>> takes 7 lines of asynch code
Except that the 7 lines of async code *can handle other events while
waiting*, without the overhead of coroutines or threads.
And, with the Java version, there's a *ton* of stuff going on below
the surface. Actually doing real work in a higher-level-than-c
language with an event loop is kind of a newish thing. Granted, those
of us who cut our teeth on web browsers have been doing it forever,
which is part of why JS is so right for this task. But the tooling is
not super mature, and node is a lowlevel library by design.
>> Named intermediate functions
>> don't help -- see how Isaac's example stretched to 14 lines.
They reduce indentation, label blocks, and improve stack traces. They
won't improve your golf game.
>> But for a lot of common programming tasks, Node.js
>> is verbose, for sure.
The advantages are that you can do a bunch of things in parallel, and
handle "local" stuff in the pattern as "remote" stuff.
I write most of my shell scripts in Bash.
On Wed, Jan 12, 2011 at 23:39, Sam McCall <sam.m...@gmail.com> wrote:
> Plate(mainWindow).menu('File').openMenu().item('Open').click().end(function(err)
> {
> if(err) throw err;
> });
>
> Where all the library functions are still written in callback-as-last-
> parameter style.
See? I *told you* you could do better than my example ;D
If you can come up with a terser way to express callbacks, that
doesn't lose anything in translation, then by all means, implement it.
I think that this problem, such as it is, will be solved by either
writing a new language no one uses, or just sucking it up and using
the JavaScript we have instead of the JavaScript we wish we had.
--i
Really? Seems to me like it's working. Qv: this thread.
> What is needed is a special operator to
> distinguish async function definitions and calls from sync ones. If we
> had such an operator in the language, we could happily use Javascript
> to mix sync and async calls.
Sure. That's a fair point. So... go do that.
But it won't be JavaScript. It'll be some other thing. If you want
to change JavaScript-the-language, you're posting on the wrong mailing
list.
If you wanna design a new language that compiles to node-friendly JS,
you can clearly do that. Some people will love it, others will hate
it, sky's still blue, pope's still catholic, etc.
> It will not
> put the full power of continuations in the hands of programmers
You may find that programmers are more power-hungry than you can
possibly imagine. ;)
On Thu, Jan 13, 2011 at 02:35, Sam McCall <sam.m...@gmail.com> wrote:
> First, there's a difference between syntax and idiom.
Fair point.
Node's idiom is the most simple idiom possible to *enable* more
creative things in userland. It's the base. Think of the
continuation-callback style as the bytecode that your flow control
library interfaces with.
When you're interfacing with the low-level node libraries a lot,
callbacks are actually a very nice abstraction that gives you a lot of
control over what's going on. If your program is going to deal with
streams more often than single fs operations, then perhaps it makes
more sense to structure your program differently. No one's trying to
say that callbacks are best for everything.
In npm, I *have* to do a lot of lowlevel file system operations.
That's what npm is for. So, I need some abstraction around it so that
I can do a list of N things sequentially or in parallel, and compose
multiple callback-taking actors into one.
Since I have those abstractions, it makes sense to kind of shove the
stream-ish operations (writing or uploading a file, for instance) into
a function that takes a callback, so that I can say "Write this file,
create a symlink, make this directory, upload this thing, and then
signal success or failure" and wrap that up as one "operation" to be
composed in another area of my program.
The thing about closures and continuations is that, while they look
obnoxious in these little examples, once embraced fully, they allow
for a huge degree of DRYness and consistency.
> And I think it's entirely appropriate to try to find the most
> expressive idioms, given the syntax we have. In public, both because
> otherwise we might give up (see first post) and because having idioms
> in common mean we can read each others' code.
Rewind the NodeJS clock about 10 months, to when everyone was debating
how to do Promises in the most powerful and correct way.
Shooting that horse was the only way to get everyone to stop beating
it. Now we have 3 promise libraries that are all better than the one
that was in node-core.
It is important for a platform to be very careful about what it lets
in, lest it prevent innovation by the community. Ryan's extreme
aesthetic in that regard is a big part of what has made node
successful.
> if people are aware of the friction points then maybe
> there'll be a proposal or two when ES6 comes around.
I don't disagree. But I've got programs to write in the thousand
years or so before ES6 will ever be a reality.
--i
Yep.
> I really wish we had continuations in node, this stuff would be cake.
The actor/callback pattern basically is CPS, though of course it's
much uglier in JS than in Scheme. There's no TCO, of course, but the
event loop's stack-destruction performs a similar effect as TCO.
The thing is that, like in Scheme, JS's continuations trap their state
in a closure, rather than a stack-saving Continuation object that
implements coroutines. It's very similar in practice to the Promise
pattern, except done in FP style rather than OOP.
--i
but it requires yield (generators) in the JS implementation.
Havoc
Node is the present of JavaScript.
--i
On Fri, Jan 14, 2011 at 15:52, Mikeal Rogers <mikeal...@gmail.com> wrote:
> you should tell that the es-discuss list :)
>
== Java: ==
mainWindow.menu("File").openMenu().item("Open").click();
Window dialog = mainWindow.getChild(type(Window.class));
Coroutines are possible with V8. That is, you can longjmp into another
stack without disrupting the VM - there are several implementations on
the internet. Node is not going to do this because multiple call
stacks dramatically complicate the mental model of how the system
works. Furthermore it's a hard deviation from JavaScript - browser
programmers have never had to consider multiple execution stacks - it
would confuse them to subtly yet significantly change the language.
Adding multiple stacks is not a seamless abstraction. Users must know
worry that at any function call they may be yielded out and end up in
an entirely different state. Locking is now required in many
situations. Knowing if a function is reentrant or not is required.
Documentation about the "coroutine safety" of methods is required.
Node offers a powerful guarantee: when you call a function the state
of the program will only be modified by that function and its
descendants.
On Sun, Jan 16, 2011 at 4:07 AM, Ryan Dahl <r...@tinyclouds.org> wrote:
> Adding multiple stacks is not a seamless abstraction. Users must know
> worry that at any function call they may be yielded out and end up in
> an entirely different state. Locking is now required in many
> situations. Knowing if a function is reentrant or not is required.
I sympathize with what you're saying and with node's design. There are
real tradeoffs in simplicity and also in overhead. My first reaction
to this was also, "it's better to be explicit and type in your
callbacks."
That said:
- I'm not sure all the points you mentioned about coroutines apply.
JavaScript generators are more controlled.
- I'd love to try something built on node or like node that goes
slightly differently, even if node itself remains with the tradeoffs
it has. I think the way node works is good, but it's not the only
useful way.
It sort of depends on what kind of developer and what kind of task.
- Raw-callback code is truly rocky in many instances. Chains of
callbacks where async result C depends on async result B depends on
async result A, are the worst. And trying to ensure that the outermost
callback always gets called, exactly once, with only one of either an
error or a valid result. It's not writable or readable code.
Some details that I think matter:
1. Generators here are just an implementation detail of promises. They
don't "leak" upward ... i.e. I don't think a caller would ever care
that somewhere below it in the call stack, someone used "yield" - it
should not be distinguishable from someone creating a promise in
another way.
See fromGenerator() here, which is just a promise factory:
http://git.gnome.org/browse/gjs/tree/modules/promise.js#n317
you mentioned:
> Node offers a powerful guarantee: when you call a function the state
> of the program will only be modified by that function and its
> descendants.
fwiw, I don't think generators and yield modify this guarantee. From
"inside" the generator during a yield, yes program state can be
changed by the caller; but from "outside" the generator, the generator
is just a function. Anytime program state can change "under you,"
you're going to be seeing the yield keyword, and you're going to be
implementing a generator yourself.
2. Promises in this code are just callback-holders. That is, if you
have an API that takes a callback(result, error) then you can
trivially wrap it to return a promise:
function asyncThingWithPromise() {
var p = new Promise();
asyncThingWithCallback(function(result, error) { if (error)
p.putError(error); else p.putReturn(result); })
return p;
}
And vice versa:
function asyncThingWithCallback(cb) {
asyncThingWithPromise.get(cb);
}
3. The way I'd see APIs and libraries in this world would be
promise-based, i.e. an async routine returns a promise. Given
asyncThing() that returns a promise, you can set up a callback:
asyncThing().get(function(result, error) { });
Or if you're a generator, you can kick the promise up to your caller,
save your continuation, and either resume or throw when the result
arrives:
result = yield asyncThing();
"yield" only goes up one frame though, it just makes an iterator from
the continuation and returns it, it doesn't unwind the stack or allow
arbitrary code to run. The caller (or some caller of that caller)
still has to set up a callback on the promise. The callback would
resume the continuation.
4. I could be wrong but I don't think JS generators in SpiderMonkey
get longjmp() involved. The yield keyword saves a continuation, but
the generator function then returns "normally" returning an iterator
object. The calling function has to take that iterator object (which
is implemented with a saved continuation) and if it chooses, it can
further unwind the stack by itself yielding or returning, or it
doesn't have to. Anyway there's not really much magic. Continuations
are an implementation detail of iterators.
Another way to put it, is that at each re-entrancy point you're going
to see either the "yield" keyword (push the promise to the caller, by
returning a promise-generating iterator) or you're going to see a
callback (add a callback to the promise).
It isn't like exceptions or threads where you can be interrupted at
any time. Interruptions only happen when you unwind all the way back
to the event loop which then invokes a callback.
5. The flip side of this is that "yield" isn't magic; people have to
understand how it relates to promises, what a generator function is,
etc. Pasting yield-using code from one function to another might not
work if the destination function isn't a generator.
6. I piled several potentially separable but kinda-related things into
the blog post I linked to. One is the "yield" convention. Others
include:
- have "actors" which have their own isolated JS global object, so no
state shared with other actors.
- share the event loop among actors such that callbacks from
different actors can run concurrently but those from the same actor
can't
(i.e. from perspective of each distinct JS global object, there's
only one thread, from perspective of the overall runtime, there are
multiple - but not thread-per-request! the thread pool runs libev
callbacks)
(actors think they each have their own event loop, but it's
implemented with a shared one for efficiency)
- have a framework where each http request gets its own actor
(therefore global object) ... (something that's only efficient with
SpiderMonkey in their most bleeding edge stuff, and I have no idea
about any other JS implementation. and for sure it would always have
_some_ overhead. I haven't measured it.)
- I'd expect state shared among http requests to be handled as in
other actor frameworks, for example you might start an actor which
holds your shared state, and then http request handlers could use
async message passing to ask to set/get the state, just as if they
were talking to another process.
7. In addition to using all cores without shared state or locking, an
actors approach could really help with the re-entrancy dangers of both
callbacks AND a "yield promise" syntax. The reason is that each actor
is isolated, so you don't care that the event loop is running handlers
in *other* actors. The only dangerous re-entrancy (unexpected callback
mixing) comes from callbacks in the actor you're writing right now.
One way to think about actors is that each one works like its own
little server process, even though implementation-wise they are much
lighter-weight.
8. So the idea is Akka-style functionality, and ALSO libev, and ALSO
ability to write in JavaScript, and ALSO wire it up nicely to http
with actor-per-request ;-) and a pony of course
Anyhow, I realize there are lots of practical problems with this from
a node project perspective. And that it's harder to understand than
the browser-like model.
At the same time, perhaps it remains simpler than using actors with
Java, Scala, or Erlang and preserves other advantages of node. Once
you're doing complex things, sometimes it can be simpler to have a
little more power in the framework taking care of hard problems on
your behalf.
Just some food for thought is all since it was sort of an in-the-weeds
thread to begin with. I don't intend it as a highly-unrealistic
feature request ;-) don't worry.
I am hoping there are people out there who explore these kind of
directions sometime.
Havoc
On Sun, Jan 16, 2011 at 10:01 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:
> Furthermore, those implementations introduce a new keyword and are breaking
> changes to the language so considering them for use in node is incredibly
> premature.
Sure. Still, in my opinion there's a pretty interesting problem about
nicer coding models, and I think "callback spaghetti" is a real
problem in code that contains chains of async calls, especially in
more complex cases (making a set of parallel calls then collecting
results, or trying to handle errors robustly, are examples of
complexity).
There are various third-party attempts to address this for node as
mentioned earlier in the thread, showing the demand.
It's sort of like raw threads vs. higher-level tools such as
java.util.concurrent or actors. Those higher-level tools are useful.
> There isn't a task you can't accomplish because you don't have generators
> except sugar.
I've also used them pretty heavily (exactly as described) doing
client-side async code with gjs, and I think they work well for this.
The thing you can uniquely do is write sequential code (not
"inverted"/"nested") that waits on async events [1], because
generators allow you to save a continuation. Error handling also
happens more naturally since the "yield" statement can throw.
I agree generators aren't that exciting just as a way to write an
iterator, and that it's a bit odd that continuation-saving appears as
kind of a side effect of an iterator mechanism.
The usage of "yield" here is almost exactly like C#'s new "await"
keyword, I think. The "await" language feature is done purely for this
write-my-callbacks-sequentially purpose instead of mixing in the
iterator stuff. A JS feature like that might be nicer.
Kilim for Java does a very similar thing as well.
This stuff really is just syntactic sugar for typing your callbacks in
a nicer way, it's true. "Make the rest of this function a callback"
rather than typing the callback as a function literal.
However, sequential code is a lot easier for human brains to work with.
I think that's why multiple frameworks/languages/whatever are
inventing this kind of thing.
Havoc
[1] another way to write sequential async code, that's been used some
e.g. in GTK+ in the past, is the "recursive main loop" which is
allowed by GLib. In this case you don't return to the calling function
but just do a mainloop poll() while inside a callback from the
original mainloop poll(), then continue with your original callback.
There are some worrisome side effects and complexities caused by this
though so the yield/await type thing has more appeal imo.
Havoc
> What if I don't control A? What if I don't really trust A in that sense, and only want to "subscribe" to be notified that A
> has finished, and then let *my* code that I control be responsible for executing (and parameter passing) of B?
> And that line of questioning obviously gets much murkier if there's a chain of A, B, C, and D, etc.
>
This question is very important for programming in single-threaded
call back environment when you have various libraries of various
degrees of quality.
The canonical answer is to make your interface to the bad libs you
call be intermediates with a dict of strings of pending calls. You
get called back from that lib, you take the string handed back, look
it up in the dict, log error and return if not found, and delete the
string from the dict and proceed if found (using data from the dict
values if your caller is especially bad).
If there is some chance the cb will never be invoked, then you set a
timer in what you control and delete the pending call from your dict
there and do some time out handling.
Then you can explicitly handle the suckiness of your API.
On a broader note, the stark reality of network calls is that any
given call have have at least six results:
Request receieved, succeeded, reply received.
Request receieved, succeeded, reply lost.
Request receieved, failed, reply received.
Request receieved, failed, reply lost.
Request fatal to receiver, no reply, endpoint killed. (retries do not
fix all problems).
Request lost.
People that think about these outcomes and see how different from
in-stack function calls it is will write more robust networking code,
that will be faster and have fewer weird failures than those that use
an RPC type model.
The amazing power that the one-thread, async framework gets you is an
ability to reason much more strongly about your code. This reasoning
power saves you more programmer time than the async syntax is costing
you, at least once you have any trouble or surprises.
--Chris
That's exactly what I'm proposing with the @ operator, a very low-level
basic building block *built into JavaScript* for negotiating async
continuations, localized to a single statement. Many of the common use-cases
would be composable by simple use of the @ operator, and the libraries would
then be more for the advanced use-cases and for providing additional
syntactic sugar as desired.
I'm thinking about implementing this idea as an extension to V8 to make it
available to node. Even if the idea never caught on for mainstream
JavaScript engines, that doesn't mean something low-level like this (or
similar) couldn't be useful to the V8 that powers node.
--Kyle