I love async, but I can't code like this
Sam McCall
don't want to change it.
I don't feel like Javascript is desperately missing macros.
But I need someone to talk me down, because I'm coming to the
conclusion that the certain sorts of programs are unwritable without
language extensions.
Not technically unwritable, of course, they work really well. But the
code is so full of callback noise that you can't see the logic.
I have a server that speaks an HTTP/JSON based protocol, and client
libraries for Java + Node.JS. (It's for GUI automation, think Selenium
for windows apps. Open source soon, waiting on corp lawyers).
Client apps are highly sequential. They're easy to write in Java,
they're easy to write in synchronous JavaScript, and they'd be easy to
write using Bruno Jouhier's transformation library.
The consensus seems to be that threads are evil (I agree), source
transformation is evil (I agree), and that you can write anything you
want in vanilla JS today.
Well, show me how to write this in a way that's nearly as concise as
Java (surely not a high standard!).
I don't have an easy solution, using anything other than standard
JavaScript syntax + semantics is incredibly problematic.
But can we admit that there's a problem?
== Java: ==
mainWindow.menu("File").openMenu().item("Open").click();
Window dialog = mainWindow.getChild(type(Window.class));
...
== Evil synchronous JavaScript ==
mainWindow.menu('File').openMenu().item('Open').click();
var dialog = mainWindow.getChild(type('Window'));
== JavaScript: ==
mainWindow.menu("File", function(err, file) {
if(err) throw err;
file.openMenu(function(err, menu) {
if(err) throw err;
menu.item("Open", function(err, item) {
if(err) throw err;
item.click(function(err) {
if(err) throw err;
mainWindow.getChild(type('Window'), function(err, dialog) {
if(err) throw err;
...
});
});
});
});
});
== JavaScript + Seq: ==
Seq()
.seq(function() { mainWindow.menu("File", this); })
.seq(function(file) { file.openMenu(this); })
.seq(function(menu) { menu.item("Open", this); })
.seq(function(open) { open.click(); })
.seq(function(){ mainWindow.getChild(type('Window'), this); })
.seq(function(dialog){ ... });
---
Cheers,
Sam McCall
node.js newbie
Nick Husher
blocking operation? The point of Node isn't to make everything
asynchronous, the point is to note clearly and loudly where you have
places in your code where you're communicating with a slow data
source. In other words, the file system or the network.
Let's say you're communicating with a GUI over the network: that's
cool, sometimes it's necessary. I recommend writing a wrapper that
abstracts all that asynchronous business into a set of commands that
can be queued up and run, since it's only the last callback you're
really interested in.
> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>
Guillermo Rauch
Sam McCall
> You're just using a blocking API. You can do the same thing with Node.JS
Guillermo Rauch
browser
.chain
.session()
.open('/')
.assertTitle('Something')
.and(login('foo', 'bar'))
.assertTitle('Foobar')
.and(login('someone', 'else'))
.assertTitle('Someone else')
.end(function(err){
if (err) throw err;
});Sam McCall
> Is interacting with (what appears to be) a GUI menu actually a
> blocking operation?
on machine A, if we call openMenu on a menu item then we can't do
anything[1] until we get the HTTP response. But of course we can do
work on another session targeted at another machine.
[1] Actually we can, but whether the menu is opened/closed/midway
isn't defined, so we can't usually do anything useful.
> Let's say you're communicating with a GUI over the network: that's
> cool, sometimes it's necessary.
> I recommend writing a wrapper that
> abstracts all that asynchronous business into a set of commands that
> can be queued up and run, since it's only the last callback you're
> really interested in.
window.clickMenuItem("File", "Open", cb);
Unfortunately, I chose it because it was universal/familiar, in
practice the code snippets tend to be application specific because the
UIAutomation tree (the Windows equivalent of the DOM) tends to vary in
unintuitive per-application ways.
Therefore the wrappers couldn't be part of the library, they'd be
written by whoever uses the library, for every app-specific task they
want to do. So you're writing the same noisy code, but in a function.
The JavaScript client is experimental, but we've used the Java client
to write automated tests for in-house software and in practice there's
little to no redundancy in the API - a higher level API would stop you
doing things you need to be able to do.
On Jan 11, 8:18 pm, Guillermo Rauch <rau...@gmail.com> wrote:
> If you're writing a desktop GUI app, you have two options:
> - Blocking API + web workers
overhead and sync problems of threads just to expose an API you can
write readable programs in. (Plus it doesn't work in Node yet,
right?)
> - Non-blocking and some API that simplifies the nesting and error handling.
Our API is object-oriented (very similar to WebDriver) so we need to
deal with different object handles rather than just a single Selenium
connection, and there's sometimes complex flow-control mid-chain, but
I'll see if I can get something that works reasonably. (And ideally
that separates cleanly from both the app code and the library, so both
are readable). At this point i'm starting to disagree with people who
think everyone should write their own!
Cheers,
Sam
Bruno Jouhier
I wrote streamline.js for people like you and me, people who like
Javascript, who like node's event model and who are involved in
industrial projects, which means that they will, at one point or
another, need to involve "mainstream" programmers into it.
Mainstream programmers understand Javascript's flow control
statements. So I think that they'll understand streamline.js. I don't
think that they will be ever as productive and as comfortable with a
DSL however smart the DSL might be. Why would you need a DSL anyway
when all you're asking for is basic flow control statements that
actually exist in the language but got broken somehow. What you want
is that someone fixes the language so that you can use for loops, try/
catch, dot notation again, not that someone introduces new
abstractions that will never be as concise as the language's
primitives, that will have a learning curve and that will be obsolete
the next month because a newer abstraction will have taken the
spotlights.
I released streamline.js under MIT license which is not our standard
corporate license so that the community can feel relaxed about it (I'm
probably more fortunate than you because I have a kind CTO who
expedited things with the lawyers). So it is available and free and we
can start using it.
I am not particularly fond of preprocessors and I went this route
simply because I did not see any way to fix the problem in user-land
(I tried to write my own DSL like everyone else of course but I felt
like loosing my time). That's all. There is no ideological stand in
this, just pragmaticism.
So even if code transformation is evil, we may have to sympathize with
the devil, and write rock'n roll code. At least that's what I'm doing
from this point on. And maybe some day, the fix will get into the
language for real. After all, all it needs is a small operator or
keyword (to replace the underscore hack) and a relatively simple
algebraic code transfomation that can be pipelined into the other
stages of the compiler.
Bruno
Liam
entities, and returning only the result you need for a particular
sequence...
Isn't there a serialization format for UI automation sequences that
allows them to be saved and played back?
Isaac Schlueter
of the excluded middle. No one actually writes code like this. The
options are not "73 layers of indentation, or threads."
The language includes named function declarations. Use them. That
alone would make your example way less terrible.
--i
Bradley Meck
Sam McCall
it.
I really don't like the idea of a preprocessor, with and the
associated deployment hassles and ambiguous semantics. So if it can be
done with something like a proxy object that records a chain of method
calls synchronously, then plays them back asynchronously, I'd prefer
that. But if it isn't manageable, then pragmatically I think it'll be
the best option. (Your implementation looks really good, it's a purely
philosophical objection).
Liam: we haven't defined such a format. The actions are functions with
complex object parameters and many different types of return values
that can raise errors at any step etc. This would basically mean
defining a new scripting language with synchronous semantics and
writing an asynchronous interpreter for it. It seems strange to have
to do this.
Isaac: there are 5 inline functions declared in the example.
'var x = function(){};' 5 times is a lot of code even before you do
anything in them, it can never be concise or really readable.
Besides, I'm not really sure what to name them as they don't really
have much independent behaviour, they're just glue.
But I'd be happy to be convinced, could you show me how you'd write it?
Liam
style programming...
But if you don't need tons of concurrent script "threads", perhaps
write sync JS scripts and use a pool of Node processes consuming a
queue of scripts?
Isaac Schlueter
> Isaac: there are 5 inline functions declared in the example.
> 'var x = function(){};' 5 times is a lot of code even before you do
> anything in them, it can never be concise or really readable.
> Besides, I'm not really sure what to name them as they don't really
> have much independent behaviour, they're just glue.
> But I'd be happy to be convinced, could you show me how you'd write it?
Cut out the "var" and the "=". But yeah, "function" is long and
klunky for how common it is. I'd love a shorter keyword for it.
Here's a very brief untested example of what I'm talking about:
https://gist.github.com/775591
I usually only care about success or failure, so I don't use any kind
of "passalong" util in my programs. But it's not hard to see how this
general technique could be applied in other ways.
--i
shaun etherton
IMO this is a great example; especially for new comers. It makes the
strange looking yet common function nesting look more familiar, then
progresses to show the awesomeness of JS in a simple, clear and
understandable way.
Sometimes i think those who are intimately familiar with the nested
callbacks/anon function style may not realize the amount of angst and
confusion it can cause for the uninitiated.
- shaun
mgutz
# using named functions
doSomething = (done) ->
mainWindow.menu "File", onFile
onFile = (er, file) ->
return cb(er) if er
file.openMenu onMenu
onMenu = (er, menu) ->
return er if er
menu.item "Open", onOpen
onOpen = (er, item) ->
return er if er
item.click onClick
onClick = (er) ->
return er if er
mainWindow.getChild type("Window"), onGetWindow
onGetWindow = (er, dialog) ->
return er if er
dialog.doSomething done
mgutz
# using named functions
function doSomething (cb) ->
onFile = (er, file) ->
return cb(er) if er
file.openMenu onMenu
onMenu = (er, menu) ->
onOpen = (er, item) ->
onClick = (er) ->
onGetWindow = (er, dialog) ->
dialog.doSomething cb(null)
aaronblohowiak
back and edit some of my source to do this.
Floby
variables.
all your coffeescript function will come up as anonymous in a stacj
trace.
Preston Guillory
I run into the same situation frequently, so I feel his pain. My code has lots of nesting and "if (err) return callback(err)". Any time I touch the file system, or call exec(), or query a database, or interact with the network. And hey, that describes basically everything the code does!
It's not a problem in the sense that the approach doesn't work -- the code functions and it's fast. But for a lot of common programming tasks, Node.js is verbose, for sure.
John Taber
code w/ assoc. callbacks is great when apps need non-blocking, but for
many "normal" CRUD type apps where blocking isn't that much an issue,
either a simpler approach to using Node or a synchronous js server side
package (w/o the huge JVM) would make programming simpler.
Sam McCall
very narrow case (calling methods on the 'return value' of an async
result), but I got this working:
var Plate = require('plate'); // [1]
Plate(mainWindow).menu('File').openMenu().item('Open').click().end(function(err)
{
if(err) throw err;
});
Where all the library functions are still written in callback-as-last-
parameter style.
Plate returns a proxy (hoxy?) object that records the method calls and
starts the chain when you call end().
I'll code using this for a while and see if there are other common
patterns I can't live without.
[1] https://github.com/sam-mccall/node-plate
Mihai Călin Bazon
that it turns your code inside-out, almost like disassembling a piece
of code from a high-level language into a lower-level one. Async:
get_val("foo", function(error, foo){
if (error) print("there was an error");
else get_val("bar", function(error, bar){
if (error) print("there was an error");
else get_val("baz", function(error, baz){
if (error) print("there was an error");
else print(foo + bar + baz);
});
});
});
Sync:
try {
print(get_val("foo") + get_val("bar") + get_val("baz"));
} catch(error) {
print("there was an error");
}
The sync version is cleaner: you can see at a glance what it does. It
prints something -- the sum of 3 values returned by get_val. In case
of an error, you get "there was an error". You get the luxury of
handling the error in a single place.
The async looks like the low-level version of the sync one. If we
could disassemble a JavaScript function, it would look pretty much
like the async version (in terms of program flow). What happens is
that get_val("foo") is executed first, then get_val("bar"), then
get_val("baz"), then their sum is computed and printed -- that's
exactly what both versions do. But which one do you prefer to write?
I don't want to hurt anyone's feelings, but to me at least, Node is
unusable for serious work because of this. I'm prepared to lobby
against do-everything-asynchronously for a long time. :-)
Cheers,
-Mihai
--
Mihai Bazon,
http://mihai.bazon.net/blog
Guillermo Rauch
print(a,b,c)
Sam McCall
blocking APIs is the essential premise behind node.js.
Sure, it has issues that could be addressed nicely by using threads
instead, and I agree that with the current conventions, code can be
hard to read and debug.
But fixing this by using threads would be a solution that had nothing
in common with node, so it's not really on-topic for the node mailing
list.
Isaac Schlueter
> a synchronous js server side package would make programming simpler.
There are many of those. Have fun. (Not being facetious.
Programmers write better programs while enjoying the process, so if
that means using Rhino or v8cgi or Ruby, more power to you.)
This conversation is a bit silly. I mean, conceptually, asynchronous
code *is* a bit more complicated.
For many problems, threads do not scale effectively. But for many
problems, they're perfectly adequate, and if you're using an
implementation that's well understood, you may even be able to work
around the Hard Problems that they introduce.
Node is cool and hot and all the tech blorgs in the blagodome are
weblooging about how awesome and fast and super insanely stupid sexy
it is. But that doesn't mean it's best for every problem, or that
JavaScript is everyone's favorite language, or that you have to love
callbacks, or that you should use it.
When you're writing network programs, your program is a proxy the vast
majority of the time. It is a glue that wires together a few
different things that *by their very nature* are most efficiently
handled with asynchronous evented message-passing. XMPP, HTTP, AMQP,
child processes, etc. Those problems are well served by an event
loop.
You can run an event loop in C. You know what? After juggling
structs and function pointers for a bit, all this "verbose" JavaScript
starts to look pretty nice.
> for many "normal" CRUD type apps where blocking isn't that much an issue...
Are you joking? That's *exactly* the sort of app where blocking very
quickly becomes a *huge* issue.
Or do you mean the kind of app where only one user is ever going to be
creating, reading, updating, and deleting data at once? Because
that's what blocking IO means.
All websites are fast with one user on localhost.
> On 01/12/2011 06:54 PM, Preston Guillory wrote:
>> Sam has a point. Asynchronous code is much larger and has a lot of
>> syntactic noise. Look at his example: synchronous Java does in 2 lines what
>> takes 7 lines of asynch code
Except that the 7 lines of async code *can handle other events while
waiting*, without the overhead of coroutines or threads.
And, with the Java version, there's a *ton* of stuff going on below
the surface. Actually doing real work in a higher-level-than-c
language with an event loop is kind of a newish thing. Granted, those
of us who cut our teeth on web browsers have been doing it forever,
which is part of why JS is so right for this task. But the tooling is
not super mature, and node is a lowlevel library by design.
>> Named intermediate functions
>> don't help -- see how Isaac's example stretched to 14 lines.
They reduce indentation, label blocks, and improve stack traces. They
won't improve your golf game.
>> But for a lot of common programming tasks, Node.js
>> is verbose, for sure.
The advantages are that you can do a bunch of things in parallel, and
handle "local" stuff in the pattern as "remote" stuff.
I write most of my shell scripts in Bash.
On Wed, Jan 12, 2011 at 23:39, Sam McCall <sam.m...@gmail.com> wrote:
> Plate(mainWindow).menu('File').openMenu().item('Open').click().end(function(err)
> {
> if(err) throw err;
> });
>
> Where all the library functions are still written in callback-as-last-
> parameter style.
See? I *told you* you could do better than my example ;D
If you can come up with a terser way to express callbacks, that
doesn't lose anything in translation, then by all means, implement it.
I think that this problem, such as it is, will be solved by either
writing a new language no one uses, or just sucking it up and using
the JavaScript we have instead of the JavaScript we wish we had.
--i
Bruno Jouhier
This is why I wrote the little streamline.js transformation engine. I
was getting sick of turning the code inside out, not being able to use
the language's native constructs, etc. I felt like programming with a
low-level language (see http://bjouhier.wordpress.com/2011/01/09/asynchronous-javascript-the-tale-of-harry/)
Maybe some people find it "fun" to write such disassembled code. I
don't. This is just a waste of energy.
There seems to be a golden rule out there saying: "You have to solve
this problem in user-land, through DSLs, libraries, whatever, ...".
But this is just not working, or rather it always only works "up to a
point". So maybe someone has to break the rule some day, abandon this
dogma that the solution must lie in user land, write a transformation
engine, design it so that it fits the node model (you should be able
to consume node functions, and the functions that you write should in
turn be regular node functions) and hook it up into node. See
https://github.com/Sage/streamlinejs
I used a "naming" hack because I want to be able to use this today,
with today's Javascript editors (I cannot write JS code if the
"reformat entire file" function is broken). But this is not how it
should be done ultimately. What is needed is a special operator to
distinguish async function definitions and calls from sync ones. If we
had such an operator in the language, we could happily use Javascript
to mix sync and async calls.
There are probably some other solutions, like introducing true
continuations in the language (my CPS transform is probably only half-
baked) but we have to also be careful to not lose sight of what makes
Javascript so successful: its simplicity and its "mainstream-ness". So
it is probably more important to "repair" the built-in keywords of the
language so that they work in async-land the same way they have always
worked in sync-land, than to introduce more abstract features into the
language.
Also, I have the weird feeling that continuations are somewhat like
gotos (they really allow you to jump anywhere) while the
"streamlining" that I am proposing is more "structured". It will not
put the full power of continuations in the hands of programmers (so
that they don't shoot themselves in the foot) but it will give them
enough to solve their main problem (writing async code).
Bruno
Sam McCall
> See? I *told you* you could do better than my example ;D
I'm not convinced that "write your own, it's easy" is a the right long
term answer though.
You learn a lot by implementing your own hashtable, but most of the
time you use a library.
> sucking it up and using the JavaScript we have instead of the JavaScript we wish we had
'Functions take parameters, perform actions, and return values' is
syntax.
'Functions take a callback parameter, and call it on completion with
the error as the first argument' is idiom.
And I think it's entirely appropriate to try to find the most
expressive idioms, given the syntax we have. In public, both because
otherwise we might give up (see first post) and because having idioms
in common mean we can read each others' code.
Secondly, node.js is 'cool and hot' and could well be influential in
JS's future development.
That doesn't mean wishing for a language feature will magically make
it appear, but if people are aware of the friction points then maybe
there'll be a proposal or two when ES6 comes around.
Sam
billywhizz
i would also recommend anyone who is advocating adding threads/
blocking calls to node.js to have a look at this presentation by the
godfather of javascript:
http://www.yuiblog.com/blog/2010/08/30/yui-theater-douglas-crockford-crockford-on-javascript-scene-6-loopage-52-min/
if you're pressed for time, skip to approx 20 mins in where he starts
to get into the nitty-gritty of event loops v threading. mr crockford
explains it a lot better than i would be able to...
yes, async coding in an event loop is different and does have some
pain associated with it, but from where i am standing the pain is a
lot less than dealing with all the synchronization and locking issues
that threading and shared memory bring to the table...
On Jan 13, 10:04 am, Isaac Schlueter <i...@izs.me> wrote:
Preston Guillory
I haven't tried Bruno's solution yet, but I like direction it's taking. Programming languages trend toward compact, expressive syntaxes. See Ruby and Python.
Bradley Meck
Bruno Jouhier
me :-)
What do you mean by concurrent editting? Can you give an example?
The generated code corresponds to what you would have to write with
callbacks to reproduce the sequential chaining that is expressed in
the sequential version of the code (if this mumbo jumbo makes any
sense to you). If you want to see examples of what is generated, you
can take a look at the unit test: https://github.com/Sage/streamlinejs/blob/master/test/transform-test.js
I'm actually thinking of revisiting this a bit. The idea is that when
I write something like
var x = foo_().bar_().x + zoo_().y
it is very important to have a sequential chain for foo_().bar_().x on
one side and another one for zoo_().y but it rather counterproductive
to chain the two together to apply the plus. Instead, we should let
them execute in parallel and join the two chains before the plus.
My current transform doesn't do that but it should be relatively easy
to modify it so that it does that. With this, the programmer would get
transparent parallelization of the evaluation of sub-expressions. I
find the idea pretty cool. Actually, if I apply the same principle to
array and object literals, I get a pretty neat way to parallelize
function calls and put their results into an array (or an object):
var results = [ f1_(), f2_() ]; // this is neater than the spray/
colllectAll API that I proposed.
But then, you'll need a mechanism to control the level of parallelism.
There are places where you really want to serialize execution. This is
where the little "funnel" utility comes into play. The diskUsage2.js
shows how I use it to avoid exhaustion of file descriptors when
parallelizing a recursive traversal of directories. (https://
github.com/Sage/streamlinejs/blob/master/examples/diskUsage2.js).
So maybe the funnel is what you're asking for. If you set its max to
1, you get some kind of monitor that queues incoming calls and only
lets them go through one at a time.
Bruno.