blog post: Why We’ve Bet on Node plus Streamline (Hint: It’s Productivity)

[Seth Pollack]

I've written up a blog post here http://bit.ly/KtMIYS that explains why we've chosen Node.js plus Streamline as our platform for our product (www.rivaliq.com). 

We are all Node fans in this forum, but i'm guessing that fewer folks have given Streamline (https://github.com/Sage/streamlinejs) a spin. We have found significant developer productivity (and happiness!) benefits by making the switch to using it. I'm not saying that here to troll for more "my control flow is better than your control flow" debates, but rather to share our experiences in order to hopefully help others. Please give it a read, and I welcome your comments on the blog. 

Seth

[Mark Hahn]

The examples are missing in my browser.  I'm seeing the titles with nothing between them.  I'm on the latest chrome.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Seth Pollack]

Mark, sorry about that. Code examples are all here: https://gist.github.com/spollack/8260776

Let me look into why that is happening (I am on latest Chrome too, and i’m seeing them inline ok)

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/atYV13nATA8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

[Mark Hahn]

When I went back and checked they were OK.  Weird.  I didn't imagine it.

Are they by any chance loaded separately?  Maybe it was just a slow load.

[Seth Pollack]

 Yes, the code examples use embedded github gists. which work really great, when they work ;-)

[Alex Kocharin]

 

I wonder how having different code in the editor and in the debugger helps productivity.

 

Oh come on... If you're using a compiler anyway, why not move to iced coffee or something? I'm avoiding it recently, but it's still better than streamline.

 

 

07.01.2014, 22:53, "Seth Pollack" <se...@pollackphoto.com>:

I've written up a blog post here http://bit.ly/KtMIYS that explains why we've chosen Node.js plus Streamline as our platform for our product (www.rivaliq.com). 

We are all Node fans in this forum, but i'm guessing that fewer folks have given Streamline (https://github.com/Sage/streamlinejs) a spin. We have found significant developer productivity (and happiness!) benefits by making the switch to using it. I'm not saying that here to troll for more "my control flow is better than your control flow" debates, but rather to share our experiences in order to hopefully help others. Please give it a read, and I welcome your comments on the blog. 

Seth

 

 

--

[Seth Pollack]

Hi Alex, please see my comment below about not trolling for more "my control flow is better than your control flow" debates ;-) 

All joking aside, there are many innovative approaches out there -- an embarrassment of riches. For folks who are into coffee script, streamline has full support for that as well. 

Also, i may not have been clear enough about the debugging experience under streamline. The code in the editor and debugger are the same. 

Seth

[Alex Kocharin]

 

I run "node_ test._js", then "node debug -p `pidof node`" (node_ doesn't have debug option), and that's the code I see:

 

```

debug> c
break in test._js:4
  2     debugger ;
  3     return setTimeout(__cb(_, __frame, 2, 0, function __$main() {
  4       debugger ;
  5       console.log("... world"); _(); }, true), 10000); });}).call(this, __trap);
  6 });
```

 

And this is an example stack trace:

 

```

Error: xx
    at __$main (/tmp/node_modules/zz/test._js:5:16)
    at ___ [as _onTimeout] (/tmp/node_modules/streamline/lib/callbacks/runtime.js:100:13)
    at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
```

 

Needless to say, I don't remember writing any of these underscore methods. And as I said earlier, I avoid compilers for this exact reason. Personally, I'd much rather stick to the callbacks and not trade writing time for debugging time. But if you're using a compiler anyway, I'd rather look for await/defer features of IcedCoffeeScript because it comes with much more obvious syntax.

 

PS: yes I know, it's possible to use fibers or generators or whatever with streamline. But I can use them without streamline just as well, and it'd be much easier.

 

 

08.01.2014, 05:55, "Seth Pollack" <se...@pollackphoto.com>:

[Seth Pollack]

The way we debug our code is via the WebStorm debugger, and we run Streamline with the fibers backend. This experience is very slick -- you are debugging the streamline source files, you can set breakpoints, step across async function invocations in a natural way without having to add a bunch of temporary breakpoints, etc. I've had several requests to demo this, so as soon as i get a bit of time i'll record a screencast that demonstrates this. 

If you are a command line debugger / callbacks backend person, streamline also has source map support (https://github.com/mozilla/source-map) to allow debugging via the source files. (although i haven't played around with it myself yet)

Seth

[Mark Hahn]

Can you debug node on a linux server remotely from webstorm on windows?

--

[Bruno Jouhier]

The async stack is reconstructed when you *catch* the exception (would be very inefficient otherwise). The normal pattern is to have at least one try catch in your high level HTTP dispatcher. You actually automatically get one if you use the companion streamline-streams or ez-streams companion libraries.

Try the following:

    function a(_) { return b(_); }
    function b(_) { return c(_); }
    function c(_) { return d(_); }
   
    function d(_) {
        setTimeout(_, 1000);
        throw new Error("xx");
    }
   
    try { a(_); }
    catch (ex) { throw ex.stack; }

Output:

Error: xx
    <<< async stack >>>
    at d (/Users/bruno/dev/syracuse/junk/stack._js:7:18)
    at c (/Users/bruno/dev/syracuse/junk/stack._js:3:23)
    at b (/Users/bruno/dev/syracuse/junk/stack._js:2:23)
    at a (/Users/bruno/dev/syracuse/junk/stack._js:1:23)
    at main (/Users/bruno/dev/syracuse/junk/stack._js:10:6)
    <<< raw stack >>>
    at __$d (/Users/bruno/dev/syracuse/junk/stack._js:7:18)
    at ___ [as _onTimeout] (/Users/bruno/dev/syracuse/node_modules/streamline/lib/callbacks/runtime.js:100:13)
    at Timer.listOnTimeout (timers.js:124:15)

[Alex Kocharin]

 

I'm not looking for async stack trace. If I were, I'd probably pick up "trycatch" module instead.

 

What I'm asking is that what is the point of writing javascript if you can't use it without compiling?

 

If you can use a compiler, you can use typescript, iced coffee and a dozen languages better than javascript. If you don't want to use a compiler, fibers and generators will help without any streamline stuff. So what is the point...

 

-----

 

I'm using callbacks only. People who don't understand them just can't work with node, so I have no regrets about that. The biggest drawback so far was that debugger can't step in async calls, don't know what to do about it.

 

If I wanted to write sync-like code, I'd probably turn to generators and "co" module. It looks nice, but I didn't actually manage to try it, since generators aren't on by default even in 0.11.x...

 

 

// alex

 

08.01.2014, 11:37, "Bruno Jouhier" <bjou...@gmail.com>:

[Seth Pollack]

I haven't tried that specific configuration (my dev environment is OSX), but webstorm does have a windows version, and does support remote debugging. 

[Seth Pollack]

Alex, no worries -- not everyone is optimizing for the same things. For our team, things like async stack traces have proven valuable to speed up debugging. 

The way we use streamline, we never have to explicitly compile. We just call require('streamline').register(...) at the top of our main module, and then the streamline runtime automatically re-compiles any streamline source files if needed (and persistently caches the results) as they are require()'d along the way. so while technically there is compilation going on, it's a JIT process, completely transparent and painless in my experience. In other words, the streamline language still feels like javascript, the streamline dev experience still feels like javascript (no explicit compile). 

Generators certainly seem to be the future for javascript. Streamline can target generators as one of its backends btw. However, as you point out, generators are still pretty early. As part of that, if you look at the performance tests (https://gist.github.com/bjouhier/5554200) that i referenced in the blog post, you can see that the results with generators lagged behind, at least as of when these tests were done. 

Seth

[Peter Rust]

Seth,

Thanks for the post and the write-up. It took me a year of full-time node programming -- all the while loving callbacks and caolan's async library -- until I finally reached some of the same conclusions. I landed on generators instead of streamline, but it's the same basic idea. None of the 5 async challenges you enumerate are horrible by themselves, but taken together, they add unnecessary friction, even for experienced devs.

-- peter

[Peter Rust]

Alex,

> The biggest drawback so far was that debugger can't step in async calls, don't know what to do about it

I always stick a breakpoint on the first line inside the callback and hit the run button, then remove the breakpoint. It's annoying -- especially if you need to step through a bunch of async calls -- but it works.

-- peter

[Seth Pollack]

I've uploaded a screencast of debugging Streamline source with WebStorm, here: http://www.youtube.com/watch?v=duC1Sqy66IE

Peter, it totally solves the temporary breakpoint problem btw. 

Seth

[Bruno Jouhier]

Alex,

I developed galaxy as a pure generators alternative to streamline, with the hope of switching to a pure JS solution in the future.

But after some experiments with it I decided to stay with streamline, simply because the syntax is more concise and plays better. For example, here is how you chain 3 async calls with streamline:
   

function asyncFn4(_) { return asyncFn1(_).asyncFn2(_).asyncFn3(_);  }

and the same thing with galaxy:
   

function* asyncFn4() { return (yield (yield (yield asyncFn1()).asyncFn2()).asyncFn3());  }

More keystrokes, more difficult to parse visually, more risks of mistakes (closing parenthesis in the wrong spot, forgetting a yield), etc.


And other generators libraries won't do any better because the problem comes from yield's precedence. They'll likely even require more code, like a wrapper around every async function definition.

Fibers, with the default futures library, preserves the nice chaining:

function asyncFn4() { return asyncFn1().wait().asyncFn2().wait().asyncFn3();  }

But this is less efficient than the corresponding streamline code (in fibers mode) because every asyncFnN() call allocates a future (streamline optimizes this case).

I'm probably a bit extreme in my quest for concise and composable syntax but these things matter when you deal with a big code base.

Bruno

[Jake Verbaten]

your asyncFn4 chaining problem can be solved with functional programming and composition

Rather then using `this` implicitely in chaining format you can pass the argument in.

```

function* asyncFn4(arg) {

    // I hope the yield order works like what you would expect here :/

    return (yield asyncFn3(yield asyncFn2(yield asyncFn1(arg)));

} 

// as seperate calls. pretty ugly but linear and not LISPY

function* asyncFn5(arg) {

    var res = yield asyncFn1(arg);

    var res2 = yield asyncFn2(res);

    var res3 = yield asyncFn3(res2);

    return res3;

}

// manually compose and call

function* asyncFn6(arg) {

    return yield* compose([asyncFn1, asyncFn2, asyncFn3])(arg)

}

// point free style function composition

var asyncFn6 = compose([asyncFn1, asyncFn2, asyncFn3])

// compose is really simple with yield in for loops.

function compose(fns) {

    return function* (result) {

        for (var i = 0; i < fns.length; i++) {

            result = yield fns[i](result);

        }

        return result

    }

}

```

--

[Bruno Jouhier]

Of course I can use all sorts of API/functional twists to improve things but:

1) Code will still be more difficult to read. For example, compare:

function asyncFn4(_) { return asyncFn1(_, a, b).asyncFn2(_, c, d, e).asyncFn3(_, f);  }

and:

function asyncFn4(_) { return yield(asyncFn3(yield asyncF2(yield asyncF1(a, b), c, d, e), f);  }

asyncFn3 and its argument (f) are completely split apart. And this is not even real code: things get worse when arguments are complex parameters containing parentheses (with more risks of parentheses pairing errors).

You can also do:

function asyncFn4(_) { return yield util.chain(["asyncFn1", a, b], ["asyncFn2", c, d, e], ["asyncFn3", f]);

Better. But to me this looks less like JavaScript (where are the functions? ah yes, those string literals!), more like code that a preprocessor would generate (and this is the kind of code that streamline generates in fibers and generators mode). And there is more noise (line gets longer).

2) Why should we have to twist our APIs to cope with deficiencies of our async tools? In sync-land you have the choice between target.fn(arg) and fn(target, args). Both have a very terse notation and target.fn(arg) just works better in many situations (array.filter(fn1).map(fn2).reduce(fn3) for example). Why should things be different in async-land? The goal of streamline is to allow you to use *all* JS constructs in a natural way, even if you are in async-land. It tries to keep the syntactic overhead as low as possible. Most of the time, the syntactic overhead is reduced to an extra _ parameter, and this is a very useful visual clue because it tells you where your code does yield to the event loop.

3) What about training overhead? If you have a growing team, you'll need to get newcomers up to speed and give them lots of guidelines (favor composition over chaining, use helper xyz). I'm not saying that streamline has 0 cost but once you know that _ means "wait" (and yield) and !_ means "don't wait" (and return a future on which you'll be able to wait later), you know most of it and you can start to use your existing JS skills to write async code.

But as I said earlier I probably have an obsession with the signal to noise ratio in code. If you find the alternatives ok (gap between asyncF3 and f, util.chain helper), then fine. I'm ready to pay the price of a transparent pre-processing pass to get the leanest code possible.

[Bruno Jouhier]

There is also the issue of being "future-proof".

Streamline gives me the choice between 3 runtime options: callbacks, fibers, generators. Each option has its own advantages and drawbacks: callbacks run everywhere, fibers are fast but have memory overhead, generators are "pure" JS and will probably get faster over time. The same source works with  these 3 runtimes.

But this may not be the end of the game. JS may get a true await keyword or some other syntactic sugar tomorrow (fn!(arg)). If this happens and the new syntax is highly optimized I can take advantage of it by tweaking the pre-processor. I don't have to commit myself to a specific runtime/async support library today. I'll be able to switch later. And BTW, this is not an academic case, this is what we did with our product when we decided ot deploy with the fibers option rather than the callback one.