Fwd: How batteries included is node supposed to be?

[Tom Robinson]

I forgot to CC us, but in case anyone is interested...

Begin forwarded message:

> From: Tom Robinson <tlrob...@gmail.com>
> Date: November 28, 2009 6:08:10 PM PST
> To: nodejs <nod...@googlegroups.com>
> Subject: Re: How batteries included is node supposed to be?
> Reply-To: nod...@googlegroups.com
>
> This seems like an appropriate place for me to chime in...
>
> I'd like to propose some sort of collaboration between Node and
> Narwhal (and CommonJS). So far their goals have been pretty
> orthogonal: Node is very focused on evented IO, while Narwhal is more
> focused on the "batteries included" stuff, package management, and
> CommonJS APIs. I think both are important for the success of
> JavaScript outside of the browser. I also believe server-side
> JavaScript hasn't been more popular in the past in part due to the
> lack of interoperability. Node's use of the CommonJS module system is
> a great start.
>
> Narwhal is designed to work with multiple JavaScript interpreters.
> Currently there are "engines" for Rhino, JavaScriptCore, and
> XULRunner. Node could probably replace Narwhal's JavaScriptCore
> engine. It would be great to have Node as the main Narwhal engine.
>
> Narwhal includes a package manager which works well (packages are
> trivial to host on GitHub) and has quite a few packages already. Many
> of the packages don't require CommonJS's synchronous APIs and thus
> could be useful in Node. At some point we'll switch to a shared
> CommonJS package repository.
>
> In addition to CommonJS modules and testing there are a few areas
> where I'd like to see Node work with CommonJS on standardization, in
> particular binary data types, promises, packages, and asynchronous
> JSGI. I'd also like to see Node's asynchronous APIs (or something
> similar) make their way into CommonJS.
>
> I know Ryan wants to focus on Node's core functionality, but if anyone
> else is interested in working on unifying these projects let me know.
>
> -tom
>
> --
> tlrobinson{.net,@gmail.com, on twitter}
>
>
> On Nov 28, 10:15 am, Karl Guertin <grayr...@gmail.com> wrote:
>> I've been working on the CommonJS conversion for the tests* and I'm
>> writing/planning to write a reasonable amount of generally reusable
>> code for the runner: color term escaping, option parsing, utility
>> functions that JS is missing and every JS library adds back in (e.g.
>> about 60% of narwhal/lib/util.js), etc. My question is whether this
>> sort of thing belongs in node's standard modules or whether I should
>> package it all up as my own library.
>>
>> I know the goal for node is to be low level, but what is low level? I
>> come from Python, so my idea of a stdlib would include all of the
>> above and they're all pretty useful regardless of what you're making.
>> I know there's a line, e.g. a port of SproutCore's data layer would
>> be
>> right out, but I'm not sure of the line's location.
>>
>> * assert.js is done, converted from Thomas' implementation in
>> narwhal,
>> but is untested due to my desire to dogfood. I've got good size
>> chunks
>> of the rest done. I'll have it out on github once assert and the
>> runner are working and before I get to converting the current unit
>> tests.
>
> --
>
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com
> .
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en
> .
>
>

[Kris Zyp]

I completely agree, the potential of synergy of these projects
collaborating would be huge. Fortunately Narwhal has moved towards
shared-nothing event-loop concurrency, just like Node, so we are on the
same track. I'd be glad to help with the stuff that I have worked on for
Narwhal (which happens to be some of Narwhal's async stuff). More
specifically:
* I could write a JSGI (0.3) shim for Node. Node and JSGI have
everything in place to make this very easy (and preserve the async
nature of Node), and Node's lower-level style stream-based API is
perfect for layering JSGI on top of.

* promises - Narwhal has a promise module that provides a number of
utility functions for creating and working with promises. This is not
engine specific and should be available anywhere (including Node),
except that Node currently has a divergent promise API. I am not sure if
this is because Node is just a few conversations behind on the CommonJS
promise spec (their API seems to reflect an earlier proposal) or if it
is based on an earlier generation promise-based framework like Twisted
or Dojo. Either way, it should be very easy for Node to upgrade to the
CommonJS promise module (just provide a 'then' method), and still
preserve backwards compatibility. Basically to extend what already
exists in Node, I think you would do:
Promise.prototype.then = function(successCallback, errorCallback){
var promise = new process.Promise();
this.addCallback(function(){
successCallback.call(this, arguments);
promise.emitSuccess.call(promise, arguments);
});
this.addErrback(function(){
errorCallback.call(this, arguments);
promise.emitError.call(promise, arguments);
});
return promise;
}

* event-queue module - Narwhal has an event-queue module that allows you
to interact with the event-queue, and this seems to work well on the
event-loop system for Rhino as well Mozilla's XPCom. We have generally
taken an explicit-is-better philosophy to the event-loop, rather than
having it be completely magical. It would really be nice if at least
some of this functionality was exposed (explicitly processing events,
like what promise.wait() does, except explicitly), using a platform
specific version of the same module and API.

* worker module - Does V8/Node already have workers, or do you have to
use ChildProcess with stream communication? Either way, it should be
very doable to provide a worker module (implementing the W3C WebWorkers
API) that wraps whatever Node uses.

Anyway, I am looking forward to running Pintura on Node at some point,
so I am very eager to see this convergence take place.

Thanks,
Kris

> You received this message because you are subscribed to the Google Groups "Narwhal and Jack" group.
> To post to this group, send email to narw...@googlegroups.com.
> To unsubscribe from this group, send email to narwhaljs+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/narwhaljs?hl=en.
>
>
>

[Neville Burnell]

> Anyway, I am looking forward to running Pintura on Node at some point,
> so I am very eager to see this convergence take place.

awesome Kris!

[Christoph Dorn]

+1 I am on board too and am looking forward to contributing in any way I
can. Still have to finish the new tusk though ...

I just finished watching Ryan's talk. The internals are over my head but
I am wondering if we can expose the nodejs threadpool/piping API to
allow us to wrap existing sync IO APIs in an async way in JavaScript to
drive the ecosystem of available packages/bindings for the platform
(even if not as efficient as native nodejs code).

Christoph

[Kris Zyp]

Yes, you can turn existing sync APIs into async APIs by using a wrapper
that spawns a thread and queues up the result in the event queue after
completing. The event-queue module's enqueue function is thread-safe for
this purpose. And yes, this does defeat the thread conservation
advantage of true async architecture. I'd be glad to write a conversion
module (for Rhino) for creating async functions from sync ones (and vice
versa).

Kris

[Tom Robinson]

Of course we can also just implement the async APIs directly using
Java NIO, right? I might do that at some point.

[Ryan Dahl]

On Mon, Nov 30, 2009 at 1:49 AM, Christoph Dorn
<christ...@christophdorn.com> wrote:
> I just finished watching Ryan's talk. The internals are over my head but
> I am wondering if we can expose the nodejs threadpool/piping API to
> allow us to wrap existing sync IO APIs in an async way in JavaScript to
> drive the ecosystem of available packages/bindings for the platform
> (even if not as efficient as native nodejs code).

The thread pool is only for executing little bits of C, you cannot
execute javascript there. Exposing some of the pipes can be done,
though.

[Neville Burnell]

>
> The thread pool is only for executing little bits of C, you cannot
> execute javascript there. Exposing some of the pipes can be done,
> though.

But you are planning to provide WebWorkers in node right? They could
execute CommonJS javascript code async.

[Ryan Dahl]

Yes, but that will be with a separate process, I think. I was just
talking about Node's thread pool.

[Kris Zyp]

And its vastly more efficient to start with native async APIs and create
sync APIs from them (with event-queue spinners), than starting with sync
APIs and trying to use threads or workers to create async APIs.
Kris

[Kris Zyp]

When I finish the JSGI shim for Node, would that be better suited to
be a module for Node or for Jack or a standalone project (or something
else entirely)?
Kris

[Tom Robinson]

Good question. I'm not sure where to draw the line for handlers
bundled with Jack.

But for Node, since Narwhal and Jack aren't yet working on Node, I'd
probably keep it separate for now.

-tom

--
tlrobinson{.net,@gmail.com, on twitter}

> --
>
> You received this message because you are subscribed to the Google
> Groups "Narwhal and Jack" group.
> To post to this group, send email to narw...@googlegroups.com.
> To unsubscribe from this group, send email to narwhaljs+...@googlegroups.com

> .
> For more options, visit this group at http://groups.google.com/group/narwhaljs?hl=en
> .
>
>

[Christoph Dorn]

It would be great if we can write modules against an async narwhal API
and be able to use them with a sync engine such as the current rhino if
not deploying on node or other async platform.

Node is going to collect a lot of functionality packages very fast.
Ideally these will conform to the CommonJS standards and thus be
supported by tusk.

If we can tag packages as async compatible we can offer async and sync
catalogs of packages.

Christoph

[Ray Morgan]

How will you be handling the input stream from Node.js since it uses
an EventEmitter that emits an event for each packet received?

[Tom Robinson]

I plan to support asynchronous APIs (whether it's Node's or something
defined by CommonJS) at some point.

-tom

On Dec 1, 2009, at 11:35 PM, Ray Morgan wrote:

> How will you be handling the input stream from Node.js since it uses
> an EventEmitter that emits an event for each packet received?
>