99.999 node

[John Wright]

I was talking to a programmer from Ericsson this weekend who had
never heard of node and he works on embedded systems with 99.99
uptime. The framework they use is based on a core "supervisor" like
process written in their own assembly language that can monitor
thousands of concurrent "processes" written in C that report back
asynchronously. If any of them die they report their error and are
restarted Erlang style. I really think the average web developer
like me has no clue about building this level of fault tolerant
systems but node is allowing whole new classes of applications where
it could be useful. In my case I am build a node based product that
runs continuously on the user's desktop (doing some useful things of
course :) ). I would love to learn more about what is involved in
making sure my modules don't crash node and I wonder if some sort of
supervisor, error handling could be built into node of which hot code
loading would obviously pay a big role. The patch from Blaine/Felix
is really cool and I can imagine pushing changes to my users machines
by having node polling for changes and reloading itself without a
restart. Sorry for the naiive post but wanted to know if there was
general interest in pulling some of the best stuff from Erlang and
other 99.99 sources and building something directly into node that
would really make it as core of a feature as async.

Imagine: "So what is so great about node? It's a screaming fast non
blocking VM built on top of V8 that never crashes".

[Elijah Insua]

John,

I completely agree, however In my experience it is generally userland code that causes these crashes, unhandled exceptions, which can be avoided if coded properly. 
You say you are writing some desktop applications? what sort of things are you working on?

-- Elijah

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

[Stephen Belanger]

I agree that more fault tolerance would be nice. Typically errors big enough to crash an app are minor oversights that are rarely seen, otherwise they'd get filtered out in testing. Errors like these can be rather problematic though, if your entire system goes down because only a specific action couldn't be executed properly. A good example is web development project I was working on awhile back. It was very Javascript heavy and seemed to have randomly occurring blank screen problems in Safari--it turns out the cause was something totally obscure; Safari reserves the word 'class', so it can't be used as a variable name. 

[Elijah Insua]

I believe there has been some talk about pre-forking node into daemon mode. Perhaps the actual fork is the "master" process which forks off the actual copy of node?

-- Elijah

[CC]

I agree with the fact that Erlang style included in Node could be very
interesting.

Code properly ok, but there is always mistakes in codes. That's why
Erlang impose/propose some behaviors to make your app robust like the
rock.

2010/2/22 Elijah Insua <tmp...@gmail.com>:

[christkv]

One problem guys :D

Erlang can do this magic because all variables are final (can only be
assigned once) and "threads" only take up around 160 bytes of memory
and are green threaded. The language enforces this so there's no
conflict. I have no idea how you would solve this in node :(. It would
require a very strict coding style I think to do it flawlessly and as
we don't have threads right now the supervisor pattern would be hard
to implement.

I personally use node for it's server strengths and then the backend
stuff is in erlang connected via bert-rpc

Christian

On Feb 22, 11:09 am, CC <comandoc...@gmail.com> wrote:
> I agree with the fact that Erlang style included in Node could be very
> interesting.
>
> Code properly ok, but there is always mistakes in codes. That's why
> Erlang impose/propose some behaviors to make your app robust like the
> rock.
>
> 2010/2/22 Elijah Insua <tmp...@gmail.com>:
>
>
>
> > I believe there has been some talk about pre-forking node into daemon mode.
> > Perhaps the actual fork is the "master" process which forks off the actual
> > copy of node?
>
> > -- Elijah
>

> > On Mon, Feb 22, 2010 at 1:16 AM, Stephen Belanger <cyruzdr...@gmail.com>

> > wrote:
>
> >> I agree that more fault tolerance would be nice. Typically errors big
> >> enough to crash an app are minor oversights that are rarely seen, otherwise
> >> they'd get filtered out in testing. Errors like these can be rather
> >> problematic though, if your entire system goes down because only a specific
> >> action couldn't be executed properly. A good example is web development
> >> project I was working on awhile back. It was very Javascript heavy and
> >> seemed to have randomly occurring blank screen problems in Safari--it turns
> >> out the cause was something totally obscure; Safari reserves the word
> >> 'class', so it can't be used as a variable name.
>
> >> On Sun, Feb 21, 2010 at 10:04 PM, Elijah Insua <tmp...@gmail.com> wrote:
>
> >>> John,
>
> >>> I completely agree, however In my experience it is generally userland
> >>> code that causes these crashes, unhandled exceptions, which can be avoided
> >>> if coded properly.
> >>> You say you are writing some desktop applications? what sort of things
> >>> are you working on?
>
> >>> -- Elijah
>

> >>> On Mon, Feb 22, 2010 at 12:14 AM, John Wright <mrjjwri...@gmail.com>

[Hagen]

Erlang can do this magic because all variables are final (can only be
assigned once) and "threads" only take up around 160 bytes of memory
and are green threaded. The language enforces this so there's no
conflict. I have no idea how you would solve this in node :(. It would
require a very strict coding style I think to do it flawlessly and as
we don't have threads right now the supervisor pattern would be hard
to implement.

Actually, we are getting there. ECMAScript 5 defines Object.freeze() ( see http://ejohn.org/blog/ecmascript-5-objects-and-properties/ ) which gets us a long way towards what erlang does.

Applying Erlang to Node would result in creating a new event loop being as cheap as creating a new object right now.

When objects passed from one event loop to the other are forcefully frozen, it should be possible to have them all run on a thread pool instead of processes with serialized JSON in-between them.

WANT! Pretty please with sugar on top ;-)

[billywhizz]

there's no reason why node couldn't be integrated into a 5-9's uptime
system, in fact i am working on such a system at the moment, but it
won't be anything like erlang as a node process is not lightweight
enough to be able to spawn millions or even thousands of them without
running out of memory. erlang processes are not the same as OS
processes (as christkv points out above), so an "erlangy" architecture
is not going to be anything like one built around node...

On Feb 22, 11:09 am, Hagen <six...@gmail.com> wrote:
> > Erlang can do this magic because all variables are final (can only be
> > assigned once) and "threads" only take up around 160 bytes of memory
> > and are green threaded. The language enforces this so there's no
> > conflict. I have no idea how you would solve this in node :(. It would
> > require a very strict coding style I think to do it flawlessly and as
> > we don't have threads right now the supervisor pattern would be hard
> > to implement.
>

> Actually, we are getting there. ECMAScript 5 defines Object.freeze() ( seehttp://ejohn.org/blog/ecmascript-5-objects-and-properties/) which gets us a

[John Wright]

My friend from Ericsson pointed out that unless the language supports
the necessary primitives it's kind of pointless to try to emulate
Erlang. His embedded system also has very small footprint per
"thread" and they use something lighter weight and faster than
signals, perhaps hardware interrupts, to communicate back to a
supervisor process.

This being said, Billywhiz can you elaborate on what a 5-9's uptime
system might look like with node. There should be some general error
handling principles from these kind of 5-9 uptime systems that are
useful to node. Perhaps this is a good candidate library for node,
and could be come a style of working with node.

[Ryan Dahl]

On Mon, Feb 22, 2010 at 8:34 PM, John Wright <mrjjw...@gmail.com> wrote:
> My friend from Ericsson pointed out that unless the language supports
> the necessary primitives it's kind of pointless to try to emulate
> Erlang.  His embedded system also has very small footprint per
> "thread" and they use something lighter weight and faster than
> signals, perhaps hardware interrupts, to communicate back to a
> supervisor process.

In Erlang you need one "process" per connection - and it's good that
it has a small footprint. Node handles many connections on one process
in one thread - the footprint for a single connection is also very
small.

> This being said,  Billywhiz can you elaborate on what a 5-9's uptime
> system might look like with node.  There should be some general error
> handling principles from these kind of 5-9 uptime systems that are
> useful to node.  Perhaps this is a good candidate library for node,
> and could be come a style of working with node.

A high uptime system with node would look like a supervisor process
managing a pool of workers. For servers, the workers would share the
listening socket and each accept connections from it (a so-called
'pre-fork' server). When one of the workers crashed, all connections
and state to that worker would be lost - but the damage would be
isolated from the other workers because of they are implemented as
OS-level processes. The supervisor/master process would notice that a
worker died and restart it - perhaps also sending some messages (in
the form of an event emitted from process) to all the workers that one
had died and is being restarted. Assuming some time was spent to make
the master process simple and bug free, such a system could stay up
for a long time. One could imagine things where updates to the server
code could be hot loaded by forcing workers, one at time, to stop
accepting connections and finally exit. When the restarted, the new
code would be used.

All this sounds like a bunch of fun and where I'd like to see Node end
up. It would all be approchable once there are better IPC bindings,
particularly sending ancillary data in sendmsg. (net2 supports this
but it's future is unclear - so it might have to be rewritten.)

[John Wright]

Thanks Ryan!

Here are some examples of pre-forking servers written in other
languages that can be ported to Node as a first step in getting a
rough-rough prototype of this going:

http://www.bluishcoder.co.nz/2010/02/20/pure-preforking-echo-server-example.html

Writing a pre-forking server in node looks pretty damn easy, I could
be wrong, maybe even I could do it.

Now for monitoring the worker processes, erlang style:

Here is a link of somebody who tried to build an erlang like
supervisor architecture in Scala with lots of good information:
http://jonasboner.com/2008/06/16/erlang-style-supervisor-module-for-scala-actors/.

Here is the man page on sendmsg, with a discussion of sending
ancillary data: http://www.kernel.org/doc/man-pages/online/pages/man2/sendmsg.2.html.

Finally If freezing Javascript objects is interesting, important or
relevant here is a link to John Resig's blog post from today about
this feature (supported in node's newest V8 version):
http://ejohn.org/blog/ecmascript-5-objects-and-properties

Anybody else who knows of other good links send them on so those of
us who are interested can start researching these types of
architectures.

On Feb 22, 9:49 pm, Ryan Dahl <coldredle...@gmail.com> wrote:

[Elijah Insua]

On Tue, Feb 23, 2010 at 12:36 AM, John Wright <mrjjw...@gmail.com> wrote:

Thanks Ryan!

Here are some examples of pre-forking servers written in other
languages that can be ported to Node as a first step in getting a
rough-rough prototype of this going:

http://www.bluishcoder.co.nz/2010/02/20/pure-prPosted: May 21st, 2009eforking-echo-server-example.html

Writing a pre-forking server in node looks pretty damn easy, I could
be wrong, maybe even I could do it.

Now for monitoring the worker processes, erlang style:

Here is a link of somebody who tried to build an erlang like
supervisor architecture in Scala with lots of good information:

http://jonasboner.com/2008/06/16/erlang-style-supervisor-moduPosted: May 21st, 2009le-for-scala-actors/.

Here is the man page on sendmsg, with a discussion of sending
ancillary data: http://www.kernel.org/doc/man-pages/online/pages/man2/sendmsg.2.html.

Finally If freezing Javascript objects is interesting, important or
relevant here is a link to John Resig's blog post from today about
this feature (supported in node's newest V8 version):

http://ejohn.org/blog/ecmascript-5-objects-and-properties

 

Posted: May 21st, 2009 -- not to mention, there are compatibility concerns. (if anyone cares about commonjs)

 

[Marco Rogers]

I'm out of my depth here, but it sounds like you guys are talking
about something like Unicorn from the Ruby world.

http://unicorn.bogomips.org/

It's used by Github and they have a lot of good articles about how
they're architecture benefits from it.

http://github.com/blog/517-unicorn

On Feb 23, 1:55 am, Elijah Insua <tmp...@gmail.com> wrote:

> On Tue, Feb 23, 2010 at 12:36 AM, John Wright <mrjjwri...@gmail.com> wrote:
> > Thanks Ryan!
>
> > Here are some examples of pre-forking servers written in other
> > languages that can be ported to Node as a first step in getting a
> > rough-rough prototype of this going:
>

> >http://www.bluishcoder.co.nz/2010/02/20/pure-pr*Posted:*May 21st, 2009
> > eforking-echo-server-example.html<http://www.bluishcoder.co.nz/2010/02/20/pure-preforking-echo-server-e...>

>
> > Writing a pre-forking server in node looks pretty damn easy, I could
> > be wrong, maybe even I could do it.
>
> > Now for monitoring the worker processes, erlang style:
>
> > Here is a link of somebody who tried to build an erlang like
> > supervisor architecture in Scala with lots of good information:

> >http://jonasboner.com/2008/06/16/erlang-style-supervisor-modu*Posted:*May
> > 21st, 2009le-for-scala-actors/<http://jonasboner.com/2008/06/16/erlang-style-supervisor-module-for-s...>

> > .
>
> > Here is the man page on sendmsg, with a discussion of sending
> > ancillary data:
> >http://www.kernel.org/doc/man-pages/online/pages/man2/sendmsg.2.html.
>
> > Finally If freezing Javascript objects is interesting, important or
> > relevant here is a link to John Resig's blog post from today about
> > this feature (supported in node's newest V8 version):
> >http://ejohn.org/blog/ecmascript-5-objects-and-properties
>

> *Posted:* May 21st, 2009 -- not to mention, there are compatibility

> > nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>