Web Worker API
Malte Ubl
I made an initial hacky (and probably with respect to buffering)
implementation of the WebWorker API:
http://github.com/cramforce/node/commit/91c734c9d1285b36168d4c3059224cd391d0d922
I factored out the actual communication between workers into
implementation constructors which could be easily replaced to use a
socket based approach that can scale across multiple mashines.
Cheers
Malte
PS: Ryan: Thanks again for the great talk at JSConf
Bluebie
json strings never contain a newline, so the MESSAGE_SPLITTER could
just as easily be a \n, which really looks rather nice for debugging.
Secondly, it'd probably run faster if the MESSAGE_SPLITTER were a
regexp, when used for things like this.buffer.split(), as it's my
understanding that passing a string in to that generally forces the
browser to compile it in to a regexp every time it's used.
Otherwise very nifty work!
On Nov 10, 7:42 am, Malte Ubl <malte....@gmail.com> wrote:
> Hi,
>
> I made an initial hacky (and probably with respect to buffering)
Ryan Dahl
>
> I made an initial hacky (and probably with respect to buffering)
> implementation of the WebWorker API:
> http://github.com/cramforce/node/commit/91c734c9d1285b36168d4c3059224cd391d0d922
>
> I factored out the actual communication between workers into
> implementation constructors which could be easily replaced to use a
> socket based approach that can scale across multiple mashines.
Cool! This is kind of a fundamental part of the program, so I need to
think about it for a bit. In particular I'm not sure if I want to do
the communication over STDIO. Unix sockets are going to be necessary
to pass file descriptors around (which postMessage allows), so I was
thinking about waiting until I implemented that to try workers. That
said, I think something over STDIO can be done now and changed later.
I have a few comments:
- I agree with Bluebie, can't you just use "\r\n"?
- test/mjsunit/workers/worker.js should be
test/mjsunit/fixtures/worker.js, I think
- 2 space indent, space between keywords ('for', 'if') and '('.
Missing some end of line ";"
- What is setInterval(function () {}, 10000) for?
- The test returns 0 but actually is failing:
-------------------------------------------------------------------------------------------------------
cloth 0 ~/projects/node > ./node test/mjsunit/test-worker.js
events.js:105
throw ret;
^
Error: Cannot find module '../common'
at node.js:366:30
at findModulePath (node.js:297:4)
at node.js:331:10
at EIOPromise.<anonymous> (node.js:269:31)
at Promise.wait (events.js:101:7)
at require (node.js:421:31)
at Object.<anonymous>
(/home/ryan/projects/node/test/mjsunit/workers/worker.js:1:79)
at Promise.<anonymous> (node.js:433:20)
at EIOPromise.<anonymous> (file.js:22:18)
at Promise.wait (events.js:101:7)
cloth 0 ~/projects/node >
-------------------------------------------------------------------------------------------------------
I guess this is because require.paths is not set correctly for the
child process. The environmental variable NODE_PATH might help.
ry
Malte Ubl
thanks for the feedback. I'll see to fix the syntactic stuff soon.
Can you point me to the spec that says that things which can be
expressed outside of JSON (like file descriptors) can be passed around
across workers?
For now I always use JSON as the transfer format which is the same
implementation as in Firefox.
Cheers
Malte
Ryan Dahl
>
> Hey Ryan,
>
> thanks for the feedback. I'll see to fix the syntactic stuff soon.
>
> Can you point me to the spec that says that things which can be
> expressed outside of JSON (like file descriptors) can be passed around
> across workers?
> For now I always use JSON as the transfer format which is the same
> implementation as in Firefox.
That's okay. Passing "ports" (sockets) will be done with sendmsg()'s
ancillary messages.
Malte Ubl
just two quick questions, before I continue:
On Tue, Nov 10, 2009 at 12:45 PM, Ryan Dahl <coldre...@gmail.com> wrote:
>
> 2009/11/9 Malte Ubl <malt...@gmail.com>:
>>
>> I made an initial hacky (and probably with respect to buffering)
>> implementation of the WebWorker API:
>> http://github.com/cramforce/node/commit/91c734c9d1285b36168d4c3059224cd391d0d922
>>
>> I factored out the actual communication between workers into
>> implementation constructors which could be easily replaced to use a
>> socket based approach that can scale across multiple mashines.
>
> Cool! This is kind of a fundamental part of the program, so I need to
> think about it for a bit. In particular I'm not sure if I want to do
> the communication over STDIO. Unix sockets are going to be necessary
> to pass file descriptors around (which postMessage allows), so I was
> thinking about waiting until I implemented that to try workers. That
> said, I think something over STDIO can be done now and changed later.
>
> I have a few comments:
>
> - I agree with Bluebie, can't you just use "\r\n"?
Much nicer, indeed.
> - test/mjsunit/workers/worker.js should be
> test/mjsunit/fixtures/worker.js, I think
>
> - 2 space indent, space between keywords ('for', 'if') and '('.
> Missing some end of line ";"
Are you sure about two spaces? Looked at the other code and it uses 1 space
> - What is setInterval(function () {}, 10000) for?
My stupid hack to keep the worker running when it is idle :)
Whats the correct way?
> - The test returns 0 but actually is failing:
> -------------------------------------------------------------------------------------------------------
> cloth 0 ~/projects/node > ./node test/mjsunit/test-worker.js
> events.js:105
>
> throw ret;
<snip>
> Error: Cannot find module '../common'
> at node.js:366:30
> at findModulePath (node.js:297:4)
> at node.js:331:10
> at EIOPromise.<anonymous> (node.js:269:31)
> at Promise.wait (events.js:101:7)
> at require (node.js:421:31)
> at Object.<anonymous>
> (/home/ryan/projects/node/test/mjsunit/workers/worker.js:1:79)
> at Promise.<anonymous> (node.js:433:20)
> at EIOPromise.<anonymous> (file.js:22:18)
> at Promise.wait (events.js:101:7)
>
> cloth 0 ~/projects/node >
> -------------------------------------------------------------------------------------------------------
> I guess this is because require.paths is not set correctly for the
> child process. The environmental variable NODE_PATH might help.
>
>
>
> ry
Malte Ubl
> Hey,
>
> just two quick questions, before I continue:
>
> On Tue, Nov 10, 2009 at 12:45 PM, Ryan Dahl <coldre...@gmail.com> wrote:
>> - 2 space indent, space between keywords ('for', 'if') and '('.
>> Missing some end of line ";"
>
> Are you sure about two spaces? Looked at the other code and it uses 1 space
Sorry, gotcha. Fixed :)
<snip>
Cheers
Malte
Ryan Dahl
>
> My stupid hack to keep the worker running when it is idle :)
>
> Whats the correct way?
If you open stdin, the worker will idle without exiting until stdin is closed.
Malte Ubl
just wanted to let you know that the Worker API implementation now mostly works:
http://github.com/cramforce/node/blob/master/lib/worker.js
There is still some debugging code left in and I know that Ryan is
looking for a more advanced implementation but maybe some people are
interested to play with this.
Here is some code for a pre forking web server using web workers:
http://pastebin.com/f7aa16f50
Cheers
Malte
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en
> -~----------~----~----~----~------~----~------~--~---
Ryan Dahl
> Hey,
>
> just wanted to let you know that the Worker API implementation now mostly works:
> http://github.com/cramforce/node/blob/master/lib/worker.js
>
> There is still some debugging code left in and I know that Ryan is
> looking for a more advanced implementation but maybe some people are
> interested to play with this.
longer (I will be adding pipes soon, which I want to use for ports)
but I think your code is going to be a good base to build on.
AFAIK "pre-forking" servers are ones which open a server socket, fork
the process, and accept connections on each fork. A sort of automatic
load balancing. This seems more like a typical load balancing proxy
setup (like nginx passing requests off to other processes).
Ryan Dahl
>
> just wanted to let you know that the Worker API implementation now mostly works:
> http://github.com/cramforce/node/blob/master/lib/worker.js
for the time being
Malte Ubl
I have one question: Is there a way to explicitly flush buffers when
you've written something?
Right now my code needs things like
setTimeout(function () {
self.child.write(""+MESSAGE_SPLITTER, "utf8")
}, 100)
to have things appear on the other side.
Cheers
Malte
Ryan Dahl
On 12/8/09, Malte Ubl <malt...@gmail.com> wrote:
> Thanks. Do you mean in core or as a separate distribution?
For the time being, separate. I'm getting nearer to being able to work
this stuff in - probably in a few weeks.
> I have one question: Is there a way to explicitly flush buffers when
> you've written something?
> Right now my code needs things like
> setTimeout(function () {
> self.child.write(""+MESSAGE_SPLITTER, "utf8")
> }, 100)
> to have things appear on the other side.
It could be that the child hasn't opened stdin yet? Not sure if there
is a good way around that...
Tim Caswell
Initially I was going to file a feature request/bug that process.createChildProcess have a connect event, but then I realized that it was the sqlite3 binary's own processing that I had to wait for. The child process may be already connected, but if the chid process isn't ready for input yet, then it doesn't matter.
On Dec 18, 2009, at 12:16 PM, Ryan Dahl wrote:
>
>> I have one question: Is there a way to explicitly flush buffers when
>> you've written something?
>> Right now my code needs things like
>> setTimeout(function () {
>> self.child.write(""+MESSAGE_SPLITTER, "utf8")
>> }, 100)
>> to have things appear on the other side.
>
> It could be that the child hasn't opened stdin yet? Not sure if there
> is a good way around that...
>
> --
>
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>