Web Worker API

245 views
Skip to first unread message

Malte Ubl

unread,
Nov 9, 2009, 3:42:56 PM11/9/09
to nod...@googlegroups.com
Hi,

I made an initial hacky (and probably with respect to buffering)
implementation of the WebWorker API:
http://github.com/cramforce/node/commit/91c734c9d1285b36168d4c3059224cd391d0d922

I factored out the actual communication between workers into
implementation constructors which could be easily replaced to use a
socket based approach that can scale across multiple mashines.

Cheers
Malte

PS: Ryan: Thanks again for the great talk at JSConf

--
http://twitter.com/cramforce
http://nonblocking.io

Bluebie

unread,
Nov 9, 2009, 4:31:16 PM11/9/09
to nodejs
Reading through the code, I've a few comments. Firstly, strictly valid
json strings never contain a newline, so the MESSAGE_SPLITTER could
just as easily be a \n, which really looks rather nice for debugging.
Secondly, it'd probably run faster if the MESSAGE_SPLITTER were a
regexp, when used for things like this.buffer.split(), as it's my
understanding that passing a string in to that generally forces the
browser to compile it in to a regexp every time it's used.

Otherwise very nifty work!

On Nov 10, 7:42 am, Malte Ubl <malte....@gmail.com> wrote:
> Hi,
>
> I made an initial hacky (and probably with respect to buffering)
> implementation of the WebWorker API:http://github.com/cramforce/node/commit/91c734c9d1285b36168d4c3059224...

Ryan Dahl

unread,
Nov 10, 2009, 6:45:55 AM11/10/09
to nodejs
2009/11/9 Malte Ubl <malt...@gmail.com>:

>
> I made an initial hacky (and probably with respect to buffering)
> implementation of the WebWorker API:
> http://github.com/cramforce/node/commit/91c734c9d1285b36168d4c3059224cd391d0d922
>
> I factored out the actual communication between workers into
> implementation constructors which could be easily replaced to use a
> socket based approach that can scale across multiple mashines.

Cool! This is kind of a fundamental part of the program, so I need to
think about it for a bit. In particular I'm not sure if I want to do
the communication over STDIO. Unix sockets are going to be necessary
to pass file descriptors around (which postMessage allows), so I was
thinking about waiting until I implemented that to try workers. That
said, I think something over STDIO can be done now and changed later.

I have a few comments:

- I agree with Bluebie, can't you just use "\r\n"?

- test/mjsunit/workers/worker.js should be
test/mjsunit/fixtures/worker.js, I think

- 2 space indent, space between keywords ('for', 'if') and '('.
Missing some end of line ";"

- What is setInterval(function () {}, 10000) for?

- The test returns 0 but actually is failing:
-------------------------------------------------------------------------------------------------------
cloth 0 ~/projects/node > ./node test/mjsunit/test-worker.js
events.js:105

throw ret;

^


Error: Cannot find module '../common'
at node.js:366:30
at findModulePath (node.js:297:4)
at node.js:331:10
at EIOPromise.<anonymous> (node.js:269:31)
at Promise.wait (events.js:101:7)
at require (node.js:421:31)
at Object.<anonymous>
(/home/ryan/projects/node/test/mjsunit/workers/worker.js:1:79)
at Promise.<anonymous> (node.js:433:20)
at EIOPromise.<anonymous> (file.js:22:18)
at Promise.wait (events.js:101:7)

cloth 0 ~/projects/node >
-------------------------------------------------------------------------------------------------------
I guess this is because require.paths is not set correctly for the
child process. The environmental variable NODE_PATH might help.

ry

Malte Ubl

unread,
Nov 10, 2009, 6:59:03 AM11/10/09
to nod...@googlegroups.com
Hey Ryan,

thanks for the feedback. I'll see to fix the syntactic stuff soon.

Can you point me to the spec that says that things which can be
expressed outside of JSON (like file descriptors) can be passed around
across workers?

For now I always use JSON as the transfer format which is the same
implementation as in Firefox.

Cheers
Malte

Ryan Dahl

unread,
Nov 10, 2009, 7:05:24 AM11/10/09
to nod...@googlegroups.com
On Tue, Nov 10, 2009 at 12:59 PM, Malte Ubl <malt...@gmail.com> wrote:
>
> Hey Ryan,
>
> thanks for the feedback. I'll see to fix the syntactic stuff soon.
>
> Can you point me to the spec that says that things which can be
> expressed outside of JSON (like file descriptors) can be passed around
> across workers?

http://www.whatwg.org/specs/web-apps/current-work/multipage/comms.html#posting-messages-with-message-ports

> For now I always use JSON as the transfer format which is the same
> implementation as in Firefox.

That's okay. Passing "ports" (sockets) will be done with sendmsg()'s
ancillary messages.

Malte Ubl

unread,
Nov 10, 2009, 4:05:18 PM11/10/09
to nod...@googlegroups.com
Hey,

just two quick questions, before I continue:

On Tue, Nov 10, 2009 at 12:45 PM, Ryan Dahl <coldre...@gmail.com> wrote:
>

> 2009/11/9 Malte Ubl <malt...@gmail.com>:
>>
>> I made an initial hacky (and probably with respect to buffering)
>> implementation of the WebWorker API:
>> http://github.com/cramforce/node/commit/91c734c9d1285b36168d4c3059224cd391d0d922
>>
>> I factored out the actual communication between workers into
>> implementation constructors which could be easily replaced to use a
>> socket based approach that can scale across multiple mashines.
>
> Cool! This is kind of a fundamental part of the program, so I need to
> think about it for a bit. In particular I'm not sure if I want to do
> the communication over STDIO. Unix sockets are going to be necessary
> to pass file descriptors around (which postMessage allows), so I was
> thinking about waiting until I implemented that to try workers. That
> said, I think something over STDIO can be done now and changed later.
>
> I have a few comments:
>
> - I agree with Bluebie, can't you just use "\r\n"?

Much nicer, indeed.

> - test/mjsunit/workers/worker.js should be
> test/mjsunit/fixtures/worker.js, I think
>
> - 2 space indent, space between keywords ('for', 'if') and '('.
> Missing some end of line ";"

Are you sure about two spaces? Looked at the other code and it uses 1 space

> - What is setInterval(function () {}, 10000) for?

My stupid hack to keep the worker running when it is idle :)

Whats the correct way?

> - The test returns 0 but actually is failing:
> -------------------------------------------------------------------------------------------------------
> cloth 0 ~/projects/node > ./node test/mjsunit/test-worker.js
> events.js:105
>
>      throw ret;

<snip>

> Error: Cannot find module '../common'
>    at node.js:366:30
>    at findModulePath (node.js:297:4)
>    at node.js:331:10
>    at EIOPromise.<anonymous> (node.js:269:31)
>    at Promise.wait (events.js:101:7)
>    at require (node.js:421:31)
>    at Object.<anonymous>
> (/home/ryan/projects/node/test/mjsunit/workers/worker.js:1:79)
>    at Promise.<anonymous> (node.js:433:20)
>    at EIOPromise.<anonymous> (file.js:22:18)
>    at Promise.wait (events.js:101:7)
>
> cloth 0 ~/projects/node >
> -------------------------------------------------------------------------------------------------------
> I guess this is because require.paths is not set correctly for the
> child process. The environmental variable NODE_PATH might help.
>
>
>
> ry

--
http://twitter.com/cramforce
http://nonblocking.io

Malte Ubl

unread,
Nov 10, 2009, 4:19:58 PM11/10/09
to nod...@googlegroups.com
On Tue, Nov 10, 2009 at 10:05 PM, Malte Ubl <malt...@gmail.com> wrote:
> Hey,
>
> just two quick questions, before I continue:
>
> On Tue, Nov 10, 2009 at 12:45 PM, Ryan Dahl <coldre...@gmail.com> wrote:
<snip>

>> - 2 space indent, space between keywords ('for', 'if') and '('.
>> Missing some end of line ";"
>
> Are you sure about two spaces? Looked at the other code and it uses 1 space

Sorry, gotcha. Fixed :)

<snip>

Cheers
Malte

Ryan Dahl

unread,
Nov 11, 2009, 8:11:19 AM11/11/09
to nod...@googlegroups.com
On Tue, Nov 10, 2009 at 10:05 PM, Malte Ubl <malt...@gmail.com> wrote:
>
>> - What is setInterval(function () {}, 10000) for?
>
> My stupid hack to keep the worker running when it is idle :)
>
> Whats the correct way?

If you open stdin, the worker will idle without exiting until stdin is closed.

Malte Ubl

unread,
Dec 7, 2009, 4:52:33 PM12/7/09
to nod...@googlegroups.com
Hey,

just wanted to let you know that the Worker API implementation now mostly works:
http://github.com/cramforce/node/blob/master/lib/worker.js

There is still some debugging code left in and I know that Ryan is
looking for a more advanced implementation but maybe some people are
interested to play with this.

Here is some code for a pre forking web server using web workers:
http://pastebin.com/f7aa16f50

Cheers
Malte
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en
> -~----------~----~----~----~------~----~------~--~---

Ryan Dahl

unread,
Dec 8, 2009, 12:33:52 PM12/8/09
to nod...@googlegroups.com
On Mon, Dec 7, 2009 at 10:52 PM, Malte Ubl <malt...@gmail.com> wrote:
> Hey,
>
> just wanted to let you know that the Worker API implementation now mostly works:
> http://github.com/cramforce/node/blob/master/lib/worker.js
>
> There is still some debugging code left in and I know that Ryan is
> looking for a more advanced implementation but maybe some people are
> interested to play with this.

Yeah, I think I want to hold on with the worker stuff for a little
longer (I will be adding pipes soon, which I want to use for ports)
but I think your code is going to be a good base to build on.

> Here is some code for a pre forking web server using web workers:
> http://pastebin.com/f7aa16f50

AFAIK "pre-forking" servers are ones which open a server socket, fork
the process, and accept connections on each fork. A sort of automatic
load balancing. This seems more like a typical load balancing proxy
setup (like nginx passing requests off to other processes).

Ryan Dahl

unread,
Dec 8, 2009, 12:36:03 PM12/8/09
to nod...@googlegroups.com
On Mon, Dec 7, 2009 at 10:52 PM, Malte Ubl <malt...@gmail.com> wrote:
>
> just wanted to let you know that the Worker API implementation now mostly works:
> http://github.com/cramforce/node/blob/master/lib/worker.js

It's really nice. Would be cool to have it as a require()able library
for the time being

Malte Ubl

unread,
Dec 8, 2009, 3:31:20 PM12/8/09
to nod...@googlegroups.com
Hey
Thanks. Do you mean in core or as a separate distribution?

I have one question: Is there a way to explicitly flush buffers when
you've written something?
Right now my code needs things like
setTimeout(function () {
self.child.write(""+MESSAGE_SPLITTER, "utf8")
}, 100)
to have things appear on the other side.

Cheers
Malte

Ryan Dahl

unread,
Dec 18, 2009, 1:16:55 PM12/18/09
to nod...@googlegroups.com, Malte Ubl
Sorry for the late reply.

On 12/8/09, Malte Ubl <malt...@gmail.com> wrote:
> Thanks. Do you mean in core or as a separate distribution?

For the time being, separate. I'm getting nearer to being able to work
this stuff in - probably in a few weeks.

> I have one question: Is there a way to explicitly flush buffers when
> you've written something?
> Right now my code needs things like
> setTimeout(function () {
> self.child.write(""+MESSAGE_SPLITTER, "utf8")
> }, 100)
> to have things appear on the other side.

It could be that the child hasn't opened stdin yet? Not sure if there
is a good way around that...

Tim Caswell

unread,
Dec 18, 2009, 1:25:34 PM12/18/09
to nod...@googlegroups.com
I had the same problem with my wrapper around the sqlite3 binary. My solution was to issue the client a message and wait for the response before I deemed the connection ready. You'll probably have to implement some sort of handshake protocol on top of the existing interface. Once the worker responds to the greeting, then you can proceed to send stuff across the wire.

Initially I was going to file a feature request/bug that process.createChildProcess have a connect event, but then I realized that it was the sqlite3 binary's own processing that I had to wait for. The child process may be already connected, but if the chid process isn't ready for input yet, then it doesn't matter.

On Dec 18, 2009, at 12:16 PM, Ryan Dahl wrote:

>
>> I have one question: Is there a way to explicitly flush buffers when
>> you've written something?
>> Right now my code needs things like
>> setTimeout(function () {
>> self.child.write(""+MESSAGE_SPLITTER, "utf8")
>> }, 100)
>> to have things appear on the other side.
>
> It could be that the child hasn't opened stdin yet? Not sure if there
> is a good way around that...
>

> --


>
> You received this message because you are subscribed to the Google Groups "nodejs" group.

> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>

Reply all
Reply to author
Forward
0 new messages