best practice for real-time socket-based cross clustered-process data communication and computation

[Yi Tan]

Hi node mates:

I'm looking for your advice about how to design and implement an mechanism for real-time socket-based cross clustered-process data communication and computation. 

[The server layout]

clientA <--tcp socket--> node service A <--> DataModelA <--> unique radis data store

                                            

clientB <--tcp socket--> node service B <--> DataModelB <--> unique radis data store

* node service A and B are cluster on the same server

* 1 data model represent 1 client at the run time

* both node service A and B talk to the same redis data store

[The function request]

I need to implement a mechanism, in which:

1.  clients in difference processes can communication with each other efficiently.

2. there need to be a centralized place for data computation base on data models represents difference clients 

the logic looks like the following diagram:

clientA <--tcp socket--> node service A

                                            |

                                      socket pipe

                                            |

                                           v

clientB <--tcp socket--> node service B <--> DataModelA and B <--> unique radis data store

Do you know what is the best way to do this, or is there something already been built

Many thanks,

ty

[hd nguyen]

Did you take a look at cluster api?

http://nodejs.org/api/cluster.html#cluster_worker_send_message_sendhandle

I think we can send msg to master process, and from master process, broadcast msg to other workers.

More discussion can be referred in this link:  http://stackoverflow.com/questions/8534462/nodejscluster-how-to-send-data-from-master-to-to-all-or-single-child-workers 

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--
Nguyen Hai Duy
Mobile : 0914 72 1900
Yahoo: nguyenhd_lucky

[Yi Tan]

Hi  hd nguyen,

Thank you for the reply. 

Yes, I have tried cluster-worker message sending. There are 3 issues block me from using that built-in mechanism:

1. the app I'm working on is talking in raw buffer. but when sending buffer in node's built-in cluster, it actually transmit the SlowBuffer instance, which is 8k in fixed length. That means even sending 1 byte, the actually data transition needs be 16k (8k workerA->master, 8k master-> workerB). That is a huge waste.

2. the mechanism I'm looking for is a peer-to-peer implementation. the built-in cluster is a server-route implementation, which doubles the data cost of communication, another waste.

3. I can't find an easy way to pipe a stream to a process. Am I missing something?

Thank you,

ty

2012/9/10 hd nguyen <nguyen...@gmail.com>

[Duy Nguyen]

For communicating directly between processes in nodejs, I have not found/known any solution so far.

Btw, I'm not sure about "duplicating transfer data" as you mentioned, initially I think data packet just has been forwarded literally, no duplication (Am I  wrong here:-??).

 
And I think we can use redis pub/sub to communicate between processes (with Redis server as bridge).

Actually in my application now I'm using redis pubsub to communicate between servers, but in fact you can think each server is a process, so this is the solution for communicating between processes.

I use redis pubsub, socket.io with RedisStore to run  the app on multi servers (external duplication), each server is multi core (internal duplication), everything is quite good now.

[rubyonrailsx]

Did you consider remove the need for communication between service A and service B . Instead use redis as message pipe?

-- 

rubyonrailsx

Sent with Sparrow

[Yi Tan]

Thanks rubyonrailsx and Duy Nguyen,

Redis pub/sub had been tried, but it quickly become the performance bottleneck.

The app we are working on requires high performance capability. It handles about 10k client socket stream simultaneously on a 8-core linux box.

Averagely, each client stream send/receive 10 messages of 1kb in total buffer every seconds.

By using node cluster, we have managed to greatly reduce message process lag and cpu usage. Now la is low.

But the question we are facing is how to let 2 clients in different processes(workers) talk to each other efficiently.

I think the exact thing I'm looking for is a tcp or udp based cross-process peer-to-peer node js implementation.

BTW, regarding to Duy's question 

> Btw, I'm not sure about "duplicating transfer data" as you mentioned, initially I think data packet just has been forwarded literally, no duplication (Am I  wrong here:-??).

I haven't read node source of cluster, but if my assumption is correct. Nodejs has implement a js object serialization in its cluster implementation. In that way, cluster worker and master can talk to each other in js object. And data sent by process.send is serialised to buffer and sent to master. When the other end receive that buff, it parse the buffer back to js object. That's why when send a buffer instance, it will be serialised as an array. 

Regards,

ty

2012/9/10 rubyonrailsx <rubyon...@gmail.com>

[Johnny Honestly]

I heard about this cluster manager today directly from the guys at ebay working on ql.io

https://github.com/ql-io/cluster2

You can stream buffers from parents to child process using stdin/out

If you need sockets try dnode

https://github.com/substack/dnode

[Yi Tan]

Thanks Johnny,

dnode is very inspiring. cluster2 may help us a lot in production!

Regards,

ty

2012/9/11 Johnny Honestly <mostmo...@gmail.com>

[Dominic Tarr]

You need to make every "client" a tcp server.
if they live on the same machine, use a unix socket instead of a port number.
see the docs for "net".

Obviously, each node will need to keep track of every other node it
needs to communicate with. something like http://npm.im/seaport might
be good for that.

Also, what kind of messages are they sending?

[Yi Tan]

Thanks Dominic,

That's the way I'm implementing the cross-process module. 

One thing difference is, I'm using UDP datagram rather then TCP, because:

1. the service we are working on is lag-sensitive, but accuracy insensitive. 

2. domain socket is slow on Xen, learned from using Redis.

3. data of each transmission is very small, < 100bytes. 

> Also, what kind of messages are they sending?

message sent from clients are controlling command write in a custom binary protocol. 

the service we are building can be seen as to to host a good number of virtual rooms in several processes. And a few clients on different process can interact with each other in the same virtual room.

Regards,

ty

2012/9/13 Dominic Tarr <domini...@gmail.com>