Received: by 10.58.187.168 with SMTP id ft8mr5236486vec.7.1347287200432;
Mon, 10 Sep 2012 07:26:40 -0700 (PDT)
X-BeenThere: nodejs@googlegroups.com
Received: by 10.220.154.77 with SMTP id n13ls1793012vcw.4.gmail; Mon, 10 Sep
2012 07:26:30 -0700 (PDT)
Received: by 10.52.88.169 with SMTP id bh9mr4544452vdb.6.1347287190458;
Mon, 10 Sep 2012 07:26:30 -0700 (PDT)
Received: by 10.52.88.169 with SMTP id bh9mr4544451vdb.6.1347287190438;
Mon, 10 Sep 2012 07:26:30 -0700 (PDT)
Return-Path: <yi2...@gmail.com>
Received: from mail-vc0-f169.google.com (mail-vc0-f169.google.com [209.85.220.169])
by gmr-mx.google.com with ESMTPS id ef10si3131876vdb.3.2012.09.10.07.26.30
(version=TLSv1/SSLv3 cipher=OTHER);
Mon, 10 Sep 2012 07:26:30 -0700 (PDT)
Received-SPF: pass (google.com: domain of yi2...@gmail.com designates 209.85.220.169 as permitted sender) client-ip=209.85.220.169;
Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of yi2...@gmail.com designates 209.85.220.169 as permitted sender) smtp.mail=yi2...@gmail.com; dkim=pass header...@gmail.com
Received: by mail-vc0-f169.google.com with SMTP id fl13so1419550vcb.28
for <nodejs@googlegroups.com>; Mon, 10 Sep 2012 07:26:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:in-reply-to:references:from:date:message-id:subject:to
:content-type;
bh=8aMxx7dFglaCPD2Sa8M7386Udd59nnGBxSr/UxFzrDE=;
b=v55hMlSoyIiDbCMHEPfqf/0oiuf9umkmoPySVDNzoBd4kCJ2Yzv7spUMM2k2Cj+EZs
X1zuYogmnMNnpT/k0nWihUpYolTTKjJ6bsECj18HHb3KY2AMmcspzFLRmjZIq65cxQM2
ae0/JOIvgWfAMdjH6SPLXdq7fWEvHGnehGHhxhwVMQ7FNk9zxMp8M7H+TFh0MGs+6Asb
SF/3eO55ZSjk6zOBmoBJ6eR3bDIlaSptzkz9u97ywjh5PT7fr1j+CvNhqPjKryoyucVE
mkNda6tuWhghFBDcLsdus9Et7bqDVICgK62kRf5Gjdy9YuYefytZkvivBL/gn9QHI15R
ApDQ==
Received: by 10.52.70.65 with SMTP id k1mr1741040vdu.95.1347287189675; Mon, 10
Sep 2012 07:26:29 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.58.133.37 with HTTP; Mon, 10 Sep 2012 07:26:09 -0700 (PDT)
In-Reply-To: <9F76F49E5A0045629E32809BB4045...@gmail.com>
References: <CAFtNwMo4+W79VPUU5RRk1YFtTkz1=aRXR_AQgWbC-fhs0Hf...@mail.gmail.com>
<9F76F49E5A0045629E32809BB4045...@gmail.com>
From: Yi Tan <yi2...@gmail.com>
Date: Mon, 10 Sep 2012 22:26:09 +0800
Message-ID: <CAFtNwMqPDpOBjtPqCfNb3APJyO8KGg-2DH7ucP_URcusTD-...@mail.gmail.com>
Subject: Re: [nodejs] best practice for real-time socket-based cross
clustered-process data communication and computation
To: nodejs@googlegroups.com
Content-Type: multipart/alternative; boundary=20cf307ac06959de4b04c959be61
--20cf307ac06959de4b04c959be61
Content-Type: text/plain; charset=ISO-8859-1
Thanks rubyonrailsx and Duy Nguyen,
Redis pub/sub had been tried, but it quickly become the performance
bottleneck.
The app we are working on requires high performance capability. It handles
about 10k client socket stream simultaneously on a 8-core linux box.
Averagely, each client stream send/receive 10 messages of 1kb in total
buffer every seconds.
By using node cluster, we have managed to greatly reduce message process
lag and cpu usage. Now la is low.
But the question we are facing is how to let 2 clients in different
processes(workers) talk to each other efficiently.
I think the exact thing I'm looking for is a tcp or udp based cross-process
peer-to-peer node js implementation.
BTW, regarding to Duy's question
> Btw, I'm not sure about "duplicating transfer data" as you mentioned,
initially I think data packet just has been forwarded literally, no
duplication (Am I wrong here:-??).
I haven't read node source of cluster, but if my assumption is correct.
Nodejs has implement a js object serialization in its cluster
implementation. In that way, cluster worker and master can talk to each
other in js object. And data sent by process.send is serialised to buffer
and sent to master. When the other end receive that buff, it parse the
buffer back to js object. That's why when send a buffer instance, it will
be serialised as an array.
Regards,
ty
2012/9/10 rubyonrailsx <rubyonrai...@gmail.com>
> Did you consider remove the need for communication between service A and
> service B . Instead use redis as message pipe?
>
> --
> rubyonrailsx
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On Monday, September 10, 2012 at 4:44 PM, Yi Tan wrote:
>
> Hi node mates:
>
> I'm looking for your advice about how to design and implement an mechanism
> for real-time socket-based cross clustered-process data communication and
> computation.
>
> [The server layout]
>
> clientA <--tcp socket--> node service A <--> DataModelA <--> unique radis
> data store
>
> clientB <--tcp socket--> node service B <--> DataModelB <--> unique radis
> data store
>
> * node service A and B are cluster on the same server
> * 1 data model represent 1 client at the run time
> * both node service A and B talk to the same redis data store
>
> [The function request]
>
> I need to implement a mechanism, in which:
> 1. clients in difference processes can communication with each other
> efficiently.
> 2. there need to be a centralized place for data computation base on data
> models represents difference clients
>
> the logic looks like the following diagram:
>
> clientA <--tcp socket--> node service A
> |
> socket pipe
> |
> v
> clientB <--tcp socket--> node service B <--> DataModelA and B <--> unique
> radis data store
>
> Do you know what is the best way to do this, or is there something already
> been built
>
> Many thanks,
>
> ty
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscribe@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscribe@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
--20cf307ac06959de4b04c959be61
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Thanks=A0rubyonrailsx and=A0Duy Nguyen,<div><br></div><div>Redis pub/sub ha=
d been tried, but it quickly become the performance bottleneck.</div><div><=
br></div><div>The app we are working on requires high performance=A0capabil=
ity. It handles about 10k client socket stream=A0simultaneously on a 8-core=
linux box.</div>
<div>Averagely, each client stream send/receive 10 messages of 1kb in total=
buffer every seconds.</div><div><br></div><div>By using node cluster, we h=
ave managed to greatly reduce message process lag and cpu usage. Now la is =
low.</div>
<div><br></div><div>But the question we are facing is how to let 2 clients =
in different processes(workers) talk to each other efficiently.<br><div><br=
></div><div>I think the exact thing I'm looking for is a tcp or udp bas=
ed cross-process peer-to-peer node js implementation.</div>
<div><br></div><div>BTW, regarding to=A0Duy's question=A0</div><div><br=
></div><div>>=A0<span style=3D"background-color:rgb(255,255,255);color:r=
gb(34,34,34);font-family:arial,sans-serif;font-size:14px">Btw, I'm not =
sure about "duplicating transfer data" as you mentioned, initiall=
y I think data packet just has been forwarded literally, no duplication (Am=
I =A0wrong here:-??).</span></div>
<div><span style=3D"background-color:rgb(255,255,255);color:rgb(34,34,34);f=
ont-family:arial,sans-serif;font-size:14px"><br></span></div><div><span sty=
le=3D"background-color:rgb(255,255,255);color:rgb(34,34,34);font-family:ari=
al,sans-serif;font-size:14px">I haven't read node source of cluster, bu=
t if my assumption is correct. Nodejs has implement a js object=A0</span><f=
ont color=3D"#222222" face=3D"arial, sans-serif"><span style=3D"font-size:1=
4px">serialization in its cluster implementation. In that way, cluster work=
er and master can talk to each other in js object. And data sent by process=
.send is serialised to buffer and sent to master. When the other end receiv=
e that buff, it parse the buffer back to js object. That's why when sen=
d a buffer instance, it will be serialised as an array.=A0</span></font></d=
iv>
<div><br clear=3D"all">Regards,<br><br>ty<br>
<br><br><div class=3D"gmail_quote">2012/9/10 rubyonrailsx <span dir=3D"ltr"=
><<a href=3D"mailto:rubyonrai...@gmail.com" target=3D"_blank">rubyonrail=
s...@gmail.com</a>></span><br><blockquote class=3D"gmail_quote" style=3D"m=
argin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style=3D"font-family:Helvetica;font-size:13px">Did you=
consider remove the need for communication between service A and service B=
. Instead use redis as message pipe?<span class=3D"HOEnZb"><font color=3D"=
#888888"><br>
</font></span></div><span class=3D"HOEnZb"><font color=3D"#888888">
<div><div><br></div><div>--=A0</div><div>rubyonrailsx</div>=
<div>Sent with <a href=3D"http://www.sparrowmailapp.com/?sig" target=3D"_bl=
ank">Sparrow</a></div><div><br></div></div></font></span><div><div class=3D=
"h5">
=20
<p style=3D"color:#a0a0a8">On Monday, September 10, 2012 at=
4:44 PM, Yi Tan wrote:</p>
</div></div><blockquote type=3D"cite" style=3D"border-left-=
style:solid;border-width:1px;margin-left:0px;padding-left:10px">
<span><div><div><div><div class=3D"h5"><div>Hi node mat=
es:</div><div><br></div><div>I'm looking for your advice about how to d=
esign and implement an mechanism for real-time socket-based cross clustered=
-process data communication and computation.=A0</div>
<div>
<br></div><div>[The server layout]</div><div><br></div><div>clientA <--t=
cp socket--> node service A <--> DataModelA <--> unique radi=
s data store</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0=A0=A0 =A0 =A0 =A0=A0</div><div>clientB <--tcp socket--> node serv=
ice B <--> DataModelB <--> unique radis data store</div><div><b=
r></div><div>* node service A and B are cluster on the same server</div>
<div>* 1 data model represent 1 client at the run time</div><div>* both=A0n=
ode service A and B=A0talk to the same redis data store</div><div><br></div=
><div>[The function request]</div><div><br></div><div>I need to implement a=
mechanism, in which:</div>
<div>1. =A0clients in difference processes can communication with each othe=
r efficiently.</div><div>2. there need to be a=A0centralized=A0place for da=
ta=A0computation base on data models represents difference clients=A0</div>=
<div>
<br>
</div><div>the logic looks like the following diagram:</div><div><br></div>=
<div><div>clientA <--tcp socket--> node service A</div><div>=A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
=A0 |</div><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 socket pipe</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 |</div><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0v</div><div>clientB <--tcp s=
ocket--> node service B <--> DataModelA and B <--> unique ra=
dis data store</div>
</div><div><br></div><div>Do you know what is the best way to do this, or i=
s there something already been built</div><div><br></div><div>Many thanks,<=
/div><div><br></div>ty<br><br>
<p></p></div></div><div class=3D"im">
-- <br>
Job Board: <a href=3D"http://jobs.nodejs.org/" target=3D"_blank">http://job=
s.nodejs.org/</a><br>
Posting guidelines: <a href=3D"https://github.com/joyent/node/wiki/Mailing-=
List-Posting-Guidelines" target=3D"_blank">https://github.com/joyent/node/w=
iki/Mailing-List-Posting-Guidelines</a><br>
You received this message because you are subscribed to the Google<br>
Groups "nodejs" group.<br>
To post to this group, send email to <a href=3D"mailto:nodejs@googlegroups.=
com" target=3D"_blank">nodejs@googlegroups.com</a><br>
To unsubscribe from this group, send email to<br>
<a href=3D"mailto:nodejs+unsubscribe@googlegroups.com" target=3D"_blank">no=
dejs+unsubscribe@googlegroups.com</a><br>
For more options, visit this group at<br>
<a href=3D"http://groups.google.com/group/nodejs?hl=3Den?hl=3Den" target=3D=
"_blank">http://groups.google.com/group/nodejs?hl=3Den?hl=3Den</a><br>
</div></div></div></span>
=20
=20
=20
=20
</blockquote><div class=3D"HOEnZb"><div class=3D"h5">
=20
<div>
<br>
</div>
=20
<p></p>
-- <br>
Job Board: <a href=3D"http://jobs.nodejs.org/" target=3D"_blank">http://job=
s.nodejs.org/</a><br>
Posting guidelines: <a href=3D"https://github.com/joyent/node/wiki/Mailing-=
List-Posting-Guidelines" target=3D"_blank">https://github.com/joyent/node/w=
iki/Mailing-List-Posting-Guidelines</a><br>
You received this message because you are subscribed to the Google<br>
Groups "nodejs" group.<br>
To post to this group, send email to <a href=3D"mailto:nodejs@googlegroups.=
com" target=3D"_blank">nodejs@googlegroups.com</a><br>
To unsubscribe from this group, send email to<br>
<a href=3D"mailto:nodejs%2Bunsubscribe@googlegroups.com" target=3D"_blank">=
nodejs+unsubscribe@googlegroups.com</a><br>
For more options, visit this group at<br>
<a href=3D"http://groups.google.com/group/nodejs?hl=3Den?hl=3Den" target=3D=
"_blank">http://groups.google.com/group/nodejs?hl=3Den?hl=3Den</a><br>
</div></div></blockquote></div><br></div></div>
--20cf307ac06959de4b04c959be61--
Message from discussion best practice for real-time socket-based cross clustered-process data communication and computation
| Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy |
| ©2013 Google |