Server-to-Server Communication

[Jacob Groundwater]

I was wondering what others are using for communication between back-end servers. Since I control all involved servers, I would rather not include authentication at the application layer. Forwarding SSH ports is perfectly acceptable.

On that note, my own research has lead me to three options, in order of personal preference:

1. a message queue (ZeroMQ)
2. websockets (socket.io)
3. synchronize against the database (MySQL or Redis)

ZeroMQ seems pretty awesome, but I am curious if anyone has tried it, and what there experience was. For example, I stumbled across an article discussing how their REQ/REP model can lock up easily. The workaround is fairly simple, but I am interested in soliciting more experience in the area.

Websockets seem like a "native" way, but I see them as living in the client-server domain. For example, I would have to setup express and a basic restful service on each back-end server. Websockets are also 1-to-1, where as ZeroMQ supports N-to-N connections.

Synchronizing against the database would involve polling to achieve real-time like events. I know Redis supports a pub/sub system, but does not seem to have any RPC-like mechanisms.

In the end, I will make the decision best suited to our needs, but I am sure I can gain from some discussion on the matter.

Thanks everyone,

- Jacob Groundwater

[Alexey Kupershtokh]

I'm also interested in this theme.

AFAIR, dnode 6 months ago was good in features but terribly slow comparing to socket.io/axon.

https://github.com/visionmedia/axon looks interesting, but unstable yet - I have already found an issue ( https://github.com/visionmedia/axon/pull/62 ). Also I don't like it's feature to open a new socket/port for each communication type.

Also I don't like Socket.io's feature that it doesn't free ACK callbacks (probably already does) which is a leak for long & intensive s2s sockets.

воскресенье, 28 октября 2012 г., 1:32:19 UTC+7 пользователь Jacob написал:

[Jacob Groundwater]

It looks like ZeroMQ does not play well with the asynchronous nature of Node. Take a look at the following gist:

https://gist.github.com/3966362

I would expect to see three replies from the server, except there is only one. This occurs when the server tries to reply out of order. Changing the send order to the following works:

sockit('thr').send( JSON.stringify({timeout:0,name:'thr'}) )

sockit('two').send( JSON.stringify({timeout:1000,name:'two'}) )

sockit('one').send( JSON.stringify({timeout:5000,name:'one'}) )

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Marak Squires]

Use https://github.com/learnboost/engine.io

[Dan Milon]

IMO socket.io/engine.io aim for server-client communication.
But it would be interesting to see some numbers.

danmilon.

On 10/28/2012 12:49 AM, Marak Squires wrote:
> Use https://github.com/learnboost/engine.io
>
> On Sat, Oct 27, 2012 at 2:31 PM, Jacob Groundwater <ja...@nodefly.com
> <mailto:ja...@nodefly.com>> wrote:
>
> It looks like ZeroMQ does not play well with the asynchronous nature
> of Node. Take a look at the following gist:
>
> https://gist.github.com/3966362
>
> I would expect to see three replies from the server, except there is
> only one. This occurs when the server tries to reply out of order.
> Changing the send order to the following works:
>
> sockit('thr').send( JSON.stringify({timeout:0,name:'thr'}) )
> sockit('two').send( JSON.stringify({timeout:1000,name:'two'}) )
> sockit('one').send( JSON.stringify({timeout:5000,name:'one'}) )
>
>
>
> On Sat, Oct 27, 2012 at 11:46 AM, Alexey Kupershtokh

> <alexey.ku...@gmail.com <mailto:alexey.ku...@gmail.com>>

> wrote:
>
> I'm also interested in this theme.
>
> AFAIR, dnode 6 months ago was good in features but terribly slow

> comparing to socket.io/axon <http://socket.io/axon>.

>
> https://github.com/visionmedia/axon looks interesting, but
> unstable yet - I have already found an issue
> ( https://github.com/visionmedia/axon/pull/62 ). Also I don't
> like it's feature to open a new socket/port for each
> communication type.
>
> Also I don't like Socket.io's feature that it doesn't free ACK
> callbacks (probably already does) which is a leak for long &
> intensive s2s sockets.
>
> воскресенье, 28 октября 2012 г., 1:32:19 UTC+7 пользователь
> Jacob написал:
>
> I was wondering what others are using for communication
> between back-end servers. Since I control all involved
> servers, I would rather not include authentication at the
> application layer. Forwarding SSH ports is perfectly acceptable.
>
> On that note, my own research has lead me to three options,
> in order of personal preference:
>

> 1. a message queue (ZeroMQ)
> 2. websockets (socket.io <http://socket.io>)
> 3. synchronize against the database (MySQL or Redis)

>
> ZeroMQ seems pretty awesome, but I am curious if anyone has
> tried it, and what there experience was. For example, I
> stumbled across an article discussing how their REQ/REP
> model can lock up easily.

> <http://lucumr.pocoo.org/2012/6/26/disconnects-are-good-for-you/> The

> workaround is fairly simple, but I am interested in
> soliciting more experience in the area.
>
> Websockets seem like a "native" way, but I see them as
> living in the client-server domain. For example, I would
> have to setup express and a basic restful service on each
> back-end server. Websockets are also 1-to-1, where as ZeroMQ
> supports N-to-N connections.
>
> Synchronizing against the database would involve polling to
> achieve real-time like events. I know Redis supports a
> pub/sub system, but does not seem to have any RPC-like
> mechanisms.
>
> In the end, I will make the decision best suited to our
> needs, but I am sure I can gain from some discussion on the
> matter.
>
> Thanks everyone,
>
> - Jacob Groundwater
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com

> <mailto:nod...@googlegroups.com>

> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com

> <mailto:nodejs%2Bunsu...@googlegroups.com>

> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com

> <mailto:nod...@googlegroups.com>

> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com

> <mailto:nodejs%2Bunsu...@googlegroups.com>

[Jacob Groundwater]

Axon looks promising. I will be evaluating that next.

I'm am unclear as to what engine.io is trying to accomplish.The page says:

Engine is the implementation of transport-based cross-browser/cross-device bi-directional communication layer for Socket.IO

This to me implies socket.io is built on top of engine.io, however the socket.io sources does not mention engine.io in the package.json file as a dependency.

This Stackoverflow page  says:

Engine is to Socket.IO what Connect is to Express

 

engine.io is of more interest to you if your building a library/framework on top of socket.io

 

I am definitely not aiming to build a library / framework. I would like a simple, reliable, means of server-to-server communication.

[Michael Schoonmaker]

ZeroMQ is designed for server-to-server communication, and works really well. For my current project I wanted a call-and-callback style structure, so https://github.com/Schoonology/shuttle was born.

The Shuttle framework may be exactly what you're looking for, but if not I hope it's a useful example of what can be easily accomplished with Zero. (Shuttle was built in a weekend.)

-Schoon

[Arnout Kazemier]

Durrrrrrrrr.... engine.io is for client side communication, no serverside.

[Marak Squires]

No, that is not correct.

https://github.com/LearnBoost/engine.io-client

[Arunoda Susiripala]

Marak,

How about hook.io, what happen to it?

Arunoda Susiripala

@arunoda

https://github.com/arunoda

http://www.linkedin.com/in/arunoda

[Arnout Kazemier]

Sure you can communicate with engine.io using a server component.. but then you might as well create plain HTTP server and use the HTTP client to communicate with it…

But what ever..

Arnout Kazemier, Founder observe.it

http://observe.it - observing & learning from your visitors.

Skype: arnoutkazemier

Mobile: +31623254031

[Ryan Schmidt]

On Oct 28, 2012, at 03:07, Arunoda Susiripala wrote:

> How about hook.io, what happen to it?

hook.io is dead:

https://groups.google.com/d/msg/nodejs/FCsTCopm_dE/kfCS-1nRfrUJ

[Marak Squires]

For server-to-server communication, engine.io will use the 'ws' module ( https://github.com/einaros/ws )

See: https://github.com/LearnBoost/engine.io-client/blob/master/lib/transports/websocket.js#L147

[rektide]

There's node-webworker at https://github.com/pgriess/node-webworker . It's programmed via an
API alike W3C's WebWorkers. Internally it's a very simple message passing protocol, hosted
on top of WebSockets. Between the too low level (ZeroMQ, dealing intimiately with sockets)
and too high level (plenty of targets, let's call out hook.io randomly), creating
MessageChannels and MessagePorts is to me a great middle-level of messaging abstraction, and
it's one we'll continue to see more of in places outside of Node-land.

-rektide

[Tim Caswell]

Internally at cloud9, we use the smith protocol. It's based on the
same idea of dnode, but has restricted abilities so that it can be
leak-proof using normal javascript (dnode requires --harmony to
prevent leaks) and very fast. We usually use the protocol over
msgpack + tcp between servers (or over the stdio channel of an ssh
connection to a remote server).

https://github.com/c9/smith

[Adam Crabtree]

We use Schoon's Shuttle at RedRobot, and it's been working out perfectly as a drop-in replacement for hook.io with dramatic performance increases. It uses msgpack and 0mq. Definitely check it out if you're wanting to go the 0mq route. Otherwise, Engine.io looks promising, though I've never used it. 

https://github.com/Schoonology/shuttle

Cheers,

Adam Crabtree

--
Better a little with righteousness
       than much gain with injustice.
Proverbs 16:8