MMO game node balancing multi servers

[hd nguyen]

Hi all,

I'm building a MMO (Massive Multiplayer Online) game with nodejs built in server side, but now I'm getting stuck in how to make load balancing for nodejs between multiple servers.

The game is using web socket to communicate with server. Example we have 2 servers that host GAME1, server1 and server2.

User1 joins GAME1 on server1, User2 joins GAME1 on server2, so each time User1 makes move/action, User2 need to see User1's move/action  and vice versa (each user uses his own browser screen separately).

I know NGNIX as an option, but not sure whether it supports broadcasting data between nodejs servers or not, in our game data for the same game at a server should be broadcast to all other servers that host this game.

Appreciate any suggestion/comment from you.

Thanks in advance.

-- 

Nguyen Hai Duy
Mobile : 0914 72 1900
Yahoo: nguyenhd_lucky

[hd nguyen]

Sorry NGINX :| 

[Arunoda Susiripala]

I don't think nginx supports websockets transport.  
You can use node http proxy.
Not sure about the performance.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

--
Arunoda Susiripala

@arunoda

https://github.com/arunoda

http://www.linkedin.com/in/arunoda

[mscdex]

On Jul 5, 1:14 am, Arunoda Susiripala <arunoda.susirip...@gmail.com>
wrote:

> I don't think nginx supports websockets transport.
> You can use node http proxy.
> Not sure about the performance.

Websockets have been compatible with just about any load balancer for
awhile now. Most clients have something newer than draft-76 these
days....

[hd nguyen]

I looked at node http proxy and see that it lists ' Supports WebSockets ' feature, maybe need some more test on it to make sure it fits my requirement.

@mscdex: you mean most load balancers now support websocket?

Any suggestion more? I'm eager to get more different comments. 

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--

[Oleg Efimov (Sannis)]

Latest (v1.x.y) Nginx supports websockets :)

четверг, 5 июля 2012 г., 9:14:22 UTC+4 пользователь Arunoda Susiripala написал:

> nodejs+unsubscribe@googlegroups.com

[Yi Tan]

for real time mmo games, I'd suggest you to look at other server side solution then node. 

the problem is not in node, but in javascript language. It lacks of native map-based data structure, and will cause you much trouble in code implantation and of cause slows down performance. 

Regards,

ty

2012/7/5 hd nguyen <nguyen...@gmail.com>

[hd nguyen]

OK thanks for your update.

@Yi Tan: have you had a chance to take a look at BrowserQuest(http://browserquest.mozilla.org/)? It's impressive MMO game built with nodejs on server side and work in very responsive manner. 

If I host server code in one server, everything is ok, in the same world, different users can see others action/move perfectly.

And our game is not quite complicated, so I think javascript is not problem here :)

What I concern here if I want to host server code in 2 or more servers(replication code to load balance), and just expose proxy node to client to connect to game, then every processing should be behind the scene(each user should see others move/action as hosted in one server).

I'm still looking for a solution/module/demo for this situation.

Thanks.

[Dominic Tarr]

what sort of messages are you sending?

if your messages have the right properties, and allow eventual consistency,
you should just be able to connect your servers, and it should just work.

what you need is for each node (client, server, etc) to end up with
the same result eventually, even if their messages arrive in different
orders (commutative), or arrive more than once (idempotent).

if you have these properties then you have eventual consistency, and
scaling is easy.

[hd nguyen]

I think it turns out too detail when discussing about code/business processing here.

To imagine easier, please look at below diagram:

As you can see, we have 2 users access to the same game hosted in 2 nodejs servers. Client communicates to server through web socket protocol, example user1 joins game and stick with server1, user2 joins and stick with server2, of course through proxy server as load balancer. 2 users on the same scene of game, assume user1 attacks user2, so user2 must see user1's action and vice versa. 

So nginx or node http proxy can help us on this situation transparently with as less effort as possible? Or any different solution to get over it?

[Charles Care]

I think you're asking about two things: load balancing and replication of state between instance. These are separate problems.

Nginx and http proxy will help you with load balancing and the configuration of sticky sessions etc. This allows you to horizontally scale your front-end servers.

However, replication of state between your application servers, that's more tricky. You need to think about what state needs replicating, how often, and what happens if the system has a temporary network split. As Dominic says above, a lot of these problems become easier if you can architect your system to support eventual consistency (i.e. not every server will always be exactly the same, but that they will converge on the same state). Ideally your nodes should be able to deal with events arriving out of order.

In order to share this state, you'll need to open a communication channel between the two node.js processes (raw TCP might be enough, personally I would use Zeromq, but that's an implementation detail)

USER1 -----> GAME1 <-----> GAME2 <----- USER2

Hope that helps,

Charles

[hd nguyen]

It's right Charles.

And I also understand you and Dominic's idea, I just want to find a tool/module that supports replicating game state between different servers.

If such a tool exists, it's very helpful to us than starting from scratch :)

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Dan Milon]

Your requirement to be able and spin up more servers of the same "game"
requires a lot of sharing between each of these servers, because they
need (eventually) to be in the same state. The only advantage of this
over having a single server for each "game" is that you load balance
incoming traffic and concurrent connections, but introduce
connections/traffic between the servers. Then even more problems arise
when a 3rd server comes in, and every action means notifying also the
other two servers.

Even if you implement this, i dont think it will be sustainable, as you
can imagine.
Maybe we have to redesign how we think about MMO servers. Just a quick
idea: Only one server handles a specific area/city (literally) of the
game. So you limit the shared state, and can still scale to many machines.

You also have to classify your data.
Eg in-game messaging system can be in some db that each server asks when
player visits mailbox. (no need to IPC since it is not a "realtime" message.
But player location has to be the same on all servers every moment, so
movement messages must be sent out straight away (no db).

Good luck with your project ;)

danmilon.

> <mailto:nguyen...@gmail.com>> wrote:
>
> I think it turns out too detail when discussing about
> code/business processing here.
>
>
> To imagine easier, please look at below diagram:
>
> As you can see, we have 2 users access to the same game hosted
> in 2 nodejs servers. Client communicates to server through web
> socket protocol, example user1 joins game and stick with
> server1, user2 joins and stick with server2, of course through
> proxy server as load balancer. 2 users on the same scene of
> game, assume user1 attacks user2, so user2 must see user1's
> action and vice versa.
>
> So nginx or node http proxy can help us on this situation
> transparently with as less effort as possible? Or any
> different solution to get over it?
>
>
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com

> <mailto:nod...@googlegroups.com>

> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com

> <mailto:nodejs%2Bunsu...@googlegroups.com>

[hd nguyen]

Thanks Dan,

Separating each game on a server is good idea, but when the number of users for that game is big enough, that exceeds server processing ability, so we need to scale out (cannot scale up now, it reaches the limitation of physical machine), and we turn back to initial problem, how to maintain the same game state between different users, right?

And as you said, broadcasting between many servers is a costly task indeed.

Initially the No. of users of a game is not too big, but we want to think more far away for future, maybe we cannot figure out any solution properly(no such of tool to make it easier) :(,  but personally, I appreciate your solution a lot, Dan :)

    <mailto:nodejs@googlegroups.com>

    To unsubscribe from this group, send email to

    nodejs+unsubscribe@googlegroups.com

    <mailto:nodejs%2Bunsubscribe@googlegroups.com>

    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en




--
Nguyen Hai Duy
Mobile : 0914 72 1900
Yahoo: nguyenhd_lucky
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

nodejs+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--

Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

nodejs+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Dominic Tarr]

hi, I've been doing quite a bit of research into this area recently.
there is a lot of interesting stuff, I have a blog post that
summarizes some approaches
and links to some papers
http://dominictarr.com/post/25083602144/distributed-programming-is-easy

eventual consistency is not actually that difficult to achieve,
requires rethinking everything.

I've got a project for replicating realtime data between servers,
https://github.com/dominictarr/crdt

it has a trello like example, that replicates some lists between
multiple clients.

I'm not sure if it's assumptions meet your use-case,
and I agree with dan that you will probably need to partition your game world.
but it may be helpful, or be a useful in your research, because you
are really getting
into cutting edge territory!

cheers, Dominic

>>> <mailto:nod...@googlegroups.com>

>>>
>>> To unsubscribe from this group, send email to

>>> nodejs+un...@googlegroups.com
>>> <mailto:nodejs%2Bunsu...@googlegroups.com>

>>>
>>> For more options, visit this group at
>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>
>>>
>>>
>>>
>>> --
>>> Nguyen Hai Duy
>>> Mobile : 0914 72 1900
>>> Yahoo: nguyenhd_lucky
>>> --
>>> Job Board: http://jobs.nodejs.org/
>>> Posting guidelines:
>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>> You received this message because you are subscribed to the Google
>>> Groups "nodejs" group.
>>> To post to this group, send email to nod...@googlegroups.com
>>> To unsubscribe from this group, send email to

>>> nodejs+un...@googlegroups.com

>>> For more options, visit this group at
>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>>
>>
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to nod...@googlegroups.com
>> To unsubscribe from this group, send email to

>> nodejs+un...@googlegroups.com

>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
>
>
> --
> Nguyen Hai Duy
> Mobile : 0914 72 1900
> Yahoo: nguyenhd_lucky
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to

> nodejs+un...@googlegroups.com

[Adam Reynolds]

Hi,

Are you looking at this incorrectly? Should you be looking into 'handing off' players between game servers not locking a user to a specific server and expecting game servers to maintain state. Your solution will create an immense amount of traffic between servers and eventually you will flood your environment with unnecessary network traffic.

If you have different servers for different zones, should a zone become too popular then you can add another zone server and use some intelligence when choosing which server the user should go to.

I would split the social aspects (messages/mail/trade) of the game onto another separate server. This allows a global community to interact. Having a message arrive within 20 seconds is not an issue.

Adam

[Peter Holdsworth]

I am developing a poker server to support circa 100,000 concurrent players using node.js socket.io and redis.

My node servers are split between multiple front-end socket servers and multiple back-end game servers.

Game servers support pools of poker tables.

When a player is playing at a table, messages are passed from client to load balancer to socket server to game server.

The socket server knows on which pool of tables the player is seated (and this is persisted to support re-start) 

Responses are sent to all players seated at that table (through different socket.io servers). 

When a player leaves a table, chip count is persisted in redis. 

If a player transfers to another game server the the new routing is persisted, socket server is informed and the switch is made.

Game server picks up the chip count from redis and the player is seated in the new pool of tables.

Of course poker is a very simple game in that it naturally splits into pools of tables and a player is only in one pool at a time. Responses to user actions are limited to a small number of players. Switching between pools is not frequent and only the chip count and some simple routing information needs to be passed through redis.  Nevertheless, you should be able to use some elaboration of these techniques to build your game. 

[박성우]

I have a similar problem in that issue.

I was developped PvP msg relay server using nodeJS & socket.IO and redis. 

For load balancing, Try to develop another server (front-end).

I am considering the technology are as follows:

  1. websocket prxoy using node http proxy module.

  2. another websocket server.

  3. read transaction guaranteed rest API server.

In the case of 1 and 2, Front and rear of the server the server load is almost the same Backed'm wondered

2012년 7월 7일 토요일 오전 2시 18분 13초 UTC+9, Peter Holdsworth 님의 말:

[Charlie Crane]

You can try pomelo framework.   All these difficult things are done by the framework.

https://github.com/NetEase/pomelo

--

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

---

You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.