A very small game server architecture

[Zippoxer]

Hi,

I'm building a server for a very tiny MMORPG.

My main goal with this project is learning how to break down my project to modules. Independent modules that aren't tightly, can do their thing by themselves and be tested by themselves.

So I'm trying to achieve such modularity, and I'm not sure I'm doing this right.

Here's what I have now:

- The world package is responsible for initializing and managing a world instance which holds the map, items, monsters and players in it.

  The World struct holds the world's state and disallows concurrent read-write by restricting you to only view and update the state via transactions.

  Similarly to boltdb, you must utilize a transaction with World.View or World.Update in order to view/update the world's map, players or anything within the world's state.

  Updates to anything should be sent to players within the world. If you change the map, that change should be sent to nearby players.

- The server package, holding a world.World instance, is responsible for taking actions from the player's client and mutate the world instance accordingly. The server package also receives updates from it's world instance and sends them to relevant players.

The world and the server package have a two way connection. The Server mutates the world according to players' commands, and the World passes messages to the server to send to relevant players on any change in the map, or anything within it's state.

My problem is, how do I bridge between the world and the server? The world has no knowledge of how to do the networking and update players of changes itself, so how should it pass updates to the server to be sent for players?

In Go, we can't have circular dependency, so I thought the server could pass a channel or an interface to the world as it registers a player and the world can use it to update the player of any changes down the road.

However, such thing would require redefining the commands the server can send (like MapChange, etc.) in the world package and thus making me maintain basically the same thing in two places.

The only way out that I can see right now would be to have the server and world packages merged, then the world could do networking itself.

If any of you have basically anything to say, I'd really like to hear it :)

--

Cheers,

Moshe

[C Banning]

• Server has SSE/websocket/etc. subscribers on a message bus, if clients subscribe to a multicast addr then channel will suffice.
• World exposes a PublishState() function that puts messages on bus.
• Server passes transactions to World.
• If transactions have been processed within a concurrency interval, say, "1s", Server calls World.PublishState().
• World writes messages on bus/channel.
• Server subscribers (or multicast writer) grab messages of interest and push them to clients.

Let Server set up the bus/channel and register it with World. World just exposes ProcessTransaction() and PublishState().

[FIX Rob Lapensee]

Moshe,

I was good up to this point:

>>However, such thing would require redefining the commands the server can send (like MapChange, etc.) in the world package and thus making me maintain basically the same thing in two places.

which I did not understand.

Perhaps without fully understanding the problem, this is what I would do:

The Server package does method calls to the World package.

Those calls would include information such as "I am interested in this map area because a player is there"

and "I'm no longer interested in that map area because a player is no longer there".

The Server package sends a single channel to the World package (only once),

the data on the channel is "World package to Server package".

Now the World package knows what information the Server package needs,

and sends that on the single channel whenever anything changes.

The Server package has a go routine handling that channel.

The data on the channel is for all changes and all players.

the data definition for the strucs on that channel are defined in the World package.

I created the foundation of an MMORPT here: https://github.com/gk-turnip

Server is written in Go, the client is a browser (lots of javascript).

but it needed so much refactoring that I decided to start over.

The "start over" project has only just started.

hope this helps,

Rob

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christian Mauduit]

Hi,

On 03/25/2016 12:40 AM, Zippoxer wrote:
> Hi,
>
> I'm building a server for a very tiny MMORPG.

Nice to hear about another game builder using Golang ;)

> So I'm trying to achieve such modularity, and I'm not sure I'm doing
> this right.

Basic fact: you're probably going to get it wrong the first time, but
this is not a problem, just try & repeat.

> My problem is, how do I bridge between the world and the server? The
> world has no knowledge of how to do the networking and update players of
> changes itself, so how should it pass updates to the server to be sent
> for players?

IMHO your server should be "game agnostic", possibly not knowing what
kind of game it is running. Indeed, what you describe is quite generic,
as you infer, it looks pretty much like a standard DB (you mention
boltdb), so what tie it to a peculiar game.

AFAIK when you encounter a problem like this, a common solution is to
provide hooks. In C you'd send pointers on funcs and opacify everything,
hopefully, Golang interface are pretty convenient for this, at least I
do find them very helpful. Typically you'd design a "game world
interface" which would define the basic methods appliable to a game
world. Then you could have your server to be able of those features, but
only of those, not he implementation. I've no precise idea what your
game is about, but you could end up with something quite generic such as :

------8<---------------------------------------------------
type GameWorld interface {
CreateMonster(serializedJSON string) (int, error)
UpdateMonster(RefID int, serializedJSON string) error;
GetMonster(RefID int, serializedJSON string) (string, error);
// ...
}
------8<---------------------------------------------------

Now you get a "design" choice wether you want GameWorld to explicitely
state it's about Monsters & Projectiles & <other-game-stuff> or wether
you are just handling abstract "objects". There's no perfect decision
here, the more generic, the more powerful, but not always the easiest to
grasp when reading the code. Try and judge yourself.

And not the above code is just an arbitrary choice, you could do it the
other way and expose a network server interface, and have the game world
be aware of that interface only ;)

But typically, your network code could be aware of the GameWorld
interface (which you could externalize in a third package and include in
both server code and game world implementation) but know nothing about
the game details.

Another way to think about it is: should you make two different games
with the same network code, where would you draw the line? An option is
to have a "toy, demo game" along with your official one, typically a
test dummy game you'd run for regression tests on your server.

Have a nice day,

Christian.

--
Christian Mauduit
uf...@ufoot.org
http://www.ufoot.org
int q = (2 * b) || !(2 * b);

[oju...@gmail.com]

Hi, Moshe.

 

I think all you need is to link the server and the world by channels passing a known type (as string), but just in case you really need to pass new types from one side to another, I made a little toy to show how I would approach the problem. In this code the server sends specific commands to the world and the world sends status updates to the server.

 

I hope you find it useful.

 

http://www.jucie.com.br/moshe.zip

 

[]s

[Zippoxer]

Rob,

Thanks for the input. I find your project very interesting ;)

I didn't mention I also have a net package that is responsible for encoding and decoding network messages and defines messages like OnCreatureAppear.

If I'm to remove knowledge of networking from the world package and instead have it define and publish it's own messages, it would have to redefine messages like OnCreatureAppear that would then be translated by the recipient side (server) to net.OnCreatureAppearPacket and sent to the player.

I can see the upside of redefining tho, this makes adding another network protocol much easier if messages are defined by the world package and not the net package which currently only encodes messages to binary.

There's another downside to this I'm not sure how to counter - identical messages would be encoded multiple times. If the world broadcasts OnCreatureMove to 10 players, then 10 gorountines that handle the connections of those players would receive a world.OnCreatureMove, fill a net.OnCreatureMovePacket with it and encode to binary. Encoding would happen 10 times for an identical message. I think a good solution is caching recently encoded messages and reusing them in the server package.

I know it may sound unreasonable, but I'm trying to not utilize channels in the very core of the system. To detail, there may be thousands of listeners for a certain location since monsters (not only players) would also listen for world changes and behave accordingly (basically a dummy chase & attack on any player who appears). I'm not sure a channel for every monster would be performant enough, as the world changes *very* frequently and there could easily be thousands of monsters on the map listening for changes.

I thought about calling on an interface to send world messages (if any) that resulted from a transaction immediately after it ends and the world mutex has been unlocked. This would result in these calls happening on the caller's thread and not locking the world from any reads or writes. Here's a dummy example similar to what I have now:

package world

import "world/msg"

type World struct {

// map, players and monsters...

}

type (w *World) Begin() *Tx {

w.mu.Lock()

}

type Tx struct {

w *World

// msg contains messages collected while running the transaction.  

msg map[[]int][]msg.Msg // map[creature ids]messages

}

func (tx *Tx) Send(to []int, m ...msg.Msg) {

tx.msg[to] = append(tx.msg[to], m...)

}

// AddCreature is an example of how a transaction can mutate the world state,

// and it may be called by the server package or any other client of World.  

func (tx *Tx) AddCreature(name string, l Loc) int {

// add creature to tx.w.map ...

// Broadcast CreatureAppear to spectators.

id := tx.nextCreatureID()

tx.w.Send(tx.Spectators(l), msg.CreatureAppear{

ID: id,

Name: name,

Loc: l,

})

return id

}

// Spectators returns ID of any creature within range to see Loc.

func (tx *Tx) Spectators(Loc) []int

func (tx *Tx) Close() {

tx.w.mu.Unlock()

// The world lock has been freed and the following would be

// called from the goroutine that initiated this transaction.

// This is the time to send messages collected from the transaction.

for recipients, messages := range tx.msg {

for _, id := range recipients {

// tx.Creature returns a Creature interface assigned by the server package

// or any other client of this World. If it is a player, it should

// implement Send to transfer this message to it's client. If it's

// a monster, it may implement logic to read those changes

// and respond accordingly by initiating another transaction.

tx.Creature(id).Send(messages...)

} 

}

}

What do you think of this model as opposed to channels?

Cheers,

Moshe

[Zippoxer]

There are a few logic mistakes in the dummy example above. Here's the fixed code:

package world

import "world/msg"

type World struct {

// map, players and monsters...

}

type (w *World) Begin() *Tx {

w.mu.Lock()

return &Tx{w: w}

}

type Tx struct {

w *World

// msg contains messages collected while running the transaction.  

msg map[[]int][]msg.Msg // map[creature ids]messages

}

// Send adds a message to be sent immediately after the transaction ends.

func (tx *Tx) Send(to []int, m ...msg.Msg) {

tx.msg[to] = append(tx.msg[to], m...)

}

// AddCreature is an example of how a transaction can mutate the world state,

// and it may be called by the server package or any other client of World.  

func (tx *Tx) AddCreature(name string, l Loc) int {

id := tx.nextCreatureID()

// add creature to tx.w.map ...

// Broadcast CreatureAppear to spectators.

tx.Send(tx.Spectators(l), msg.CreatureAppear{

[Rob Lapensee]

Moshe,

It looks like the World struct holds world context (world map, creatures, players)

but does not directly handle creatures (just an observation).

You lock the entire World context, make the changes for the transaction,

then unlock the entire World and send all the updates to anything

that needs to know of the World context change.

At the end you unlock the world, then traverse your tx.msg map, which might be

in the middle of changes from the next transaction that will be free to start.

You may need to unlock after you send out the "world change" messages

to avoid a "race" condition on tx.msg.

Mind you if your tx.Creature(id).Send can result in another transaction

it would cause a dead lock.

A buffered channel helps in this situation,

since you can send the message out the buffer and continue,

the receiving program will pick up the message when it is not busy.

If the routine sending the message to the channel sees the channel get too full

it can log an error and discontinue sending on that channel.

In this way the world context never gets hung up.

If the Send is going to a player update, via the network, then you would

have to make sure it won't block (slow network), because that would block

your world transactions.

Doing transactions this way opens up your world context to any

problems in the code that is doing a Begin/Close,

if that Close goes missing after a Begin (error condition or whatever reason),

the all transactions stop.

Also, if the calling program blocks or is slow doing calls within a transaction,

your world locks until the transaction is closed.

Consider encapsulating any manipulation / query of the world into a single

call, that way no matter what the calling programs do,

the world won't get locked up.

What does the calling program(s) look like?

Is there a  go routine per creature for example?

At some point in time we may be deviating from Go issues,

and getting into MMORPG game design issues,

at which point we should gather a list of anyone who is interested

and switch to email.

Anyone can chime in if they feel we are getting too far away from Go issues.

Regards,

Rob

[oju...@gmail.com]

Rob, I think you hit the nail on the head.

 

IMHO concurrent design is an underestimated topic.

 

Synchronization problems are not unique to MMORPG, much to the contrary. It is a very serious issue and Go offers new constructions to help design programs, avoiding common synchronization pitfalls, routinely found when using other technologies.

 

There is a great number of Go programmers that could benefit by learning how to leverage goroutines and channels to build robust and efficient software. Concurrent design, that is.

 

One must learn to think different.

[Zippoxer]

Hi Rob,

You're right, this is topic covers issues beyond Go. However, the fundamental issue we're

discussing now is what fits this concurrent system best - mutex or channels. So opinions and

conclusions from this discussion may be relevant to many concurrent systems written in Go as well.

Callers who execute transactions in world.World may dedicate a goroutine per creature or one per many. In the case of players, there's already a goroutine dedicated for every connection with him. Monsters may have one goroutine to manipulate them all.

Here's an example of player handling:

import "world/msg"

type Server struct{}

func (s *Server) Start() error {

// accept connections and call `go s.handleConnection(conn)` on each.

}

func (s *Server) handleConnection(conn net.Conn) {

// login ...

// Register player to the world.

var creatureID int

rcv := worldMsgReceiver{conn: conn} // defined below

s.world.Update(func(tx *world.Tx) {

creatureID = tx.AddPlayer(name, health, ..., rcv)

})

for {

// znet is the networking package that speaks the game's binary protocol.

i, err := znet.Receive(znet.Client, conn)

if err != nil {

break

}

switch pkt := i.(type) {

// znet.NorthPacket is sent when the player wants to walk north.

// It does not carry any data except it's own identity.

case znet.NorthPacket:

// world.World.Update locks the world for readwrite and executes

// a transaction.

// No other transactions can run before this one finished.

s.world.Update(func(tx *world.Tx) {

// Walk sends a msg.UpdateMap message if the walk

// went well. Otherwise, it sends msg.CancelWalk.

// Immediately after this transaction is executed,

// and the world lock has been freed, either message 

// would be sent to our rcv instance and other receivers

// of nearby creatures.  

tx.Player(creatureID).Walk(world.North)

})

case znet.InventoryOpenPacket:

// world.World.View locks the world for read and executes a

// transaction.

// Only other read transactions can run while this one is running. 

s.world.View(func(tx *world.Tx) {

inv := tx.Player(creatureID).Inventory()

err := znet.Send(conn, znet.InventoryPacket{

Bags: inv.Bags,

Items: inv.Items,

})

checkErr(err)

})

}

}

conn.Close()

}

type worldMsgReceiver struct{

conn net.Conn

}

func (r *worldMsgReceiver) Receive(creatureID int, m msg.Msg) {

switch v := m.(type) {

case msg.CreatureAppear:

err := znet.Send(r.conn, znet.CreatureAppearPacket{

ID: v.ID,

Name: v.Name,

Look: v.Look,

// ...

})

checkErr(err)

}

}

world.Update and world.View are there to safely execute transactions. Under the hood, they call World.Begin, execute your func and Tx.Close (which sends any collected message after unlocking the world).

With this model, there can be one goroutine handling all thousands of monsters. It can register all monsters with the same receiver, as the message receiver is passed the subject creature ID as well.

[Zippoxer]

I forgot to address your other comments.

Message objects received by world.MsgReceiver interfaces should not be mutated. I could send a copy of the message to each receiver and be done with this problem.

Since, I'm freeing the world lock before sending messages, I should not worry about those messages' receivers executing other transactions.

I could just as well free the world lock after sending messages, and pass world.Tx to message receivers and thus allow receivers to continue operate in the same transaction that sent them a message. It would be an interesting idea to explore.

There would be no unlocking then locking again for receivers who want to execute transactions as a response to a message. They would instead just keep working with the transaction that called them.

As you can see in my latest dummy snippet, callers should call the wrappers World.View and World.Update instead of World.Begin to avoid such problems of late or never closing of transactions. This is exactly what boltdb does, and where I adopted this idea from.

I find this model very interesting :)

Of course, channels can do the same job I'm using mutexes to do.

When I started with Go, I kept looking for every place I can shove a channel into. Now I'm trying to do the opposite ;)

Cheers,

Moshe

On Friday, March 25, 2016 at 9:26:35 PM UTC+3, Rob Lapensee wrote:

[oju...@gmail.com]

Why are you trying to avoid using channels? Have you some specific motivation or is that by mere scientific curiosity?

[Zippoxer]

I'm trying to avoid using channels where they don't fit best.

Channels can make things simpler in many cases. However, sometimes, locking can do just as good.

I'm still not sure when channels fit better than mutexes, but I use them myself when they seem more intuitive.

See https://github.com/golang/go/wiki/MutexOrChannel

In the case of world.World, I don't see how channels could do any better than a mutex. If I were

to use channels for running transactions, suddenly world.World has to run it's own goroutine to listen for

transactions. I really don't care for another goroutine and channel, but why? What would be the benefit in this case?

Cheers,

Moshe

[Zippoxer]

Your program very simple and clear :)

I didn't mention in my first post, I also have a net package for speaking the game's binary protocol. It has every packet

as an individual struct, and what I wanted to avoid was to not have the world package import the net package.

I wanted to separate the world from anything related to networking, so I ended up defining a package *very* similar to your cmd package that only defines

commands the world package speaks. The server package gets those and is then free to translate them to network messages however it likes :)

Cheers,

Moshe

[oju...@gmail.com]

On Saturday, March 26, 2016 at 11:42:46 AM UTC-3, Zippoxer wrote:

Your program very simple and clear :)

Thank you, Moshe.

 

In my experience, explicit locking, using mutex and such, doesn't scale well.

 

Using explicit locking, the programmer tries to restrain simultaneous access by allocating synchronization devices to protect a variable or groups of variables. The general thinking goes something like "oh man, we may have simultaneous threads trying to access this thing. Easy enough, let's create a mutex."

 

First and foremost, the programmer assumes he knows all places where locking is needed. Does he? Maybe a coworker is working in a patch at that same exact moment, that access the variable, without locking, because there was no mutex before. In other words, because that is manual synchronization, it is difficult to be sure all access are correct. That is the first trap.

 

The second trap is, as the software matures, whenever there is a new situation that implies simultaneous access, the programmer will create yet another mutex. Soon we have dozens of mutex. A headache farm, you can bet, because as soon as the complexity grows a little, it is not possible to predict how that mutex locking will interact together.

 

Imagine a function calling another function. The first one locks a mutex, to be unlocked an soon as it exits. The second one locks yet another mutex. In the meantime, and depending on several hard to reproduce circumstances, there is another thread trying to access the same resources, so as expected, it locks the same set of mutex, but in a different order. Now we can't proceed anymore, because there is a reciprocal dynamic locking dependency of one another. The software freezes and nobody has the slightest idea of why.

 

Now I am explaining the situation under a very light and theoretical view. That is easy. Imagine the bad luck of the guy that is presented with such a freeze in a real situation, in a software counting hundreds of thousants code lines. Remember: his boss wants the problem solved before lunch time, of course.

 

First our hero will have a hard time trying to figure what really happened. Probably he has no clue and may loose a spectacular amount of time and energy reasoning about what is wrong in business rules. Probably nothing.

 

Let's suppose that by divine mercy he discovers during a dream the root reason is manual synchronization of hundreds of mutex. How to fix that? What must be done? A non trivial examination is needed. The bad news is: there is no easy way out, because the entire architecture is compromised. With some luck, he will fix that situation, only to be presented with yet another one week later, that must be examined and fixed as well, and so on. Great! Job security!

 

I have seen scenarios like these more than once in serious, big production software. That's why I run from these mutex festivals as to save my life.

 

Albeit being a small language, Go offers the elements to avoid an entire class of such problems.

 

I like command queues. I am a firm believer in using Go channels and goroutines to implement that. That way the programmer can build software that doesn't depend on extreme caution from anyone. If some part needs another part to do something, it sends a command and the other part will attend it in due time, in all safety. No synchronization needed, because there is no simultaneous access to the same resource. Each part being responsible for administering its resources.

 

I have code using this strategy for some years now, with no problem at all. It works like a charm.

 

[]s

O JuciÊ

　

[FIX Rob Lapensee]

I was going to create a channel example, but I like O Juice's example better :)

--

[terrel....@gmail.com]

On Thursday, March 24, 2016 at 5:40:33 PM UTC-6, Zippoxer wrote:

Hi,

I'm building a server for a very tiny MMORPG.

 

- The world package is responsible for initializing and managing a world instance which holds the map, items, monsters and players in it.

  The World struct holds the world's state and disallows concurrent read-write by restricting you to only view and update the state via transactions.

  Similarly to boltdb, you must utilize a transaction with World.View or World.Update in order to view/update the world's map, players or anything within the world's state.

 

  Updates to anything should be sent to players within the world. If you change the map, that change should be sent to nearby players.

This is an interesting problem. etcd or serf could provide a part of the solution -- without a server -- a true P2P RPG.

Every node holds its own map of the world, which it shares with its peers. Agreement on the outcome of a battle, for example, could be messy, just like in real life.  If every node is running the same software, the agreement would come, eventually. (Hostile nodes running custom software would add an extra dimension to the game.)

- The server package, holding a world.World instance, is responsible for taking actions from the player's client and mutate the world instance accordingly. The server package also receives updates from it's world instance and sends them to relevant players.

IOW: The server Must Be Obeyed (tm). The players (nodes) are second-class citizens. ;-)

The world and the server package have a two way connection. The Server mutates the world according to players' commands, and the World passes messages to the server to send to relevant players on any change in the map, or anything within it's state.

My problem is, how do I bridge between the world and the server? The world has no knowledge of how to do the networking and update players of changes itself, so how should it pass updates to the server to be sent for players?

It's just a database, that happens to be distributed with very low latency. This is what serf was built for.
 
https://www.serfdom.io/