Using Protocol Buffers for client server Game API determining what type of message to decode

1,689 views
Skip to first unread message

Doug Daniels

unread,
Aug 1, 2008, 1:20:39 AM8/1/08
to Protocol Buffers
I'm not sure if Protocol Buffer is really intended for this use (but
it'd be nice if it could be used for this).

I want to make a game network protocol for client/server communication
where the client could receive a variety of protocol buffer defined
messages, after negotiating the login protocol e.g.:

1. client: send LoginInfo
2. server: sends Entity object in response

(This could be defined as an RPC service that could be implemented
using the RPC Channel implementation on the client side).
service GameService {
rpc login(LoginInfo) returns (Entity);
}

After the login is completed the client will poll the socket
InputStream and parse out generic messages e.g.
- MoveEntity
- DamageEntity
- EntityList

From reading the documentation of protocol buffer it seems like the
API is built to encode and decode expected messages, as in I'd have to
know that the server was sending a MoveEntity or DamageEntity message
before I try to decode it off the socket bytestream. Is there any way
to determine what type of Message it is (kind of like using Java
reflection or checking the instanceof for the Message being decoded?)

Some work arounds I've come up for this is to define a message that
contains as optional members EVERY message defined in the protocol,
like this:

message GameProtocolMessage {
optional LoginInfo loginInfo = 1;
optional Entity entity = 2;
optional Attack attack = 3;
optional DamageEntity damageEntity = 4;
optional EntityList entityList = 5;
optional MoveEntity moveEntity = 6;
}

and additonally we could include a list of messages so the server can
send a batch of messages to the client:

message GameProtocolMessages {
repeated GameProtocolMessage messages = 1;
}

Then the client could figure out which message was sent and handle
that correctly. The question I have is what's the overhead of having
all those optional fields (especially with a large network protocol
400+ messages) if you plan on having only one filled in each time.

The other solution I thought of (that I don't think is possible right
now) is to have some sort of reverse defined RPC where the server
could call the client with a defined message e.g.

//Server notifies client of move
rpc notifyMove(MoveEntity) returns (VOID);

(As a side question what's the proper way to have a rpc message take
no parameters, and how do you have it return nothing, like a void
method. I've worked around it by creating a blank VOID message
object).

Maybe Protocol Buffer isn't meant to do this type of task, but I know
from experience that a standard binary protocol message code binding
generator would be very useful for network protocols that are more
"event" based and less deterministic, there just needs to be a way to
determine what type of message your decoding (generally a unique ID is
appended to the message so the recipient knows what it's decoding).


Here's my complete game network .proto example:

package game;

option java_package = "net.ddaniels.game.network.generated";
option java_outer_classname = "GameProtocol";

message LoginInfo {
required string name = 1;
required string password = 2;
}

message Entity {
required string name = 1;
required int32 id = 2; // Unique ID number for this person.
required int32 hp = 3;
required int32 attackPower = 4;

enum EntityType {
PLAYER = 0;
NPC = 1;
ENEMY = 2;
}

}

message Attack {
required int32 targetEntityId = 1;
}

message DamageEntity {
required int32 hpLost = 1;
required int32 targetEntityId = 2;
optional int32 sourceEntityId = 3;
}

message Vector2f {
optional float x = 1 [default = 0.0];
optional float y = 2 [default = 0.0];
}

message MoveEntity {
required int32 targetEntityId = 1;
required Vector2f location = 2;
}

// A list of all entities in the world
message EntityList {
repeated Entity entity = 1;
}

//Void so i can pass no argument RPC calls
message VOID { }

service GameService {
rpc getAllEntities (VOID) returns (EntityList);
rpc login(LoginInfo) returns (Entity);
rpc attack(Attack) returns (VOID);
}

message GameProtocolMessage {
optional LoginInfo loginInfo = 1;
optional Entity entity = 2;
optional Attack attack = 3;
optional DamageEntity damageEntity = 4;
optional EntityList entityList = 5;
optional MoveEntity moveEntity = 6;
}

message GameProtocolMessages {
repeated GameProtocolMessage messages = 1;
}

Kenton Varda

unread,
Aug 1, 2008, 1:47:32 AM8/1/08
to Doug Daniels, Protocol Buffers
Using a set of optional messages is a reasonable way to solve this (the optional messages that are not filled in will not take any space on the wire).  You might also want to include an enum which identifies which of the messages is filled in, so that you can switch on it.

message GameProtocolMessage {
  enum Type { ... }
  required Type type = 1;

  optional LoginInfo login_info = 2;
  optional Entity entity = 3;
  ...
}

Another possibility is to do something like this:

message GameProtocolMessage {
  enum Type { ... }
  required Type type = 1;
  required bytes message = 2;
}

Here, "message" itself contains an encoded protocol message of the type identified by Type.

However, I think that using a set of optional fields is a better choice.  This is what we usually do at Google.  Note that if the number of optional fields becomes unwieldy you may want to switch to using extensions.  You can convert fields to extensions without breaking wire-compatibility.

If the input or output should be empty, just define an empty message type and use that.  If you decide later that you want it to be non-empty, you can easily add new fields.  If there were some generic "void" type you would not be able to change it later without updating all the code that uses your service.

Doug Daniels

unread,
Aug 1, 2008, 10:07:04 AM8/1/08
to Kenton Varda, Protocol Buffers
Thanks Kenton!

Sounds like creating a message with all the protocol messages as optional fields is the optimal way to go, and including an enum Type field would allow the network code to quickly switch on which message type is being received. It also sounds like with extensions I could have a main .proto defining:

GameProtocolMessage {
  enum Type { ... }
  required Type type = 1;
  //... Include up to 100 basic messages
  extensions 100 to 1024
}

and then I could seperate the game protocol into things like Combat.proto (extensions 100 to 300), Network.proto (extensions 301 to 500) etc. The only problem would be making sure that no extension id's clashed. The only problem is the main enum Type{} would have to be updated everytime an extension is updated.

The next question I have is, how could I best automate the creation of the network message decoder so that if I update the enum it could regenerate the switch{} statement that decodes the messages (or maybe in java I could create a HashMap<Type, Class<? extends Message>> that could lookup the class and then use reflection to instantiate it, but maybe a inline switch statement would be faster than using reflection.).

I'm thinking that if I follow a naming convention same as the Message definitions for the enum Type then I could have my own custom code generator make a pass over it and generate the network translation code when my .proto file is updated.

What I really like about the Protocol Buffer approach to binary message code generation is that it makes the process as automated as possible, I've dealt with some binary protocol code bases where a new message would require 4 or 5 manual steps, before you could even write the code to send/receive that message.

Johan Euphrosine

unread,
Aug 1, 2008, 2:56:28 PM8/1/08
to Doug Daniels, Kenton Varda, Protocol Buffers
Hi,

I'm implementing something using Protocol Buffers for client server
Game API here:
http://proppy.aminche.com/hg/protobuf-packet/file/tip/

It Targets Python/Twisted and C++/BoostAsio.

And uses message.GetDescriptor().name() for type dispatching, (and not
optional fields), and a byte field for message nesting.

This is the only way I figured out, for separating the network and
dispatch code from the client code.

Hope it was worth sharing.

Source are availabe using mercurial:
hg pull http://proppy.aminche.com/hg/protobuf-packet/

--
bou ^

Kenton Varda

unread,
Aug 1, 2008, 4:45:44 PM8/1/08
to Doug Daniels, Protocol Buffers
Sorry, I don't have any good answers as this problem seems to be getting outside the scope of protocol buffers.  Protobufs just deal with serialization, not dispatch.  Everything you said sounds reasonable to me, though.

One thing that might help would be to change the "type" enum to an integer so that you don't have to define all its possible values in one place.

Note that when using extensions, you will still probably want your top-level message to contain only one extension at a time, rather than having a range of extensions be set, since extensions are not as efficient as regular fields (they have to be stored in a hashtable rather than a simple struct).  Not sure if that's what you meant.
Reply all
Reply to author
Forward
0 new messages