Design patterns for servers with long life sockets?
magnus....@gmail.com
out of curiosity and personal interest I'm writing a simple
multiplayer game server that uses TCP.
Players will be able to log in with username and password, and with
each account one and only one "character" is associated. The game
itself will be very simple - that's not what I want to discuss.
I'm trying really hard to find an elegant structure for the whole
"connect", "log in", "start playing" procedure. Primarily it gets very
messy with the socket - character association and the messages that
need to be considered and potentially discarded, depending on "client
state". Multiple threads make it even worse.
Are there any useful design patterns out there for creating elegant,
robust and maintainable servers that have persisting sockets? What I
mean by persisting sockets is that the server does not function as a
pure "read, interpret, reply, close" service, but rather where sockets
are kept open for a long time and a continuous stream of messages are
sent back and forth. Usually this type of server also has several
states in which clients will be considered - depending on whether they
are logged in or not and so forth.
The current design:
* Server thread accepts socket
* Server thread dispatches this socket to a new client handler which
is enqueued into a thread pool for execution.
* Client handler thread reads from socket, parses text messages and
puts them in the "World message queue".
* World thread pumps the world message queue for messages.
* World thread calls a ClientState (a java enum with possible values
connected, logged_in, disconnected), that is a member of the client
handler object, which kind of filters the incoming messages. This enum
also has data that associates messages with code, String -> Method
basically.
* ClientState may or may not acknowledge the message. If it does, it
will call a function in the client handler.
* Client handler raises an event, and.. world object listens.
* I get confused and miserable.
As you can see it's a big mess. What I'm really trying to get at is a
data-oriented and elegant approach to client/character -> message ->
code association, with an easily maintainable structure with very
little code added for new message types and features.
Any good resources/books for this type of servers?
Patrick May
I'm going to try to beat Phlip to the punch and ask "Do you have
unit tests for that?"
You're getting confused and miserable because you're trying to
solve the whole problem at once. Do the bits you understand, then
look at it again. The first thing you need to do is accept a
connection from a client. Write a test client that connects to the
server, and make sure it fails. Now write a server that accepts the
connection. Your test passes.
Now write another client test that sends a login message. Watch
it fail. Modify the server until the test passes.
Next, run two clients at the same time. Watch the server fail to
handle one or the other. Modify it until the test passes (select() is
your friend here).
As you continue to write your tests and build your server, you'll
probably find that the Simplest Thing That Could Possibly Work (look
it up) will be to separate the processing from the connection
handling. Don't think of queues yet, think of a blackboard. Shared
memory is useful for that. Look at Jini and JavaSpaces, and steal
ideas if you don't have to use Java.
Finally, write some more tests, watch them fail, and then make
them pass.
Regards,
Patrick
------------------------------------------------------------------------
S P Engineering, Inc. | Large scale, mission-critical, distributed OO
| systems design and implementation.
p...@spe.com | (C++, Java, Common Lisp, Jini, middleware, SOA)
Phlip
> I'm going to try to beat Phlip to the punch and ask "Do you have
> unit tests for that?"
Honest thanks: I had already declined to answer due to inexperience with
long-term sockets. A MMORG, in theory, calls for a custom protocol on UDP, not
TCP, but then you get into hacking, security, and other fun murky middleware
issues that I have never actually done. I _try_ not to mention automated tests
if I don't have anything else to say...
The point of UDP is if your sockets are long-life but stateless, then they are
really short life. If the UDP packet does not arrive, then it does not, and life
goes on...
--
Phlip
Patrick May
> Patrick May wrote:
>> I'm going to try to beat Phlip to the punch and ask "Do you
>> have unit tests for that?"
>
> Honest thanks: I had already declined to answer due to inexperience
> with long-term sockets. A MMORG, in theory, calls for a custom
> protocol on UDP, not TCP, but then you get into hacking, security,
> and other fun murky middleware issues that I have never actually
> done. I _try_ not to mention automated tests if I don't have
> anything else to say...
Now, now, is that really the STTCPW? ;-)
I do a lot of distributed computing work and I've seen people end
up building TCP over UDP, badly, more than once. My gut tells me that
you're correct, but I don't think it's a foregone conclusion.
> The point of UDP is if your sockets are long-life but stateless,
> then they are really short life. If the UDP packet does not arrive,
> then it does not, and life goes on...
Resiliency in the face of unreliable components is essential.
Phlip
> Resiliency in the face of unreliable components is essential.
But that's what I meant: If one UDP does not get thru, the next one will, and
the state will get updated anyway. Doesn't streaming audio use UDP simply to
permit endlessly wide broadcasts without using a dedicated connection to every
recipient? Streaming MMORG graphics should work like that too, right?
And I might recall overhearing that some existing MMORG middleware stack used
UDP - but maybe I'm just mis-remembering.
--
Phlip
Patrick May
I was agreeing with you! Now you had to go and make it _violent_
agreement!
Dmitry A. Kazakov
> I'm trying really hard to find an elegant structure for the whole
> "connect", "log in", "start playing" procedure. Primarily it gets very
> messy with the socket - character association and the messages that
> need to be considered and potentially discarded, depending on "client
> state". Multiple threads make it even worse.
>
> Are there any useful design patterns out there for creating elegant,
> robust and maintainable servers that have persisting sockets?
(I don't know how sockets can "persist". Do you mean to keep a dead socket
object in order to reuse?)
> The current design:
> * Server thread accepts socket
> * Server thread dispatches this socket to a new client handler which
> is enqueued into a thread pool for execution.
> * Client handler thread reads from socket, parses text messages and
> puts them in the "World message queue".
That is when the message is to change the state of the world. But I guess
that rather each client has some associated "agents" in the world to be
influenced by the message, and these agents in turn influence the world and
each other. When an agent receives a message it can handle it (change the
state of itself), respond to the client, when necessary (normally it would
be wasting resources if you are using TCP/IP), and then it can communicate
its new state with the world or other agents (over messages or any other
mechanism).
All in one, I think you have to separate communication with the clients and
agents, active objects of the world.
> * World thread pumps the world message queue for messages.
> * World thread calls a ClientState (a java enum with possible values
> connected, logged_in, disconnected), that is a member of the client
> handler object, which kind of filters the incoming messages. This enum
> also has data that associates messages with code, String -> Method
> basically.
As I said above, I don't see a reason why this need to be pumped through
the world thread. But if you want to, put a reference to the object
servicing the client to the message and call Handle_Me on it. You could use
rendezvous to avoid messages, but Java does not have them, AFAIK.
> As you can see it's a big mess.
Hmm, not very big, actually. You need a bit more design to restructure the
problem.
> What I'm really trying to get at is a
> data-oriented and elegant approach to client/character -> message ->
> code association, with an easily maintainable structure with very
> little code added for new message types and features.
> Any good resources/books for this type of servers?
I think any book on OO design would help. Look, you are talking about
"data-oriented" approach, that is a non-starter here!
(I think topmind might have a reserved opinion on the matters (:-))
--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de
Dmitry A. Kazakov
> But that's what I meant: If one UDP does not get thru, the next one will, and
> the state will get updated anyway.
Right, provided you are dealing only with the process variables, which are
continuous in their nature. Velocity, for example. This stops working when
you need to handle events and commands. Yet another issue is data
consistency. That is when a state is composed of other states. Consider an
double value of which the first word is lost, but the second was delivered.
And UDP not only does not warranty delivery, it also does not the order of.
This may have very nasty effects on the simulation, like artefacts, jerky
movements etc, even if nothing gets lost.
Actually the only use of UDP was for broadcasting. Luckily, we finally have
in-order delivery safe multicast protocols. So UDP may rest in peace.
magnus....@gmail.com
the whole log in procedure.
I have actually had the server working quite well, even to the point
where after loggin in the client would receive a world definition (a
2d grid basically, think pacman).
Then I became dissatisfied with the structure, wanted more elegance...
and started messing about with queues and threads all over the place,
so now it's broken and I can't make up my mind on a design. :(
What I want to avoid is a ton of if/else if/else if/else if statements
for all the message handling and filtering..
> The point of UDP is if your sockets are long-life but stateless, then they are really short life. If the UDP packet does not arrive, then it does not, and life goes on...
Well, how does one reliably handle logging in and staying logged in
with UDP? Isn't that a sort of state?
> (I don't know how sockets can "persist". Do you mean to keep a dead socket
object in order to reuse?)
No. Consider a http server. It accepts a connection, reads a GET
command, builds a reply, sends the reply, closes the socket. This is
very simple and is easily programmed. But when you have a continuous
stream of messages back and forth, there is a need for more structure
to manage all the clients in an elegant fashion. I'm simply looking
for some design patterns that will properly constrain the code from
going messy and heterogenous.
Dmitry A. Kazakov
> But when you have a continuous
> stream of messages back and forth, there is a need for more structure
> to manage all the clients in an elegant fashion.
That is rather the normal case. I don't see anything difficult in hading
it. You have some object responsible for a connection. Upon establishing a
connection, the listener creates the object and passes the socket to it.
When the connection is closed, the object dies and upon its finalization it
closes the socket handled by it. Depending on the communication mode, half-
vs. full-duplex you may have one of two threads associated with the object.
Well, in a full-blown middleware there also could be more complex things
like events schedulers, which ensure certain quality of service for the
subscribers, perform coalescing of events etc. But I doubt that this would
apply to your case.
Phlip
> Thanks for the replies. I do already have an automated test that does
> the whole log in procedure.
Small or large granularity tests?
> I have actually had the server working quite well, even to the point
> where after loggin in the client would receive a world definition (a
> 2d grid basically, think pacman).
Patrick is about to give you one of his standard lectures about how lots of
small-granularity tests are a design technique, allowing you to refactor a bad
design - in between adding features and deploying versions - and fix the design
as you learn more about its requirements. Lots of tests make starting with a bad
design safe.
> Well, how does one reliably handle logging in and staying logged in
> with UDP? Isn't that a sort of state?
If UDP may indeed rest in peace, then we are talking about a layer over TCP that
is...
- stateless
- secure
- broadcastable
- guaranteed order-of-delivery
Put the first two together, then any datagram that arrives must contain a unique
code that can only have been generated by a combination of the secret keys that
your client and server exchanged at login time. So login and security provide
state over stateless wire protocols.
Then, each datagram should update an object model. That is "event driven
programming", the arch-enemy of stateful connection programming.
--
Phlip
S Perryman
> out of curiosity and personal interest I'm writing a simple
> multiplayer game server that uses TCP.
> Players will be able to log in with username and password, and with
> each account one and only one "character" is associated. The game
> itself will be very simple - that's not what I want to discuss.
> I'm trying really hard to find an elegant structure for the whole
> "connect", "log in", "start playing" procedure. Primarily it gets very
> messy with the socket - character association and the messages that
> need to be considered and potentially discarded, depending on "client
> state". Multiple threads make it even worse.
> Are there any useful design patterns out there for creating elegant,
> robust and maintainable servers that have persisting sockets? What I
> mean by persisting sockets is that the server does not function as a
> pure "read, interpret, reply, close" service, but rather where sockets
> are kept open for a long time and a continuous stream of messages are
> sent back and forth. Usually this type of server also has several
> states in which clients will be considered - depending on whether they
> are logged in or not and so forth.
1. What you are looking for (in OSI terms) is comms *session management* .
A session is a communications channel that is ignorant of the *transport*
resource activity needed to maintain that session.
With TCP being an instance of a transport resource, what this means is
any of the following may occur over the lifetime of the session :
- transport connections may be established/released any number of times
- any number of transport connections may be concurrently established to
support the session
2. Now, given session (data rate/throughput/latency etc) and transport
(IP port availability etc) requirements, this will determine how the
transport resources are utilised.
3. The simplest framework is to have a game session component.
The simplest implementation is to associate one TCP port with the
session, and have the transport connection open for the lifetime of
the session.
So ...
- Start with 3.
- Have a cursory look for something suitable providing 1.
- Gather metrics for 2. Change your session impl accordingly.
- If you feel you are doing a lot of work implementing the session,
do a more serious search on #1.
Regards,
Steven Perryman
Daniel T.
> out of curiosity and personal interest I'm writing a simple
> multiplayer game server that uses TCP.
>
> Players will be able to log in with username and password, and with
> each account one and only one "character" is associated. The game
> itself will be very simple - that's not what I want to discuss.
>
> I'm trying really hard to find an elegant structure for the whole
> "connect", "log in", "start playing" procedure. Primarily it gets very
> messy with the socket - character association and the messages that
> need to be considered and potentially discarded, depending on "client
> state". Multiple threads make it even worse.
[snip]
> Any good resources/books for this type of servers?
Phlip
> http://www.cs.wustl.edu/~schmidt/PDF/Act-Obj.pdf
Ooh, yeah!
Now, Patrick, you need to come down on Dr Schmidt like a tonne of onions, too,
regarding his students' _unit_ test situation!
Patrick May
Sheesh, I channel you once and now I'm forever haunted. Time to
call my friend who was raised in Santeria....
It was nice to see the Booch clouds again, though.
Regards,
Patric
Phlip
> call my friend who was raised in Santeria....
>
> It was nice to see the Booch clouds again, though.
Hmmmm. Santeria... clouds... Hmmmm...
--
Homer Simpson
magnus....@gmail.com
>
> Try:http://www.cs.wustl.edu/~schmidt/PDF/Act-Obj.pdf
Very cool from first look. Will definitely read the whole paper. This
is what I was looking for.
Phlip
Warning: It will lead you to ACE, which will appear to satisfy your
requirements... at the cost of your mortal soul. Other than that - no strings
attached!
--
Phlip
Daniel T.
You say that like its a bad thing...
magnus....@gmail.com
> Warning: It will lead you to ACE, which will appear to satisfy your
> requirements... at the cost of your mortal soul. Other than that - no strings
> attached!
>
> --
> Phlip
No worries - I intend to keep this in Java.. perhaps using some
existing tools like:
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/FutureTask.html
and
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ArrayBlockingQueue.html
and
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/Executor.html
Dmitry A. Kazakov
(Huh, earlier you have responded that active objects were too complex for
you... (:-))
But the pattern indeed is quite common in the middleware, as the authors
state. We have deployed something alike on multiple occasions. It is of
course a bit simplified as described in the paper. Nevertheless it is a
good start.
Patrick May
>> Warning: It will lead you to ACE, which will appear to satisfy your
>> requirements... at the cost of your mortal soul. Other than that -
>> no strings attached!
>
My condolences.
> . perhaps using some existing tools like:
> http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/FutureTask.html
> and
> http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ArrayBlockingQueue.html
> and
> http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/Executor.html
You left out http://www.jini.org. Look at the Master-Worker
pattern, mediated by a JavaSpace, among others.
Regards,
Patrick
H. S. Lahman
Perryman and Daniel T. suggested variations on a basic theme: separate
the concerns. IOW, render unto TCP the things that are communication and
render unto the Game things that are gamey.
The game needs to communicate with various players. TCP provides a
mechanism to do that. But the game doesn't care if you talk to the
player via TCP or smoke signals; it just needs to pass information to
and receive information from the player independently of particular
communication paradigms. If you provide a "firewall" interface to
separate those views you will find it is much easier to resolve them
individually.
The best way to do that is to identify subject matters that are distinct
by needing different paradigms, rules, and levels of abstraction. Then
encapsulate each subject matter as a subsystem or layer behind a generic
interface that captures the invariants of the subject matter from the
client's perspective. Then you can resolve each subject matter as
essentially a standalone problem of mapping the client's needs into the
way the subject matter views the world, which should simplify things a
great deal because you can abstract each subject matter domain in a
highly tailored fashion without being distracted by details in other
subject matters. (The "Application Partitioning" category on my blog
talks about this in more detail.)
I suspect you will find it convenient to think of <at least> three
distinct kinds of functionality: (1) game activities (e.g., login, ogre
attacks, game threads, etc.); (2) high level control of network
communications (e.g., opening sockets, queuing messages, etc.); (3) low
level message network grunt work (e.g., message formatting, ack
handshaking, etc.). In particular, note that (3) needs to know nothing
about game semantics; it just manipulates messages and talks to a port.
Similarly the view of game semantics in (2) is also very limited; it
effectively encodes/decodes interface messages into subsystem
abstractions and re-dispatches to methods based on an interface message
identifier.
--
There is nothing wrong with me that could
not be cured by a capful of Drano.
H. S. Lahman
h...@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
in...@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH
magnus....@gmail.com
> blog:http://pathfinderpeople.blogs.com/hslahman
> "Model-Based Translation: The Next Step in Agile Development". Email
Yep yep, I was doing something a little like that already, just very
unstructured. Currently working on applying the active object pattern
to my code.
Thanks.