GAE for a web based MMORPG

436 views
Skip to first unread message

Kaan Soral

unread,
Dec 19, 2011, 8:56:00 AM12/19/11
to google-a...@googlegroups.com
I have been thinking about this for a while, is this doable, how would the costs be?

My best scenario: every user posts actions it does to GAE, GAE broadcasts game state every 200ms to each user. This happens for every server, I guess servers wouldn't be able to grow that much. (server is not a real server, just imaginary)

vlad

unread,
Dec 19, 2011, 4:05:45 PM12/19/11
to google-a...@googlegroups.com
First thing to consider is how to deliver notifications and/or state updates to clients. Options:

- Short polling. Reasonable option. I am using it myself. It certainly has its cost.
- Long polling. Too expensive under new pricing regime.
- Channels API. Ill suited to this in my view.
- Ideally I would have websockets which GAE does not support.

vega

unread,
Dec 20, 2011, 3:23:13 AM12/20/11
to google-a...@googlegroups.com
you want to broadcast the actions (or results of those actions^^) from every player to every player?
how many players do you want to handle?
i mean, imagine you have 500 users at a time online - and each does just 1 operation.... that would result in 62k messages....
doesnt sound like the best solution to me

Andrius A

unread,
Dec 20, 2011, 3:34:20 AM12/20/11
to google-a...@googlegroups.com

And even if you decide to use channels api, you won't be able to deliver messages fast for masses of users. Channels api doesn't support broadcasting, so looping through users to send messages will take time.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/5DB_p45RCYIJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

Jeff Schnitzer

unread,
Dec 20, 2011, 8:05:10 AM12/20/11
to google-a...@googlegroups.com
Even worse than the communication issue is the fact that there's no
place to put game state. Putting it in the datastore is too slow &
expensive and putting it in a backend isn't reliable enough.
Gameservers need to have uptime of months at least; even with
checkpointing to the datastore, users are going to get really pissed
if their last 5 minutes of hacking and slashing dragons suddenly
disappeared.

Jeff

Gopal Patel

unread,
Dec 20, 2011, 9:57:35 AM12/20/11
to google-a...@googlegroups.com
can you do something like git ?

whenever there is state change, update hash and save as new entity. 

that way, you will have whole chain of event, current state, etc. 



Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

Kaan Soral

unread,
Dec 20, 2011, 2:38:40 PM12/20/11
to google-a...@googlegroups.com
My initial idea was to keep everything in one datastore entity, in a blob property, players would post changes to this entity, and when they request the game state, they would also get contents of this item
But I totally forgot about the 5/s datastore write limit for a single entity

So even with short-polling, it wouldn't work

Backend + short-polling sounds like a great idea now to me, although I didn't quite get the git idea

Gopal Patel

unread,
Dec 20, 2011, 9:44:15 PM12/20/11
to google-a...@googlegroups.com
you can compare a massive game to a version control system, with each file being their players, with state of game being state of directory. 

with that said, just study how git works. http://whygitisbetterthanx.com/


--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/vobs7E5lllYJ.

Andrius A

unread,
Dec 21, 2011, 4:07:13 AM12/21/11
to google-a...@googlegroups.com

You can increase entity update limit by using MS datastore and shards. This way you can have a rate of 100/s for your logical single instance. But don't forget that MS is not reliable, neither memcache or backends. You will face a nightmare trying to make it work fast and reliable.

--

Jeff Schnitzer

unread,
Dec 21, 2011, 12:12:44 PM12/21/11
to google-a...@googlegroups.com
I used to work on MMO games at EA. Not on the the core game engines,
but close enough to understand how they work.

Assuming you want to create an MMORPG (see: the subject line), there's
no way in hell this will work. The persistent-world state is composed
of every player, every monster, and what they are doing at any moment.
Every time a user clicks the mouse or hits a key on the keyboard it
changes the PSW state. Every "tick" of a timer can change the PSW
state. You can't just read and write this to the datastore - the data
volumes would be insane and even if you could afford the bill, the
latency will *destroy* your user experience.

MMORPG games, in my experience, are generally implemented as an
in-memory game state that gets written to a database at intervals.
The in-memory state of a shard is very long-lived; a badly-timed
reboot of a shard usually kicks everyone off the world. You could do
this with a GAE backend but they aren't reliable enough - users get
*very* pissy about getting booted off the server in the middle of
swordfight.

Another thing is that the MMORPG games I'm familiar with have been
implemented as more-or-less single-threaded async systems. There are
a lot of good reasons to do this. Maybe you can make a different
approach work but when you're trying to squeeze tens of thousands of
active players into a single PSW you pretty quickly hit the
scalability limits of synchronous architectures.

So really, if you are serious about building an MMORPG, you do not
want to use Appengine. If you're seriously thinking about using
Appengine, I can tell you right now that 1) MMORPG games are hard to
implement and 2) you don't know enough about how to design them to
make this work. Keep researching, there's a lot of literature out
there... and a few opensource projects to learn from. Even a few
frameworks (commercial and not). They will not run on GAE.

Note that this doesn't encompass all MMO games; you could build a
simple turn-based game on GAE quite easily. But MMORPGs are a
different beast.

Jeff

--
We are the 20%

Kaan Soral

unread,
Dec 21, 2011, 12:24:43 PM12/21/11
to google-a...@googlegroups.com
Thanks Jeff for the excellent Insight

A simple RPG is doable, a turn based game is doable, a MMORPG is impossible, I think I may shoot for something in between a MORPG and a RPG

Maybe servers can accommodate at most 100 people, so this will allow team-play and online-spirit.

But in any case I will certainly explore my options outside GAE at this point, thanks for everyone who contributed

Ikai Lan (Google)

unread,
Dec 21, 2011, 2:42:17 PM12/21/11
to google-a...@googlegroups.com
Hey Jeff,

Just out of curiosity, what did you shard by? Realm/region? Do the shards just run on gigantic machines? Does the gating factor tend to be amount of combat or players? I don't know if you can share any of this stuff - I just love hearing war stories. The game development world is one I know very little about. 

Fun fact: Alfred on the datastore team knows a TON about video game development. =)

--
Ikai Lan 
Developer Programs Engineer, Google App Engine

Jeff Schnitzer

unread,
Dec 21, 2011, 9:36:26 PM12/21/11
to google-a...@googlegroups.com
I don't think EA is going to hunt me down for spilling almost
decade-old history, so...

I worked on The Sims Online and Ultima X Odyssey (sadly cancelled).
Both were sharded the "boring way" - each shard was a separate
city/realm with an entirely separate set of character accounts. If a
shard was too full, it was closed to new players.

Each TSO shard (IIRC, there were 10 shards at launch) was a cluster of
20 top-of-the-line (at the time) 1U servers. Each shard was capable
of supporting 1k-2k peak simultaneous users... rather a disappointing
number for all that hardware, actually. The biggest bottleneck were
the "simulators" which actually ran code hacked out of The Sims PC
game to try to make the world a little more interesting than a chat
room (it didn't). Those poor boxes typically ran at load averages >
500, sometimes > 1000.

I know less about the UXO cluster since the core team was in Austin (I
was at EARS by then) and the product never made it to launch. We
never really found out how it would perform in the real world. It was
doubtlessly less CPU intensive than TSO but it was built with the same
ancient toolkit that EA inherited from some ill-considered acquisition
a half-decade prior. You can probably get a shudder out of most EA
Online veterans just by mentioning Aries/Voltron. At any rate, the
architecture was pretty similar so it's another "big cluster of
machines".

I didn't work directly on the game engine (just the login servers,
account management stuff, online community, etc) so I'd be talking out
of my ass if I offered advice on how to actually build one today...
except one thing: You want a garbage-collected language. C++ is a
horrid tool for the job. I don't think they ever got all the memory
leaks out of the servers... and segfaults take down a whole world. On
the other hand, Go looks like a language specifically designed for
MMORPG programming.

Jeff

Wilson MacGyver

unread,
Dec 21, 2011, 9:46:57 PM12/21/11
to google-a...@googlegroups.com
I worked on a MMO sometime back as well. and we too sharded the "boring way",
ie, by cities/zones. though there is a shard for each store too.

everything is designed to kept in ram as much as possible. it's all
FSMs in memory.

On Wed, Dec 21, 2011 at 2:42 PM, Ikai Lan (Google) <ika...@google.com> wrote:
> Hey Jeff,
>
> Just out of curiosity, what did you shard by? Realm/region? Do the shards
> just run on gigantic machines? Does the gating factor tend to be amount of
> combat or players? I don't know if you can share any of this stuff - I just
> love hearing war stories. The game development world is one I know very
> little about.
>
> Fun fact: Alfred on the datastore team knows a TON about video game
> development. =)
>


--
Omnem crede diem tibi diluxisse supremum.

Gerald Tan

unread,
Dec 21, 2011, 11:50:52 PM12/21/11
to google-a...@googlegroups.com
So, why wouldn't one backend = one shard work?

Brandon Wirtz

unread,
Dec 22, 2011, 1:38:25 AM12/22/11
to google-a...@googlegroups.com
You have to calculate the actions of all things in the shard. If that is
100 players, 50 MOBs and 10 NPCs with Trade, Combat and Item creation, that
is a LOT of CPU.

If you shrink the shard, then you have less going on.

In some games they do things like the user can only see so far so you have
hexagons and each Hex is a "shard" and the user pings all the "shards" that
it can see.

Some games have Object lists and you get updates from all the objects (kind
of like twitter, who am I subbed to at the moment)

There are lots of ways to do this, and you have to weight CPU, bandwidth,
Latency, and a number of factors.

--

You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit

https://groups.google.com/d/msg/google-appengine/-/WhKMhshsY30J.

Jeff Schnitzer

unread,
Dec 22, 2011, 2:17:20 AM12/22/11
to google-a...@googlegroups.com
...and don't forget that until you can expect 6months+ of uptime from
a backend, they are a non-starter for any kind of MMORPG. Users
*hate* getting bounced from a game. It's not just "oh I have to log
in again" - it's "oh I have to wander aimlessly through the maze to
find the freaking exit for fifteen minutes - AGAIN". It may take 15
minutes just to get your party together in the same place, equipped
with all the right weapons, all the right spells active, etc...

Persistent world games are hard.

Jeff

--
We are the 20%

Brandon Wirtz

unread,
Dec 22, 2011, 2:57:20 AM12/22/11
to google-a...@googlegroups.com
Ultima Online. Daily Downtime. No Voice Communication. Changed the Sharding
system at least 4 times in the beta...Rule changes with every patch...
That was the wild west... I killed Lord British, Rendered my self
invulnerable to death by everything but being "buffed" by having Negative
Hitpoints. And had a pet Chicken that could Solo a Dragon. It was a good
day.

Jeff Schnitzer

unread,
Dec 22, 2011, 3:12:56 AM12/22/11
to google-a...@googlegroups.com
And it's still running!

Jeff

Paul

unread,
Dec 22, 2011, 3:40:53 AM12/22/11
to Google App Engine
Ikai Lan:

Is Google secretly working on some game project? Are you hiring? :P


Jeff Schnitzer:

Maybe one day GAE will develop in a way that will allow games like
that written in Go...

Brandon Wirtz

unread,
Dec 22, 2011, 3:49:55 AM12/22/11
to google-a...@googlegroups.com
They are... it is called see how much spam can make it in the search index
so people have to click on ads, with out frustrating them enough to switch
to Bing.

It is very Massive Much more so than WoW and the Shards are very small, and
no Parties are allowed.

You can craft your own results, but they likely won't be included if they
aren't on WikiPedia.

Ikai Lan:


Jeff Schnitzer:

--

Ikai Lan (Google)

unread,
Dec 22, 2011, 1:38:56 PM12/22/11
to google-a...@googlegroups.com
The "game" being worked on is the opposite of what Brandon describes.

We are definitely working on games in Google+ and yes we ARE hiring:


--
Ikai Lan 
Developer Programs Engineer, Google App Engine



Ikai Lan (Google)

unread,
Dec 22, 2011, 1:39:29 PM12/22/11
to google-a...@googlegroups.com
Clarification: I'm talking about this, of course, as well as enabling people to build Android/Chrome games:


--
Ikai Lan 
Developer Programs Engineer, Google App Engine



James Jones

unread,
Dec 22, 2011, 1:48:13 PM12/22/11
to google-a...@googlegroups.com
you could some how off load the data store to a memcache or hadoop instance instance in EC2, but you have, but you have to run all the queries and writes over https scripts that interacts with the storage engine in ec2. It would most likely have a performance hit . I have done similar type of solution which works for a web based app, but may not be fast enough for a game.

timh

unread,
Dec 24, 2011, 4:14:18 AM12/24/11
to google-a...@googlegroups.com
Whilst you are doing your research, have a look at how EVE is built.
Its a single game universe, run on stackless python.  You won't be able to do this on 
appengine either, but there is lots to be learnt from those guys if you into game backends.

Cheers

T  

Brandon Wirtz

unread,
Dec 24, 2011, 4:56:35 AM12/24/11
to google-a...@googlegroups.com

Actually the EVE architecture is almost exactly how I would do it.

 

Caching proxies serve requests for the state of visible regions. (you may be able to see 4 squares in any direction of you, so you make 25 requests to get data from each square)

 

Objects have a Type when you interact with an object you interact with instances that handle that interaction.

 

Since everything else is Read only Presentation you don’t have too many writes or reads.

 

The world lives in Memcache and Databases.

 

Where GAE would run in to issues is that Memcache isn’t going to be big enough and Datastore isn’t going to be fast enough.

 

Now I do have a hack to get around it, that is with in the TOS of GAE…. Multiple Versions of the same ID share Datastore, but NOT memcache. So if you said each region/hex/Square was a version, you could keep the memcaches up to date and have more than enough memcache to have a world.

 

So you as standing on grid 4,5  you would ask for data from   1,2   1,3  1,4 1,5 1,6 1,7 etc… 25 requests looping through as 1-1.yourapp.appspot.com to keep the world updating,  If you interacted with an NPC you’d hit npc.appspot.com with their ID. And Poof everything runs as it should.

 

Again all http Requests will limit how latency free you are.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/O_PQrnrgegEJ.

Reply all
Reply to author
Forward
0 new messages