using app engine for extremely demanding multiplayer browser game

[Karel Crombecq]

Hey guys,

I am currently evaluating different possibilities for making a sequel to a very popular text-based browser game called Castle Quest 2 (http://www.castlequest.be). The original game was written in php, with MySQL as the backend. I am investigating different hosting options for making a state-of-the-art, html5-based sequel to this game.

CQ2 is extremely demanding from a server perspective. About 1500 active users cause on average 3 million page views each day (90 million each month), for a bandwidth usage of 2,5GB per month. About 250 SQL queries are executed each second, and the database is slightly less than 1 GB large, with hundreds of tables. It is expected that, with the advent of social networks and social games, the player base will grow substantially, and the numbers for CQ3 will be multiples of those for CQ2.

Is GWT and app engine with the data store a viable platform for a game with these specifications? Will the datastore be able to deal with the pressure? I can't find any data about big apps hosted on app engine, so it is hard for me to get an idea of the scale of some apps that are hosted on app engine. The main alternative (Amazon EC2) has proven to be able to deal with this kind of pressure and much more (Farmville is hosted on it) but google app engine has lots of advantages over Amazon EC2, such as the servlet architecture, easy transparent scaling and integration with GWT, so I'm definitely leaning towards app engine if it can handle it.

Any feedback would be greatly appreciated.

Thanks in advance,
Kind regards,
Karel Crombecq

[jay]

Hello Karel,

I host 3 browser games on App Engine. Neptune's Pride, Blight of the
Immortals and my new one Jupiter's Folly.

My oldest and largest game Neptune's Pride has about 2000 Daily active
users, and about 7000 visits a day according to Google Analytics. My
games are flash, so they don't heaps of "page views", but the flash
makes 100-150k requests a day.

At times I have been much bigger and the data-store easily handles the
load. Even though there has been a lot of hoopla here about the price
increases, I would still recommend app engine to a small operation who
want to get on with the business of making games rather than managing
servers.

My bill is going from about 50c a day to about $3 a day. About $1000
a year.

Before the price increase my hosting was easily covered by the little
bit of advertising I have on the game (which players don't see unless
they have been playing 7 days without becoming a paying player). After
the price increase my ads will probably only cover 2/3s of my hosting
costs.

My biggest expense is datastore writes. Because the games are
interactive, not simply passive pages to view, the players make a lot
of changes to their data which needs to be written to the datastore.
Those 2k players of Neptune's Pride generate about 2 Million writes
which is $2 of my $3 a day bill.

Feel free to shoot me any other questions you might have, just keep in
mind I'm no expert! ;)

Jay.

[Karel Crombecq]

Hey Jay,

I actually registered yesterday on your game to get an idea of a game hosted on GAE. I'm enjoying it!

But the new pricing greatly disturbs me. I'm not sure if running this game on GAE is actually viable at all in terms of costs. I did some research on the new pricing (for example http://code.google.com/appengine/kb/postpreviewpricing.html#operations_charged_for) and as far as I can see, datastore reads and writes both have a similar cost. And they don't charge per query, but they charge per object (row) fetched.

I did some calculations on my current database data, and CQ2 generates about 1M database writes for something like 650 daily users. That's about 3 times as much as your game does, which would also triple the bill. That's a lot, but something I can handle. Since most of the writes are one-record only, the total cost would be $1,5 per day for 1000 users.

However, the datastore reads are the real issue here. I have about 4M SELECT queries for 650 users. Considering that many of these return more than one row, I can easily reach 10M datastore reads each day, for an additional cost of $2,8 each day.

This results in a total of €159 per month, for 1000 users. My estimations for the Amazon cloud were a cost of $65 per 1000 users each month (based on our current system and their instances), which would make GAE 3 times more expensive. That's quite worrysome, even though these statistics were generated based on relational database writes as opposed to datastore writes. It's hard to predict if I will need less or more datastore operations to achieve the same result. I'm actually thinking less, because I can cache a lot of static data into memory.

[Jeff Schnitzer]

It sounds like your game has a relatively small amount of rapidly
mutating data. In general, this is the kind of thing that GAE is bad
at. GAE is great for large amounts of data, each piece of which
changes slowly. Aside from the read and write cost, there is an
update limit of one transaction per "piece of data" per second which
can be very challenging to work around.

Your 1GB database easily fits in RAM, even if it grows 10X that size.
It will work well with a storage system optimized for resident memory,
giving you fantastic read and write throughput. This includes most
RDBMSes and some NoSQL stores like MongoDB. You can get thousands of
queries per second out of single instances.

The GAE datastore is optimized for datasets which cannot fit in RAM.
Fetches are relatively glacial, often taking tens of milliseconds.
The good news is that this performance will be consistent even when
you have 10 petabytes of data. The bad news is that you don't care,
because you only need to store a few gigs and what you really need is
fast (and cheap!) transactions.

If you want a PaaS solution to cut out some of the maintenance
headaches, consider Heroku or Appfog. If you want to try something
exotic, look into EC2's DynamoDB, which is supposed to be blistering
fast. Although - since your dataset fits in RAM, you probably are
best to stick with the database you already know - MySQL. Nothing is
faster than a fetch from RAM.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/_-Bf3oAZxu8J.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Kristopher Giesing]

Reads can be cached through the memcached service, and memcached hits are free.

If you try GAE, I would highly recommend re-examining your architecture to make optimal use of GAE's strengths.  I haven't released my own app yet, but when I started I tried to use JDO, and the performance was awful.  After switching to Objectify not only did my performance improve significantly, but nearly all the database reads disappeared due to the ease with which Objectify integrates with memcached.

- Kris

On Tuesday, September 13, 2011 4:03:10 AM UTC-7, Karel Crombecq wrote:

Hey Jay,

[Barry Hunter]

> Although - since your dataset fits in RAM, you probably are
> best to stick with the database you already know - MySQL. Nothing is
> faster than a fetch from RAM.

And dont forget AppEngine provides MySQL instances
https://developers.google.com/cloud-sql/

https://developers.google.com/academy/apis/cloud/appengine/cloud-sql/

[Jeff Schnitzer]

Unfortunately, experimental evidence suggests Cloud SQL provides
significantly less throughput (more than an order of magnitude) than
running MySQL elsewhere. However, I didn't run the tests myself; I'm
only reporting it secondhand. If you plan to use Cloud SQL for
anything performance sensitive, *definitely* run some tests. And
please report back :-)

Jeff