Best NoSqL for multiplayer android games.

889 views
Skip to first unread message

amit

unread,
May 1, 2012, 5:03:28 AM5/1/12
to nosql-di...@googlegroups.com
Could anybody please tell me which NoSql to use, mongodb or couchdb ? It will be nice if you could refer some reliable link for guidance. I want to develop multiplayer android games. Thanks a lot.

Emin Gun Sirer

unread,
May 1, 2012, 12:55:15 PM5/1/12
to nosql-di...@googlegroups.com
Glib response:
If you think your choices are limited to just those two NoSQL systems,
you should use HyperDex.

More reasoned answer:
It sounds like you're new to NoSQL and are not sure what properties
you want from your data storage layer yet. The weak consistency
properties of most NoSQL systems make the developer's task difficult.
For instance, managing something universal and easy, like a high-score
list, is going to take some effort to design for most NoSQL stores
unless you're okay with dropping some high scores during periods of
heavy activity or failures. So, go with HyperDex, which provides a
property called "linearizability" for all key-based operations. It
also helps that HyperDex is very very fast -- it's about an order of
magnitude faster than MongoDB on the industry-standard YCSB benchmark.

Best of luck with the game,
- egs
> --
> You received this message because you are subscribed to the Google Groups
> "NOSQL" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/nosql-discussion/-/S92u5J7DbrcJ.
> To post to this group, send email to nosql-di...@googlegroups.com.
> To unsubscribe from this group, send email to
> nosql-discussi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/nosql-discussion?hl=en.

Joe Stein

unread,
May 1, 2012, 12:57:40 PM5/1/12
to nosql-di...@googlegroups.com
my glib response was going to be

If you think your choices are limited to just those two NoSQL systems, then you should choose MySQL

Amit Kumar Maurya

unread,
May 1, 2012, 1:00:26 PM5/1/12
to nosql-di...@googlegroups.com
you are correct, I am new to nosql system. any idea about couchbase, where it stands as far as scalability and speed is concerned.
--
Regards,

Amit Kumar Maurya

unread,
May 1, 2012, 1:01:46 PM5/1/12
to nosql-di...@googlegroups.com
@Joe, can mysql handle high no of concurrent users ?
Regards,

Steve Graves

unread,
May 1, 2012, 1:51:37 PM5/1/12
to nosql-di...@googlegroups.com

What Android devices are you targeting?  I’d investigate the resource consumption (memory, mostly) of the options you’re considering and make sure they’re appropriate for your target devices.  A NoSQL solution that gobbles up 500K or more of memory for code+stack+heap will not be happy on a device with 1GB of memory.  (I’m just pulling numbers out of the air; I don’t know the footprint of HyperDex, MongoDB or Cassandra are.  MySQL in its embedded form is about 500K for code.)

 

Are you concerned about high number concurrent users on a single device?  Seems contrary to the Android market (mobile devices).  MySQL can handle a lot of concurrent users, but it’s not a NoSQL solution and wouldn’t scale in the context of a massive multiplayer game, which is what I /think/ you’re contemplating (i.e. a bunch of players, each with their own Android device, playing a game together).

Amit Kumar Maurya

unread,
May 1, 2012, 2:03:16 PM5/1/12
to nosql-di...@googlegroups.com
Ok, i think i shall use an example. Just think about a tic tac toe mobile game. it is a two player game. players will use their mobiles as interface. you need a server to coordinate the game and store some values, temporary or permanently. In case the game observes too many users then the database should be able to take the load and do so many operations with smallest possible lag. For such scenario, which db should be used ?

Konstantin Osipov

unread,
May 1, 2012, 2:07:37 PM5/1/12
to nosql-di...@googlegroups.com
* Joe Stein <cryp...@gmail.com> [12/05/01 21:15]:
> my glib response was going to be
> If you think your choices are limited to just those two NoSQL systems, then
> you should choose MySQL

+1

--
http://tarantool.org - an efficient, extensible in-memory data store

Steve Graves

unread,
May 1, 2012, 2:29:46 PM5/1/12
to nosql-di...@googlegroups.com

That helps.  For that scenario, MySQL should work fine.  You could migrate to MySQL Cluster if a single instance started to sag under the load (but then your architecture becomes substantially more complex). 

 

Think about using a connection pool instead of a one-to-one mapping of player-to-MySQL-user.  The players probably only need intermittent access to the database, not a persistent connection, so one MySQL connection could serve N players. The value of N will be a function of the definition of “intermittent”, and the size of the connection pool can grow/shrink dynamically with the demand. 

 

Also, data associated with gaming tends to be structured, not unstructured (NoSQL is a better fit for unstructured data), which is another factor in favor of MySQL.

Amit Kumar Maurya

unread,
May 1, 2012, 2:43:10 PM5/1/12
to nosql-di...@googlegroups.com
so I would ask this. can mysql handle 10,000 concurrent connections. also, why omgpop used couchbase, which is nosql, for draw something ?

Martin Bruse

unread,
May 1, 2012, 2:49:06 PM5/1/12
to nosql-di...@googlegroups.com

Don't you mean 10000 concurrent users? There is seldom need for 10000 concurrent connections.

Perhaps if you were running 10000 independent server processes, but that is unlikely, and not what one ought to design for from start.

What you need to consider when selecting database layer is, at least: writes/sec, reads/sec, types of access (indexes, ordered collections, joins, searching), consistency, availability, persistence.

What kind of profile, in those terms, do you foresee?

//Martin

Joe Stein

unread,
May 1, 2012, 2:52:39 PM5/1/12
to nosql-di...@googlegroups.com
it comes down to how much RAM you have and the number of reads and writes you are trying to-do (and the amount of data involved).  If your data set can fit in RAM then it MySQL will return 100% reads from memory

mysql can handle 10,000 concurrent connections... if you use the Percona XtraDB engine it can even handle quite a bit more http://www.percona.com/software/percona-server/benchmarks/

it also comes down to what you are comfortable to mange and maintain in production.  no matter what system you chose it will only work out of the box until it starts to see real usage and then you will have to start tuning it

the biggest issue I have with MySQL is honestly sharding when the data set grows beyond the limits of the system.... I solve this by not using MySQL for those use cases and using something like Cassandra instead (but in my use case we are talking about TB of new data every single day) and Hadoop for the analytics portion.  For a single self contain data set (for me this is < 500GB per table) with enough RAM mysql is quite fantastic.  

If I don't want to bother or can't bother with the schema (because the data is so fluid yet still not very large) then I would look into using MongoDB (perhaps also CouchDB nothing against it just have no experience with it but to me same difference).  again, different set of problem that creep up with those.

good luck, I think you need to focus in more on the product because whatever you select will ultimately have problems you will have to troubleshoot you are only picking the wrong solution if you can manage the growth with it... and if the business does not get off the ground then it won't matter anyways.

Emin Gun Sirer

unread,
May 1, 2012, 3:02:14 PM5/1/12
to nosql-di...@googlegroups.com
>the biggest issue I have with MySQL is honestly sharding when
>the data set grows beyond the limits of the system...

And this is exactly why he is asking for a NoSQL solution.

I guess he could convert every move on the board game into a SQL
statement that performs an update. He could hope that he can hide the
latency of MySQL operations within the communications latency of the
mobile device. And as his system grows, he could first desperately try
to tune the countless (and I mean this literally -- if anyone has a
count that is not NaN, please share) different settings that affect
MySQL performance, and then give up to move to a NoSQL store. At that
point, he'll have to figure out which NoSQL engine to use, and a shift
from MySQL's ACID properties to Mongo's or CouchDB's eventual
consistency will not be easy, to put it mildly.

I've given the "use MySQL" advice on this board, but this is not a
good case for it. The game industry is such that the system needs to
be designed with scalability in mind from the get go. So I find the
advice to use MySQL to be misguided.

- egs

Amit Kumar Maurya

unread,
May 1, 2012, 3:09:16 PM5/1/12
to nosql-di...@googlegroups.com
you are correct emin. i want a system which can handle heavy traffic easily, i really cannot change a lot of things during production, when i shall be focusing on upgrading the game features. 

Konstantin Osipov

unread,
May 1, 2012, 4:19:08 PM5/1/12
to nosql-di...@googlegroups.com
* Amit Kumar Maurya <am...@igidr.ac.in> [12/05/01 23:05]:
> Ok, i think i shall use an example. Just think about a tic tac toe mobile
> game. it is a two player game. players will use their mobiles as interface.
> you need a server to coordinate the game and store some values, temporary
> or permanently. In case the game observes too many users then the database
> should be able to take the load and do so many operations with smallest
> possible lag. For such scenario, which db should be used ?

How many writes per second do you expect? How many reads?
How big is the data set?

What sort of data is stored? Is it key/value? Object? Relational?
Graph?

These are basic questions one needs to answer to begin choosing a
database.

Konstantin Osipov

unread,
May 1, 2012, 4:28:52 PM5/1/12
to nosql-di...@googlegroups.com
* Amit Kumar Maurya <am...@igidr.ac.in> [12/05/02 00:05]:
> you are correct emin. i want a system which can handle heavy traffic
> easily, i really cannot change a lot of things during production, when i
> shall be focusing on upgrading the game features.

Sharding with MySQL is easy if planned upfront.

In fact, all large web shops like Facebook or Google successfully
shard their MySQL installations.

There are two reasons people may need to shard: to scale RPS or to
scale the size of the data set, or both.

It's amazing how a whole bunch of recommendations emerged without
knowing the two most basic facts about the future app.

Amit Kumar Maurya

unread,
May 1, 2012, 4:39:00 PM5/1/12
to nosql-di...@googlegroups.com
@ Konstantin
there might be 100, 000 writes per second and same amount of read operations. data will be key/ value type.
what is meant by " How big is the data set? ". if u meant the total db size, then that will grow with time as users will play and each game history has to be saved.

--
You received this message because you are subscribed to the Google Groups "NOSQL" group.
To post to this group, send email to nosql-di...@googlegroups.com.
To unsubscribe from this group, send email to nosql-discussi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nosql-discussion?hl=en.




--
Regards,

Kunthar

unread,
May 2, 2012, 3:30:27 AM5/2/12
to nosql-di...@googlegroups.com
Hi Amit,

Please divide your needs according to DB operations for client side
and server side to meaningful parts.

- It is not good idea to hold huge data in mobile OS.
- Distributed Nosql DB will be operate on SERVER side, not on the client side.
- You can use a kinda REST service with JSON or even better a binary
protocol to talk with remote distributed DB.
- You can have very basic and small amount of data on client side
which can be stored in structured file even sqlite db.
- All you have to do is to serialize/de serialize incoming/outgoing
data when you do CRUD operations.
- I strongly advise that you never design all your system real time
synchronized. Strongly advise that, always try to go with soft real
time[1] approach.


[1] http://goo.gl/TYzB2

Hope this helps,
BR,
\|/ Kunthar

"Marcelo D. Ré"

unread,
May 2, 2012, 9:03:41 AM5/2/12
to nosql-di...@googlegroups.com
El 01/05/12 15:07, Konstantin Osipov escribi�:
> * Joe Stein<cryp...@gmail.com> [12/05/01 21:15]:
>> my glib response was going to be
>> If you think your choices are limited to just those two NoSQL systems, then
>> you should choose MySQL
> +1
>
I propose DB4O. It work in Android and is really fast.

Amit Kumar Maurya

unread,
May 2, 2012, 3:50:21 PM5/2/12
to nosql-di...@googlegroups.com
couchbase claims "sub-millisecond random reads and writes of data, across the entire scaling spectrum".
what could be issues in using couchbase for the purpose ?

On Wed, May 2, 2012 at 6:33 PM, "Marcelo D. Ré" <marce...@gmail.com> wrote:
El 01/05/12 15:07, Konstantin Osipov escribió:

* Joe Stein<cryp...@gmail.com>  [12/05/01 21:15]:
my glib response was going to be
If you think your choices are limited to just those two NoSQL systems, then
you should choose MySQL
+1

I propose DB4O. It work in Android and is really fast.
--
You received this message because you are subscribed to the Google Groups "NOSQL" group.
To post to this group, send email to nosql-discussion@googlegroups.com.
To unsubscribe from this group, send email to nosql-discussion+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/nosql-discussion?hl=en.




--
Regards,

Vinay Joosery

unread,
May 14, 2012, 10:21:19 AM5/14/12
to nosql-di...@googlegroups.com
This paper might be of interest.
It discusses the different types of data that need to be stored by a social MMOG with recommendations on how to handle each type.

Databases in Online Social Gaming:
http://www.severalnines.com/resources/white-papers

Good luck!

Vinay

http://www.severalnines.com/

Russell Sullivan

unread,
May 14, 2012, 7:24:44 PM5/14/12
to nosql-di...@googlegroups.com

Choosing the right NoSQL or SQL database should be done based on your gaming platform's use cases and the performance you need to support now ... and if you think you will be successful, then base it on the performance you will need when you are successful, (i.e. plan for scaling)

Games differ from websites in that games have higher velocity data and normally historical data is not as important as what is happening right now in the game. In that sense, Mysql is a terrible fit if you ever need to scale a game, I will make some points:
1.) Some NoSQL stores are 50X faster than Mysql (the Percona 120K transactions are per MINUTE, redis has numbers like this per SECOND).
2.) You can patch a bunch of 3rd party alterations into Mysql (e.g. Percona & SeveralNines) to get over Mysql's velocity and replication problems, but patching software doesn't work as well as software designed from day one to solve particular problems. For example: adding new servers (AKA elasticity) is possible in Mysql, but (referencing my point #1) bringing the new node up will be 50X slower than w/ some NoSQL solutions (at which time the traffic spike has sunk your battleship)
3.) Games often benefit from schemaless data models (e.g. add in a new attribute for a player). Adding a new column (or god forbid a new index) to a live Mysql database is the number one reason Craigslist's Jeremy Zawodny dropped Mysql. Most NoSQL platforms are schemaless and some support dynamic index creation.
4.) Mysql is a high latency database compared to most Nosql datastores, Mysql uses a heavier thread based model, which means it has trouble supporting a high number of concurrent requests and MMOG is, by definition, all about concurrent users/requests.
5.) Sharding SQL (to horizontally scale) is NOT easy, there are numerous companies: [Scalebase, dbShards, clustrix] that have HUGE toolsets to help you do this and the reality is that you have to work w/ a subset of SQL and running distributed transactions results in massive slowdowns. On the flip side, building your own sharding layer is a huge undertaking (that is why companies specialise in it).
6.) Mysql is made for HDDs, its 18 years old, it has not been optimised for SSDs at all, this is important if your data is bigger than your RAM. Games benefit from SSDs by archiving older data (which often is close to useless to them, but not so useless that it should be thrown away). A few NoSQL solutions have SSD optimisations, there are (AFAIK) two that plug in natively to Fusion-IO.

So if your game really wants to operate at high velocity and be able to scale, the choices are the following: (NOTE: the real list is much longer, I am doing this off the top of my head)
1.) High Velocity: Redis, VoltDB, MongoDB, Citrusleaf, Membase
2.) Elasticity: VoltDB, MongoDB, Citrusleaf, Membase, Riak
3.) SSD optimised: Citrusleaf, Membase
4.) Rich Data Model: Redis, MongoDB

All of these NoSQL solutions have different trade-offs (e.g. data model richness vs. wan replication capable) and different strengths/weaknesses. IMO the gaming industry should be leaning more than almost any other industry toward NoSQL, because gaming needs high velocity, high data model flexibility, high availability, etc... and Mysql does NOT deliver on these, even patched w/ 3rd party software. So think on your application's needs, read up on relevant blogs @ highscalability.com & mynosql.tv, and get your game going fast and scaling-correctly for the day it's popularity explodes.

- jaksprats

Emin Gun Sirer

unread,
May 14, 2012, 10:52:52 PM5/14/12
to nosql-di...@googlegroups.com
Russel and others,

> 1.) High Velocity: Redis, VoltDB, MongoDB, Citrusleaf, Membase
> 2.) Elasticity: VoltDB, MongoDB, Citrusleaf, Membase, Riak
> 3.) SSD optimised: Citrusleaf, Membase
> 4.) Rich Data Model: Redis, MongoDB

Don't forget HyperDex under 1, 2 and 4. It's fast
(http://hyperdex.org/performance/ has a performance comparison to
Mongo, Cassandra and Redis), provides fault-tolerance and replication,
has a rich data model, and it guarantees strong consistency. It is
relatively new compared to the other systems, but we're dedicated to
supporting our users.

- egs

manu

unread,
May 15, 2012, 3:06:53 AM5/15/12
to nosql-di...@googlegroups.com
Amit

I am most familiar with Cassandra so my comment will obviously be biased towards this nosql techno (full disclosure, I work for Acunu, who contributes to the open source Cassandra project and also makes a high performance version packaged with mgmt tools available on AWS Marketplace and also real time analytic tools for Cassandra).

In London I noticed Cassandra happened to be adopted by many online gaming companies as their main store. Rather that telling you what it's good for I thought I would point you to the people using it and you can hopefully pick their brain on how they ended up with the decisions they made.

So I suggest you have a chat with
Dominic Williams from Fight My Monster. Dominic shares a lot of good info on his blog
Other games companies in London using cassandra include Mind Candy (Moshi Monsters), Bossa Studios, and a few others. Let me know if you want me to make some introductions as these don't have blog posts about their work.

Manu Marchal

>>> > To post to this group, send email to nosql-discussion@googlegroups.com.

>>> > To unsubscribe from this group, send email to

>>> > For more options, visit this group at
>>> > http://groups.google.com/group/nosql-discussion?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "NOSQL" group.
>>> To post to this group, send email to nosql-discussion@googlegroups.com.

>>> To unsubscribe from this group, send email to

>>> For more options, visit this group at
>>> http://groups.google.com/group/nosql-discussion?hl=en.
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> /*
>>> Joe Stein
>>> http://www.linkedin.com/in/charmalloc
>>> Twitter: @allthingshadoop
>>> */
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "NOSQL" group.
>>> To post to this group, send email to nosql-discussion@googlegroups.com.

>>> To unsubscribe from this group, send email to

>>> For more options, visit this group at
>>> http://groups.google.com/group/nosql-discussion?hl=en.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "NOSQL" group.
>>> To post to this group, send email to nosql-discussion@googlegroups.com.

>>> To unsubscribe from this group, send email to

>>> For more options, visit this group at
>>> http://groups.google.com/group/nosql-discussion?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "NOSQL" group.
>>> To post to this group, send email to nosql-discussion@googlegroups.com.

>>> To unsubscribe from this group, send email to

>>> For more options, visit this group at
>>> http://groups.google.com/group/nosql-discussion?hl=en.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "NOSQL" group.
>>> To post to this group, send email to nosql-discussion@googlegroups.com.

>>> To unsubscribe from this group, send email to

>>> For more options, visit this group at
>>> http://groups.google.com/group/nosql-discussion?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "NOSQL" group.
>>> To post to this group, send email to nosql-discussion@googlegroups.com.

>>> To unsubscribe from this group, send email to

>>> For more options, visit this group at
>>> http://groups.google.com/group/nosql-discussion?hl=en.
>>
>>
>>
>>
>> --
>> Regards,
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "NOSQL" group.
>> To post to this group, send email to nosql-discussion@googlegroups.com.

>> To unsubscribe from this group, send email to

>> For more options, visit this group at
>> http://groups.google.com/group/nosql-discussion?hl=en.
>
>
>
>
> --
>
> /*
> Joe Stein
> http://www.linkedin.com/in/charmalloc
> Twitter: @allthingshadoop
> */
>
> --
> You received this message because you are subscribed to the Google Groups
> "NOSQL" group.
> To post to this group, send email to nosql-discussion@googlegroups.com.

> To unsubscribe from this group, send email to

> For more options, visit this group at
> http://groups.google.com/group/nosql-discussion?hl=en.

--
You received this message because you are subscribed to the Google Groups "NOSQL" group.
To post to this group, send email to nosql-discussion@googlegroups.com.
To unsubscribe from this group, send email to nosql-discussion+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/nosql-discussion?hl=en.




--
Regards,

Reply all
Reply to author
Forward
0 new messages