Mongodb 2.2, morphia and sharding

[Merl]

I am curious.  Is morphia still recommended to be used with Mongo?  We are having major sharding issues with Mongo.  At first it was believed that our issue was with our sharding key.  We changed the key to something more random, same problem.  We have upgraded to every new version of Mongo and still get the same result.  Our cluster loads an unreal amount of data until ram fills up.  Once ram fills up, the chunks starts building up on one shard and the cluster becomes unbalanced.  Then eventually the loading of data backlogs and the cluster becomes completely unresponsive.  The only thing that has not changed in our entire process is morphia.  We have used morphia from day one to load data into the system.  Should we be using something else?

[Sam Helman]

I do not know of any issues with Morphia, what is your current shard key?

[Merl Corry]

It is  2 random 8 character strings as a compound as a compound shard key.

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

[Sam Helman]

To isolate what might be causing your issues:

What are you connecting to from your application?  Is it a mongos, or is it a direct connection to a mongod?

When you say, "our cluster loads an unreal amount of data", what exactly do you mean?  Where does the data come from?  Is it evenly spread across your cluster?

When you say, "ram fills up, the chunks starts building up" - RAM on which machine?

[Merl Corry]

What are you connecting to from your application?  Mongos

When you say, "our cluster loads an unreal amount of data", what exactly do you mean?

We have a 3 node sharded cluster.  Each node has 96GB of Ram. When we first start our data loader, we have seen numbers in the range of 4 - 5 million documents loaded in less than an hour.  Then it slows down to something in the range of 1 - 2 once Mongo starts to cache.

Where does the data come from?  Is it evenly spread across your cluster?

The data comes from our systems reporting their transaction log files to the cluster.  All data comes into the first node?

When you say, "ram fills up, the chunks starts building up" - RAM on which machine?

RAM on the Primary Shard (The 2nd Node).  The chunks will begin to become unbalanced and the primary shard will no distribute the chunks.

So in summary:

Node 1: Mongod, Mongos, App which loads the data

Node 2: Mongod, Mongos, Primary Shard

Node 3: Mongod, Mongos, just a Node

Each Node: 96GB Ram, 24 Hyperthreaded Cores,  6 - 300GB SAS Drives 16MB Cache 10K RPM (Not High Performance Drives) Raid-10 EXT-4, 10000 mb Network connections

[Merl Corry]

Any ideas of what I should be looking for as our bottlekneck?

[Rob Moore]

Have you "presplit" and balanced the collection? 

From your description it sounds Like all of the writes are going to the primary shard, chunks are piling up and they never get moved off.  Is that right?

If so then pre-splitting will help get the cluster to a level state and then stay there longer.
   http://docs.mongodb.org/manual/administration/sharding/#sharding-administration-create-chunks

After the splits you will want to "pre-balance" the chunks to avoid issues during runtime.
   http://docs.mongodb.org/manual/administration/sharding/#migrate-chunks

If you are careful about picking split points you can initially make as many chunks as you have shards, move them to different shards and then fracture the chunks into smaller pieces to avoid lots of slow moves. 

Try to estimate the number of chunks you will have in steady state (data size / chunk size) and fracture the cluster to that level, if possible, for best performance.

Rob.