I am curious. Is morphia still recommended to be used with Mongo? We are having major sharding issues with Mongo. At first it was believed that our issue was with our sharding key. We changed the key to something more random, same problem. We have upgraded to every new version of Mongo and still get the same result. Our cluster loads an unreal amount of data until ram fills up. Once ram fills up, the chunks starts building up on one shard and the cluster becomes unbalanced. Then eventually the loading of data backlogs and the cluster becomes completely unresponsive. The only thing that has not changed in our entire process is morphia. We have used morphia from day one to load data into the system. Should we be using something else?
On Thursday, October 11, 2012 9:31:41 AM UTC-4, Merl wrote:
> I am curious. Is morphia still recommended to be used with Mongo? We are > having major sharding issues with Mongo. At first it was believed that our > issue was with our sharding key. We changed the key to something more > random, same problem. We have upgraded to every new version of Mongo and > still get the same result. Our cluster loads an unreal amount of data > until ram fills up. Once ram fills up, the chunks starts building up on > one shard and the cluster becomes unbalanced. Then eventually the loading > of data backlogs and the cluster becomes completely unresponsive. The only > thing that has not changed in our entire process is morphia. We have used > morphia from day one to load data into the system. Should we be using > something else?
On Thu, Oct 11, 2012 at 10:27 AM, Sam Helman <sam.hel...@10gen.com> wrote:
> I do not know of any issues with Morphia, what is your current shard key?
> On Thursday, October 11, 2012 9:31:41 AM UTC-4, Merl wrote:
>> I am curious. Is morphia still recommended to be used with Mongo? We
>> are having major sharding issues with Mongo. At first it was believed that
>> our issue was with our sharding key. We changed the key to something more
>> random, same problem. We have upgraded to every new version of Mongo and
>> still get the same result. Our cluster loads an unreal amount of data
>> until ram fills up. Once ram fills up, the chunks starts building up on
>> one shard and the cluster becomes unbalanced. Then eventually the loading
>> of data backlogs and the cluster becomes completely unresponsive. The only
>> thing that has not changed in our entire process is morphia. We have used
>> morphia from day one to load data into the system. Should we be using
>> something else?
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
What are you connecting to from your application? Is it a mongos, or is it a direct connection to a mongod? When you say, "our cluster loads an unreal amount of data", what exactly do you mean? Where does the data come from? Is it evenly spread across your cluster? When you say, "ram fills up, the chunks starts building up" - RAM on which machine?
On Thursday, October 11, 2012 10:42:34 AM UTC-4, Merl wrote:
> It is 2 random 8 character strings as a compound as a compound shard key.
> On Thu, Oct 11, 2012 at 10:27 AM, Sam Helman <sam.h...@10gen.com<javascript:> > > wrote:
>> I do not know of any issues with Morphia, what is your current shard key?
>> On Thursday, October 11, 2012 9:31:41 AM UTC-4, Merl wrote:
>>> I am curious. Is morphia still recommended to be used with Mongo? We >>> are having major sharding issues with Mongo. At first it was believed that >>> our issue was with our sharding key. We changed the key to something more >>> random, same problem. We have upgraded to every new version of Mongo and >>> still get the same result. Our cluster loads an unreal amount of data >>> until ram fills up. Once ram fills up, the chunks starts building up on >>> one shard and the cluster becomes unbalanced. Then eventually the loading >>> of data backlogs and the cluster becomes completely unresponsive. The only >>> thing that has not changed in our entire process is morphia. We have used >>> morphia from day one to load data into the system. Should we be using >>> something else?
>> -- >> You received this message because you are subscribed to the Google >> Groups "mongodb-user" group. >> To post to this group, send email to mongod...@googlegroups.com<javascript:> >> To unsubscribe from this group, send email to >> mongodb-user...@googlegroups.com <javascript:> >> See also the IRC channel -- freenode.net#mongodb
What are you connecting to from your application? Mongos
When you say, "our cluster loads an unreal amount of data", what exactly do
you mean?
We have a 3 node sharded cluster. Each node has 96GB of Ram. When we first
start our data loader, we have seen numbers in the range of 4 - 5 million
documents loaded in less than an hour. Then it slows down to something in
the range of 1 - 2 once Mongo starts to cache.
Where does the data come from? Is it evenly spread across your cluster?
The data comes from our systems reporting their transaction log files to
the cluster. All data comes into the first node?
When you say, "ram fills up, the chunks starts building up" - RAM on which
machine?
RAM on the Primary Shard (The 2nd Node). The chunks will begin to become
unbalanced and the primary shard will no distribute the chunks.
So in summary:
Node 1: Mongod, Mongos, App which loads the data
Node 2: Mongod, Mongos, Primary Shard
Node 3: Mongod, Mongos, just a Node
Each Node: 96GB Ram, 24 Hyperthreaded Cores, 6 - 300GB SAS Drives 16MB
Cache 10K RPM (Not High Performance Drives) Raid-10 EXT-4, 10000 mb Network
connections
On Thu, Oct 11, 2012 at 1:25 PM, Sam Helman <sam.hel...@10gen.com> wrote:
> To isolate what might be causing your issues:
> What are you connecting to from your application? Is it a mongos, or is
> it a direct connection to a mongod?
> When you say, "our cluster loads an unreal amount of data", what exactly
> do you mean? Where does the data come from? Is it evenly spread across
> your cluster?
> When you say, "ram fills up, the chunks starts building up" - RAM on which
> machine?
> On Thursday, October 11, 2012 10:42:34 AM UTC-4, Merl wrote:
>> It is 2 random 8 character strings as a compound as a compound shard key.
>> On Thu, Oct 11, 2012 at 10:27 AM, Sam Helman <sam.h...@10gen.com> wrote:
>>> I do not know of any issues with Morphia, what is your current shard key?
>>> On Thursday, October 11, 2012 9:31:41 AM UTC-4, Merl wrote:
>>>> I am curious. Is morphia still recommended to be used with Mongo? We
>>>> are having major sharding issues with Mongo. At first it was believed that
>>>> our issue was with our sharding key. We changed the key to something more
>>>> random, same problem. We have upgraded to every new version of Mongo and
>>>> still get the same result. Our cluster loads an unreal amount of data
>>>> until ram fills up. Once ram fills up, the chunks starts building up on
>>>> one shard and the cluster becomes unbalanced. Then eventually the loading
>>>> of data backlogs and the cluster becomes completely unresponsive. The only
>>>> thing that has not changed in our entire process is morphia. We have used
>>>> morphia from day one to load data into the system. Should we be using
>>>> something else?
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "mongodb-user" group.
>>> To post to this group, send email to mongod...@googlegroups.com
>>> To unsubscribe from this group, send email to
>>> mongodb-user...@**googlegroups.com
>>> See also the IRC channel -- freenode.net#mongodb
>> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
On Thu, Oct 11, 2012 at 1:52 PM, Merl Corry <merlco...@gmail.com> wrote:
> What are you connecting to from your application? Mongos
> When you say, "our cluster loads an unreal amount of data", what exactly
> do you mean?
> We have a 3 node sharded cluster. Each node has 96GB of Ram. When we
> first start our data loader, we have seen numbers in the range of 4 - 5
> million documents loaded in less than an hour. Then it slows down to
> something in the range of 1 - 2 once Mongo starts to cache.
> Where does the data come from? Is it evenly spread across your cluster?
> The data comes from our systems reporting their transaction log files to
> the cluster. All data comes into the first node?
> When you say, "ram fills up, the chunks starts building up" - RAM on which
> machine?
> RAM on the Primary Shard (The 2nd Node). The chunks will begin to become
> unbalanced and the primary shard will no distribute the chunks.
> So in summary:
> Node 1: Mongod, Mongos, App which loads the data
> Node 2: Mongod, Mongos, Primary Shard
> Node 3: Mongod, Mongos, just a Node
> Each Node: 96GB Ram, 24 Hyperthreaded Cores, 6 - 300GB SAS Drives 16MB
> Cache 10K RPM (Not High Performance Drives) Raid-10 EXT-4, 10000 mb Network
> connections
> On Thu, Oct 11, 2012 at 1:25 PM, Sam Helman <sam.hel...@10gen.com> wrote:
>> To isolate what might be causing your issues:
>> What are you connecting to from your application? Is it a mongos, or is
>> it a direct connection to a mongod?
>> When you say, "our cluster loads an unreal amount of data", what exactly
>> do you mean? Where does the data come from? Is it evenly spread across
>> your cluster?
>> When you say, "ram fills up, the chunks starts building up" - RAM on
>> which machine?
>> On Thursday, October 11, 2012 10:42:34 AM UTC-4, Merl wrote:
>>> It is 2 random 8 character strings as a compound as a compound shard
>>> key.
>>> On Thu, Oct 11, 2012 at 10:27 AM, Sam Helman <sam.h...@10gen.com> wrote:
>>>> I do not know of any issues with Morphia, what is your current shard
>>>> key?
>>>> On Thursday, October 11, 2012 9:31:41 AM UTC-4, Merl wrote:
>>>>> I am curious. Is morphia still recommended to be used with Mongo? We
>>>>> are having major sharding issues with Mongo. At first it was believed that
>>>>> our issue was with our sharding key. We changed the key to something more
>>>>> random, same problem. We have upgraded to every new version of Mongo and
>>>>> still get the same result. Our cluster loads an unreal amount of data
>>>>> until ram fills up. Once ram fills up, the chunks starts building up on
>>>>> one shard and the cluster becomes unbalanced. Then eventually the loading
>>>>> of data backlogs and the cluster becomes completely unresponsive. The only
>>>>> thing that has not changed in our entire process is morphia. We have used
>>>>> morphia from day one to load data into the system. Should we be using
>>>>> something else?
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "mongodb-user" group.
>>>> To post to this group, send email to mongod...@googlegroups.com
>>>> To unsubscribe from this group, send email to
>>>> mongodb-user...@**googlegroups.com
>>>> See also the IRC channel -- freenode.net#mongodb
>>> --
>> You received this message because you are subscribed to the Google
>> Groups "mongodb-user" group.
>> To post to this group, send email to mongodb-user@googlegroups.com
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe@googlegroups.com
>> See also the IRC channel -- freenode.net#mongodb
From your description it sounds Like all of the writes are going to the primary shard, chunks are piling up and they never get moved off. Is that right?
If so then pre-splitting will help get the cluster to a level state and then stay there longer.
If you are careful about picking split points you can initially make as many chunks as you have shards, move them to different shards and then fracture the chunks into smaller pieces to avoid lots of slow moves.
Try to estimate the number of chunks you will have in steady state (data size / chunk size) and fracture the cluster to that level, if possible, for best performance.
On Friday, October 12, 2012 2:01:09 PM UTC-4, Merl wrote:
> Any ideas of what I should be looking for as our bottlekneck?
> On Thu, Oct 11, 2012 at 1:52 PM, Merl Corry <merl...@gmail.com<javascript:> > > wrote:
>> What are you connecting to from your application? Mongos
>> When you say, "our cluster loads an unreal amount of data", what exactly >> do you mean? >> We have a 3 node sharded cluster. Each node has 96GB of Ram. When we >> first start our data loader, we have seen numbers in the range of 4 - 5 >> million documents loaded in less than an hour. Then it slows down to >> something in the range of 1 - 2 once Mongo starts to cache.
>> Where does the data come from? Is it evenly spread across your cluster? >> The data comes from our systems reporting their transaction log files to >> the cluster. All data comes into the first node?
>> When you say, "ram fills up, the chunks starts building up" - RAM on >> which machine? >> RAM on the Primary Shard (The 2nd Node). The chunks will begin to become >> unbalanced and the primary shard will no distribute the chunks.
>> So in summary:
>> Node 1: Mongod, Mongos, App which loads the data >> Node 2: Mongod, Mongos, Primary Shard >> Node 3: Mongod, Mongos, just a Node
>> Each Node: 96GB Ram, 24 Hyperthreaded Cores, 6 - 300GB SAS Drives 16MB >> Cache 10K RPM (Not High Performance Drives) Raid-10 EXT-4, 10000 mb Network >> connections
>> On Thu, Oct 11, 2012 at 1:25 PM, Sam Helman <sam.h...@10gen.com<javascript:> >> > wrote:
>>> To isolate what might be causing your issues:
>>> What are you connecting to from your application? Is it a mongos, or is >>> it a direct connection to a mongod? >>> When you say, "our cluster loads an unreal amount of data", what exactly >>> do you mean? Where does the data come from? Is it evenly spread across >>> your cluster? >>> When you say, "ram fills up, the chunks starts building up" - RAM on >>> which machine?
>>> On Thursday, October 11, 2012 10:42:34 AM UTC-4, Merl wrote:
>>>> It is 2 random 8 character strings as a compound as a compound shard >>>> key.
>>>> On Thu, Oct 11, 2012 at 10:27 AM, Sam Helman <sam.h...@10gen.com>wrote:
>>>>> I do not know of any issues with Morphia, what is your current shard >>>>> key?
>>>>> On Thursday, October 11, 2012 9:31:41 AM UTC-4, Merl wrote:
>>>>>> I am curious. Is morphia still recommended to be used with Mongo? >>>>>> We are having major sharding issues with Mongo. At first it >>>>>> was believed that our issue was with our sharding key. We changed the key >>>>>> to something more random, same problem. We have upgraded to every new >>>>>> version of Mongo and still get the same result. Our cluster loads an >>>>>> unreal amount of data until ram fills up. Once ram fills up, the chunks >>>>>> starts building up on one shard and the cluster becomes unbalanced. Then >>>>>> eventually the loading of data backlogs and the cluster becomes completely >>>>>> unresponsive. The only thing that has not changed in our entire process is >>>>>> morphia. We have used morphia from day one to load data into the system. >>>>>> Should we be using something else?
>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "mongodb-user" group. >>>>> To post to this group, send email to mongod...@googlegroups.com
>>>>> To unsubscribe from this group, send email to >>>>> mongodb-user...@**googlegroups.com
>>>>> See also the IRC channel -- freenode.net#mongodb
>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "mongodb-user" group. >>> To post to this group, send email to mongod...@googlegroups.com<javascript:> >>> To unsubscribe from this group, send email to >>> mongodb-user...@googlegroups.com <javascript:> >>> See also the IRC channel -- freenode.net#mongodb
