We are running into performance issues with our mongo cluster.
Here is what we currently have running: Four nodes running RHEL5 64bit with 24 CPUs (HT), 96GB RAM, and RAID 10 (6 disks) with 10G nics. Mongo 1.8.2 with no repl sets. 3 config servers and mongos running on each server.
Data: Trying to ingest 1B rows/day. On avg, this is 11574 rows/sec. With our current schema, each doc is ~512 bytes and the indexes add about ~400 bytes. So to make it easy, lets say 1K/doc.
Doing the math, I get an average disk write speed of ~12MB/sec. Not outrageous even if I double that rate across the 4 servers.
What eventually happens is that the primary shard gets hot because the balancer seems to run too slow. I have seen big chunk imbalances where the primary shard has ~120 chunks and the other 3 shards have ~30 chunks. If we stop ingest, the chunks eventually will balance themselves out. But for production, we can't since data is streaming in 24/7.
Our shard key is YYYYMMDD + 6 random digits. The date is the current date from the data.
The ingest code is written in Java and is currently configured to have 8 threads per server for a total of 32 threads for the entire cluster.
As another test, we setup another 4 nodes to see if this would help. Same thing. The culprit seems to be the balancer not moving files off the node fast enough to help distribute the load.
So, are we trying to load too much into mongo? Is there any tuning we can do on the balancer? Should we try bigger chunks?
Your shard key lends itself to pre-splitting your chunks. Then you would just want to move some of your new empty chunks onto the right servers before they receive data. If done right, it makes the balancer much less busy.
> We are running into performance issues with our mongo cluster.
> Here is what we currently have running: > Four nodes running RHEL5 64bit with 24 CPUs (HT), 96GB RAM, and RAID 10 (6 disks) > with 10G nics. > Mongo 1.8.2 with no repl sets. 3 config servers and mongos running on each server.
> Data: > Trying to ingest 1B rows/day. On avg, this is 11574 rows/sec. With our current > schema, each doc > is ~512 bytes and the indexes add about ~400 bytes. So to make it easy, lets say 1K/doc.
> Doing the math, I get an average disk write speed of ~12MB/sec. Not outrageous even > if I double that rate > across the 4 servers.
> What eventually happens is that the primary shard gets hot because the balancer seems > to run too slow. I have > seen big chunk imbalances where the primary shard has ~120 chunks and the other 3 > shards have ~30 chunks. > If we stop ingest, the chunks eventually will balance themselves out. But for > production, we can't since data > is streaming in 24/7.
> Our shard key is YYYYMMDD + 6 random digits. The date is the current date from the data.
> The ingest code is written in Java and is currently configured to have 8 threads per > server for a total > of 32 threads for the entire cluster.
> As another test, we setup another 4 nodes to see if this would help. Same thing. The > culprit seems to be > the balancer not moving files off the node fast enough to help distribute the load.
> So, are we trying to load too much into mongo? Is there any tuning we can do on the > balancer? > Should we try bigger chunks?
> Thoughts?
> -- > You received this message because you are subscribed to the Google Groups > "mongodb-user" group. > To post to this group, send email to mongodb-user@googlegroups.com. > To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
From the point of view of increasing the number of inserts.
have you tried doing a BULK insert.
bulk insert through Java driver is pretty much inserting a
List<OBObject> (list of documents) , rather than inserting Just
DBObject
basically, you have very high streaming rate. You can batch things up
and do a bulk insert.
Now how much you should batch, should depend on you application. But
I would say put a time and size limit.
So batch up to say 50 sec, but if the size reaches 100,000 documents,
then flush it.
On Aug 2, 9:16 am, Nathan Ehresman <nehres...@sentryds.com> wrote:
> Your shard key lends itself to pre-splitting your chunks. Then you would just want
> to move some of your new empty chunks onto the right servers before they receive
> data. If done right, it makes the balancer much less busy.
> > We are running into performance issues with our mongo cluster.
> > Here is what we currently have running:
> > Four nodes running RHEL5 64bit with 24 CPUs (HT), 96GB RAM, and RAID 10 (6 disks)
> > with 10G nics.
> > Mongo 1.8.2 with no repl sets. 3 config servers and mongos running on each server.
> > Data:
> > Trying to ingest 1B rows/day. On avg, this is 11574 rows/sec. With our current
> > schema, each doc
> > is ~512 bytes and the indexes add about ~400 bytes. So to make it easy, lets say 1K/doc.
> > Doing the math, I get an average disk write speed of ~12MB/sec. Not outrageous even
> > if I double that rate
> > across the 4 servers.
> > What eventually happens is that the primary shard gets hot because the balancer seems
> > to run too slow. I have
> > seen big chunk imbalances where the primary shard has ~120 chunks and the other 3
> > shards have ~30 chunks.
> > If we stop ingest, the chunks eventually will balance themselves out. But for
> > production, we can't since data
> > is streaming in 24/7.
> > Our shard key is YYYYMMDD + 6 random digits. The date is the current date from the data.
> > The ingest code is written in Java and is currently configured to have 8 threads per
> > server for a total
> > of 32 threads for the entire cluster.
> > As another test, we setup another 4 nodes to see if this would help. Same thing. The
> > culprit seems to be
> > the balancer not moving files off the node fast enough to help distribute the load.
> > So, are we trying to load too much into mongo? Is there any tuning we can do on the
> > balancer?
> > Should we try bigger chunks?
> > Thoughts?
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user@googlegroups.com.
> > To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
> > For more options, visit this group athttp://groups.google.com/group/mongodb-user?hl=en.
@nathan - Yeah, that is my next step. I didn't want to have to pre-
split. More code to maintain. :-(
@vermoid - We are using bulk inserts but it maxes out at 5K docs.
Once I get this pre-split going, we'll start fiddling with increasing
that number.
10gen - I assume the balancer is throttled? I have created the 300
chunks and the balancer is still moving chunks around after 20mins.
Is there anyway to get the balancer to run more aggressively? With
10G nics and 64MB chunks, this should be working faster. Would moving
the chunks myself speed things up?
I forgot to mention what our performance problem is. Our ingest can't
keep up. We get hopelessly backlogged.
Thanks all.
On Aug 2, 1:00 pm, vermoid <tdhaayushve...@gmail.com> wrote:
> From the point of view of increasing the number of inserts.
> have you tried doing a BULK insert.
> bulk insert through Java driver is pretty much inserting a
> List<OBObject> (list of documents) , rather than inserting Just
> DBObject
> basically, you have very high streaming rate. You can batch things up
> and do a bulk insert.
> Now how much you should batch, should depend on you application. But
> I would say put a time and size limit.
> So batch up to say 50 sec, but if the size reaches 100,000 documents,
> then flush it.
> On Aug 2, 9:16 am, Nathan Ehresman <nehres...@sentryds.com> wrote:
> > Otis,
> > Your shard key lends itself to pre-splitting your chunks. Then you would just want
> > to move some of your new empty chunks onto the right servers before they receive
> > data. If done right, it makes the balancer much less busy.
> > > We are running into performance issues with our mongo cluster.
> > > Here is what we currently have running:
> > > Four nodes running RHEL5 64bit with 24 CPUs (HT), 96GB RAM, and RAID 10 (6 disks)
> > > with 10G nics.
> > > Mongo 1.8.2 with no repl sets. 3 config servers and mongos running on each server.
> > > Data:
> > > Trying to ingest 1B rows/day. On avg, this is 11574 rows/sec. With our current
> > > schema, each doc
> > > is ~512 bytes and the indexes add about ~400 bytes. So to make it easy, lets say 1K/doc.
> > > Doing the math, I get an average disk write speed of ~12MB/sec. Not outrageous even
> > > if I double that rate
> > > across the 4 servers.
> > > What eventually happens is that the primary shard gets hot because the balancer seems
> > > to run too slow. I have
> > > seen big chunk imbalances where the primary shard has ~120 chunks and the other 3
> > > shards have ~30 chunks.
> > > If we stop ingest, the chunks eventually will balance themselves out. But for
> > > production, we can't since data
> > > is streaming in 24/7.
> > > Our shard key is YYYYMMDD + 6 random digits. The date is the current date from the data.
> > > The ingest code is written in Java and is currently configured to have 8 threads per
> > > server for a total
> > > of 32 threads for the entire cluster.
> > > As another test, we setup another 4 nodes to see if this would help. Same thing. The
> > > culprit seems to be
> > > the balancer not moving files off the node fast enough to help distribute the load.
> > > So, are we trying to load too much into mongo? Is there any tuning we can do on the
> > > balancer?
> > > Should we try bigger chunks?
> > > Thoughts?
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "mongodb-user" group.
> > > To post to this group, send email to mongodb-user@googlegroups.com.
> > > To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
> > > For more options, visit this group athttp://groups.google.com/group/mongodb-user?hl=en.
The balancer isn't throttled, but it needs to perform verification steps and continue to push new and updated data to the target shard, which can slow things down when you continue to add lots of data. The mongod logs have info about migration data transfers, if you want to know more about what's happening on your systems.
As Nathan mentioned, pre-splitting is probably the right way to go here - it's more code, but hopefully you should only need a small script that runs every few days. We are looking at tracking not only the current size but rate of size increase to make the balancer more predictive, but for now this may be your best solution.
On Wed, Aug 3, 2011 at 10:00 AM, otisz...@yahoo.com <otisz...@yahoo.com> wrote: > @nathan - Yeah, that is my next step. I didn't want to have to pre- > split. More code to maintain. :-(
> @vermoid - We are using bulk inserts but it maxes out at 5K docs. > Once I get this pre-split going, we'll start fiddling with increasing > that number.
> 10gen - I assume the balancer is throttled? I have created the 300 > chunks and the balancer is still moving chunks around after 20mins. > Is there anyway to get the balancer to run more aggressively? With > 10G nics and 64MB chunks, this should be working faster. Would moving > the chunks myself speed things up?
> I forgot to mention what our performance problem is. Our ingest can't > keep up. We get hopelessly backlogged.
> Thanks all.
> On Aug 2, 1:00 pm, vermoid <tdhaayushve...@gmail.com> wrote: >> From the point of view of increasing the number of inserts. >> have you tried doing a BULK insert. >> bulk insert through Java driver is pretty much inserting a >> List<OBObject> (list of documents) , rather than inserting Just >> DBObject
>> basically, you have very high streaming rate. You can batch things up >> and do a bulk insert. >> Now how much you should batch, should depend on you application. But >> I would say put a time and size limit. >> So batch up to say 50 sec, but if the size reaches 100,000 documents, >> then flush it.
>> On Aug 2, 9:16 am, Nathan Ehresman <nehres...@sentryds.com> wrote:
>> > Otis,
>> > Your shard key lends itself to pre-splitting your chunks. Then you would just want >> > to move some of your new empty chunks onto the right servers before they receive >> > data. If done right, it makes the balancer much less busy.
>> > > We are running into performance issues with our mongo cluster.
>> > > Here is what we currently have running: >> > > Four nodes running RHEL5 64bit with 24 CPUs (HT), 96GB RAM, and RAID 10 (6 disks) >> > > with 10G nics. >> > > Mongo 1.8.2 with no repl sets. 3 config servers and mongos running on each server.
>> > > Data: >> > > Trying to ingest 1B rows/day. On avg, this is 11574 rows/sec. With our current >> > > schema, each doc >> > > is ~512 bytes and the indexes add about ~400 bytes. So to make it easy, lets say 1K/doc.
>> > > Doing the math, I get an average disk write speed of ~12MB/sec. Not outrageous even >> > > if I double that rate >> > > across the 4 servers.
>> > > What eventually happens is that the primary shard gets hot because the balancer seems >> > > to run too slow. I have >> > > seen big chunk imbalances where the primary shard has ~120 chunks and the other 3 >> > > shards have ~30 chunks. >> > > If we stop ingest, the chunks eventually will balance themselves out. But for >> > > production, we can't since data >> > > is streaming in 24/7.
>> > > Our shard key is YYYYMMDD + 6 random digits. The date is the current date from the data.
>> > > The ingest code is written in Java and is currently configured to have 8 threads per >> > > server for a total >> > > of 32 threads for the entire cluster.
>> > > As another test, we setup another 4 nodes to see if this would help. Same thing. The >> > > culprit seems to be >> > > the balancer not moving files off the node fast enough to help distribute the load.
>> > > So, are we trying to load too much into mongo? Is there any tuning we can do on the >> > > balancer? >> > > Should we try bigger chunks?
>> > > Thoughts?
>> > > -- >> > > You received this message because you are subscribed to the Google Groups >> > > "mongodb-user" group. >> > > To post to this group, send email to mongodb-user@googlegroups.com. >> > > To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com. >> > > For more options, visit this group athttp://groups.google.com/group/mongodb-user?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "mongodb-user" group. > To post to this group, send email to mongodb-user@googlegroups.com. > To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Thanks for the info. It helps to understand how things work. I
started running a test using pre-splits and its working better.
Would running moveChunk myself be faster then waiting for the balancer
to do it? I'm planning to run the presplitter before midnight in
order to create the next day's chunks. My first pre-split took the
balancer 30mins to move the chunks around. I assume there is no way
to create a chunk directly on a shard.
Also, any insight into whether shoving 1B docs/day into mongo is
sane? I have searched around a bit and haven't found any other users
with that much data.
On Aug 3, 1:09 pm, Greg Studer <g...@10gen.com> wrote:
> The balancer isn't throttled, but it needs to perform verification
> steps and continue to push new and updated data to the target shard,
> which can slow things down when you continue to add lots of data. The
> mongod logs have info about migration data transfers, if you want to
> know more about what's happening on your systems.
> As Nathan mentioned, pre-splitting is probably the right way to go
> here - it's more code, but hopefully you should only need a small
> script that runs every few days. We are looking at tracking not only
> the current size but rate of size increase to make the balancer more
> predictive, but for now this may be your best solution.
> On Wed, Aug 3, 2011 at 10:00 AM, otisz...@yahoo.com <otisz...@yahoo.com> wrote:
> > @nathan - Yeah, that is my next step. I didn't want to have to pre-
> > split. More code to maintain. :-(
> > @vermoid - We are using bulk inserts but it maxes out at 5K docs.
> > Once I get this pre-split going, we'll start fiddling with increasing
> > that number.
> > 10gen - I assume the balancer is throttled? I have created the 300
> > chunks and the balancer is still moving chunks around after 20mins.
> > Is there anyway to get the balancer to run more aggressively? With
> > 10G nics and 64MB chunks, this should be working faster. Would moving
> > the chunks myself speed things up?
> > I forgot to mention what our performance problem is. Our ingest can't
> > keep up. We get hopelessly backlogged.
> > Thanks all.
> > On Aug 2, 1:00 pm, vermoid <tdhaayushve...@gmail.com> wrote:
> >> From the point of view of increasing the number of inserts.
> >> have you tried doing a BULK insert.
> >> bulk insert through Java driver is pretty much inserting a
> >> List<OBObject> (list of documents) , rather than inserting Just
> >> DBObject
> >> basically, you have very high streaming rate. You can batch things up
> >> and do a bulk insert.
> >> Now how much you should batch, should depend on you application. But
> >> I would say put a time and size limit.
> >> So batch up to say 50 sec, but if the size reaches 100,000 documents,
> >> then flush it.
> >> On Aug 2, 9:16 am, Nathan Ehresman <nehres...@sentryds.com> wrote:
> >> > Otis,
> >> > Your shard key lends itself to pre-splitting your chunks. Then you would just want
> >> > to move some of your new empty chunks onto the right servers before they receive
> >> > data. If done right, it makes the balancer much less busy.
> >> > > We are running into performance issues with our mongo cluster.
> >> > > Here is what we currently have running:
> >> > > Four nodes running RHEL5 64bit with 24 CPUs (HT), 96GB RAM, and RAID 10 (6 disks)
> >> > > with 10G nics.
> >> > > Mongo 1.8.2 with no repl sets. 3 config servers and mongos running on each server.
> >> > > Data:
> >> > > Trying to ingest 1B rows/day. On avg, this is 11574 rows/sec. With our current
> >> > > schema, each doc
> >> > > is ~512 bytes and the indexes add about ~400 bytes. So to make it easy, lets say 1K/doc.
> >> > > Doing the math, I get an average disk write speed of ~12MB/sec. Not outrageous even
> >> > > if I double that rate
> >> > > across the 4 servers.
> >> > > What eventually happens is that the primary shard gets hot because the balancer seems
> >> > > to run too slow. I have
> >> > > seen big chunk imbalances where the primary shard has ~120 chunks and the other 3
> >> > > shards have ~30 chunks.
> >> > > If we stop ingest, the chunks eventually will balance themselves out. But for
> >> > > production, we can't since data
> >> > > is streaming in 24/7.
> >> > > Our shard key is YYYYMMDD + 6 random digits. The date is the current date from the data.
> >> > > The ingest code is written in Java and is currently configured to have 8 threads per
> >> > > server for a total
> >> > > of 32 threads for the entire cluster.
> >> > > As another test, we setup another 4 nodes to see if this would help. Same thing. The
> >> > > culprit seems to be
> >> > > the balancer not moving files off the node fast enough to help distribute the load.
> >> > > So, are we trying to load too much into mongo? Is there any tuning we can do on the
> >> > > balancer?
> >> > > Should we try bigger chunks?
> >> > > Thoughts?
> >> > > --
> >> > > You received this message because you are subscribed to the Google Groups
> >> > > "mongodb-user" group.
> >> > > To post to this group, send email to mongodb-user@googlegroups.com.
> >> > > To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
> >> > > For more options, visit this group athttp://groups.google.com/group/mongodb-user?hl=en.
> > --
> > You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> > To post to this group, send email to mongodb-user@googlegroups.com.
> > To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.
> > For more options, visit this group athttp://groups.google.com/group/mongodb-user?hl=en.
Moving yourself might be faster as you won't have any balancer pauses to make sure things are ok. If you know they are going to be empty chunks, might be better.
1B docs/day is fine, planning the cluster size should just factor in working set size and load.
On Wed, Aug 3, 2011 at 5:38 PM, otisz...@yahoo.com <otisz...@yahoo.com>wrote:
> Thanks for the info. It helps to understand how things work. I > started running a test using pre-splits and its working better.
> Would running moveChunk myself be faster then waiting for the balancer > to do it? I'm planning to run the presplitter before midnight in > order to create the next day's chunks. My first pre-split took the > balancer 30mins to move the chunks around. I assume there is no way > to create a chunk directly on a shard.
> Also, any insight into whether shoving 1B docs/day into mongo is > sane? I have searched around a bit and haven't found any other users > with that much data.
> On Aug 3, 1:09 pm, Greg Studer <g...@10gen.com> wrote: > > The balancer isn't throttled, but it needs to perform verification > > steps and continue to push new and updated data to the target shard, > > which can slow things down when you continue to add lots of data. The > > mongod logs have info about migration data transfers, if you want to > > know more about what's happening on your systems.
> > As Nathan mentioned, pre-splitting is probably the right way to go > > here - it's more code, but hopefully you should only need a small > > script that runs every few days. We are looking at tracking not only > > the current size but rate of size increase to make the balancer more > > predictive, but for now this may be your best solution.
> > On Wed, Aug 3, 2011 at 10:00 AM, otisz...@yahoo.com <otisz...@yahoo.com> > wrote: > > > @nathan - Yeah, that is my next step. I didn't want to have to pre- > > > split. More code to maintain. :-(
> > > @vermoid - We are using bulk inserts but it maxes out at 5K docs. > > > Once I get this pre-split going, we'll start fiddling with increasing > > > that number.
> > > 10gen - I assume the balancer is throttled? I have created the 300 > > > chunks and the balancer is still moving chunks around after 20mins. > > > Is there anyway to get the balancer to run more aggressively? With > > > 10G nics and 64MB chunks, this should be working faster. Would moving > > > the chunks myself speed things up?
> > > I forgot to mention what our performance problem is. Our ingest can't > > > keep up. We get hopelessly backlogged.
> > > Thanks all.
> > > On Aug 2, 1:00 pm, vermoid <tdhaayushve...@gmail.com> wrote: > > >> From the point of view of increasing the number of inserts. > > >> have you tried doing a BULK insert. > > >> bulk insert through Java driver is pretty much inserting a > > >> List<OBObject> (list of documents) , rather than inserting Just > > >> DBObject
> > >> basically, you have very high streaming rate. You can batch things up > > >> and do a bulk insert. > > >> Now how much you should batch, should depend on you application. But > > >> I would say put a time and size limit. > > >> So batch up to say 50 sec, but if the size reaches 100,000 documents, > > >> then flush it.
> > >> On Aug 2, 9:16 am, Nathan Ehresman <nehres...@sentryds.com> wrote:
> > >> > Otis,
> > >> > Your shard key lends itself to pre-splitting your chunks. Then you > would just want > > >> > to move some of your new empty chunks onto the right servers before > they receive > > >> > data. If done right, it makes the balancer much less busy.
> > >> > > We are running into performance issues with our mongo cluster.
> > >> > > Here is what we currently have running: > > >> > > Four nodes running RHEL5 64bit with 24 CPUs (HT), 96GB RAM, and > RAID 10 (6 disks) > > >> > > with 10G nics. > > >> > > Mongo 1.8.2 with no repl sets. 3 config servers and mongos running > on each server.
> > >> > > Data: > > >> > > Trying to ingest 1B rows/day. On avg, this is 11574 rows/sec. With > our current > > >> > > schema, each doc > > >> > > is ~512 bytes and the indexes add about ~400 bytes. So to make it > easy, lets say 1K/doc.
> > >> > > Doing the math, I get an average disk write speed of ~12MB/sec. > Not outrageous even > > >> > > if I double that rate > > >> > > across the 4 servers.
> > >> > > What eventually happens is that the primary shard gets hot because > the balancer seems > > >> > > to run too slow. I have > > >> > > seen big chunk imbalances where the primary shard has ~120 chunks > and the other 3 > > >> > > shards have ~30 chunks. > > >> > > If we stop ingest, the chunks eventually will balance themselves > out. But for > > >> > > production, we can't since data > > >> > > is streaming in 24/7.
> > >> > > Our shard key is YYYYMMDD + 6 random digits. The date is the > current date from the data.
> > >> > > The ingest code is written in Java and is currently configured to > have 8 threads per > > >> > > server for a total > > >> > > of 32 threads for the entire cluster.
> > >> > > As another test, we setup another 4 nodes to see if this would > help. Same thing. The > > >> > > culprit seems to be > > >> > > the balancer not moving files off the node fast enough to help > distribute the load.
> > >> > > So, are we trying to load too much into mongo? Is there any tuning > we can do on the > > >> > > balancer? > > >> > > Should we try bigger chunks?
> > >> > > Thoughts?
> > >> > > -- > > >> > > You received this message because you are subscribed to the Google > Groups > > >> > > "mongodb-user" group. > > >> > > To post to this group, send email to > mongodb-user@googlegroups.com. > > >> > > To unsubscribe from this group, send email to > mongodb-user+unsubscribe@googlegroups.com. > > >> > > For more options, visit this group athttp:// > groups.google.com/group/mongodb-user?hl=en.
> > > -- > > > You received this message because you are subscribed to the Google > Groups "mongodb-user" group. > > > To post to this group, send email to mongodb-user@googlegroups.com. > > > To unsubscribe from this group, send email to > mongodb-user+unsubscribe@googlegroups.com. > > > For more options, visit this group athttp:// > groups.google.com/group/mongodb-user?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "mongodb-user" group. > To post to this group, send email to mongodb-user@googlegroups.com. > To unsubscribe from this group, send email to > mongodb-user+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/mongodb-user?hl=en.
