Mongo sharding deadlock

[Prasanna .S]

Hey guys,

We sharded mongo as per the docs. The issue is the balancer is not being very clever and not throttling at all. Now we are stuck with a half-sharded setup and we can't turn on the balancer because it'd bring the site down. All read queries timeout when the disk write rate is 100%.

Is there a way out of this mess at all? From a high-level, mongo could have handled this by any of the following means:

1. Replicate from the slaves. Mysql does this.

2. Throttle and replicate only as long as the disk util is < 50% or so.

3. Have the moveChunk interruptible. If reads and writes come in-between, consider them as higher priority, so that a read/write is never waiting for more than a second.

Have you guys heard of this problem before? Because this really make sharding impossible. I'm not sure what others are doing in production to work around this.

Thanks,

-Prasanna

Co-Founder & CTO, lal.com

[Nat]

I don't think it goes into deadlock. Can you try disabling the balancer and see whether the disk write rate going down or not.

---

From: "Prasanna .S" <mypra...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Mon, 7 May 2012 23:22:11 -0700

To: <mongod...@googlegroups.com>

ReplyTo: mongod...@googlegroups.com

Cc: Surendran Mahendran<suren...@gmail.com>; Hrushikesh Tilak<hrushike...@gmail.com>

Subject: [mongodb-user] Mongo sharding deadlock

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Prasanna .S]

More info:

- Our chunk size is 64MB.

- It takes ~4 minutes, for a single moveChunk to complete. Is that weird or expected?

- Within the 4 minutes, the site goes down, because the reads timeout.

- We are running a EBS disk.

I can try to make the chunks smaller if that's the issue?

[Nat]

It shouldn't take that long. Can you show your iostat, mongostat when the problem is happening? Also what EBS setup do you use? Did you follow the production notes at http://www.mongodb.org/display/DOCS/Amazon+EC2#AmazonEC2-StorageConfiguration?

[Prasanna]

Thanks a bunch for the response Nat.

Yes we follow the EC2 Storage configuration guide.

Here's the iostat: http://pastebin.com/nB7tbKtb

The mongostat: http://pastebin.com/qz637ViD

And the mongotop: http://pastebin.com/dTnRTMWD

Any suggestions would be much appreciated!

[Prasanna]

And yes, disabling the balancer, brings the write rate back to zero. Although after 4 minutes!

On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:

I don't think it goes into deadlock. Can you try disabling the balancer and see whether the disk write rate going down or not.

---

From: "Prasanna .S" <mypra...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Mon, 7 May 2012 23:22:11 -0700

To: <mongod...@googlegroups.com>

ReplyTo: mongod...@googlegroups.com

Cc: Surendran Mahendran<suren...@gmail.com>; Hrushikesh Tilak<hrushikesh.tilak@gmail.com>

Subject: [mongodb-user] Mongo sharding deadlock

Hey guys,

We sharded mongo as per the docs. The issue is the balancer is not being very clever and not throttling at all. Now we are stuck with a half-sharded setup and we can't turn on the balancer because it'd bring the site down. All read queries timeout when the disk write rate is 100%.

Is there a way out of this mess at all? From a high-level, mongo could have handled this by any of the following means:

1. Replicate from the slaves. Mysql does this.

2. Throttle and replicate only as long as the disk util is < 50% or so.

3. Have the moveChunk interruptible. If reads and writes come in-between, consider them as higher priority, so that a read/write is never waiting for more than a second.

Have you guys heard of this problem before? Because this really make sharding impossible. I'm not sure what others are doing in production to work around this.

Thanks,

-Prasanna

Co-Founder & CTO, lal.com

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.

To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.

[Eliot Horowitz]

What version of mongo and what kind of ebs raid?

On Tue, May 8, 2012 at 4:46 AM, Prasanna <mypra...@gmail.com> wrote:
> And yes, disabling the balancer, brings the write rate back to zero.
> Although after 4 minutes!
>
>
> On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>>
>> I don't think it goes into deadlock. Can you try disabling the balancer
>> and see whether the disk write rate going down or not.
>> ________________________________
>> From: "Prasanna .S" <mypra...@gmail.com>
>> Sender: mongod...@googlegroups.com
>> Date: Mon, 7 May 2012 23:22:11 -0700
>> To: <mongod...@googlegroups.com>
>> ReplyTo: mongod...@googlegroups.com
>> Cc: Surendran Mahendran<suren...@gmail.com>; Hrushikesh

>> Tilak<hrushike...@gmail.com>

>> Subject: [mongodb-user] Mongo sharding deadlock
>>
>> Hey guys,
>>
>> We sharded mongo as per the docs. The issue is the balancer is not being
>> very clever and not throttling at all. Now we are stuck with a half-sharded
>> setup and we can't turn on the balancer because it'd bring the site down.
>> All read queries timeout when the disk write rate is 100%.
>>
>> Is there a way out of this mess at all? From a high-level, mongo could
>> have handled this by any of the following means:
>>
>> 1. Replicate from the slaves. Mysql does this.
>> 2. Throttle and replicate only as long as the disk util is < 50% or so.
>> 3. Have the moveChunk interruptible. If reads and writes come in-between,
>> consider them as higher priority, so that a read/write is never waiting for
>> more than a second.
>>
>> Have you guys heard of this problem before? Because this really make
>> sharding impossible. I'm not sure what others are doing in production to
>> work around this.
>>
>> Thanks,
>> -Prasanna
>> Co-Founder & CTO, lal.com
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongod...@googlegroups.com.
>> To unsubscribe from this group, send email to

>> mongodb-user...@googlegroups.com.

>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.

>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

> mongodb-user...@googlegroups.com.

[Prasanna .S]

Raid 10 on 8 disks.

MongoDB shell version: 2.0.4

Tue May  8 09:02:29 [initandlisten] db version v2.0.4, pdfile version 4.5

[Eliot Horowitz]

Are these machines in MMS?
Hard to see much in the pasted stats.

[Prasanna .S]

Yes, indeed.

Username: arn...@lal.com

Machine: mongo55

[Dan Pasette]

Looks like the docs in that collection (lal_production.facebook_names) are small and likely distributed throughout the disk.  

Can you check your disk readahead setting?  

Please run and post the results of blockev --report.

[Prasanna .S]

Hey Dan,

Read ahead settings are the defaults (256 MB): http://pastebin.com/MHc8s2fq

Let me know if you need something else. 

@Elliot,

I spoke with some of my friends at foursquare and it turns out they face the same issue as well. I've had my other friends who run companies > $100M market cap (whom I'd rather not name in this list), tell me about their mongo problems in the past few days. This is pretty serious stuff and will affect if someone wants to choose mongo as their primary store or not. To not have sharding is one thing, but to promise it and being stuck in production is another.

I really thank you guys and appreciate the responsiveness so far. Happy to work together and hope we can resolve this.

Thanks so much,

-Prasanna

[Scott Hernandez]

Those values are in blocks (512 bytes), not MB.

We have an update in the upcoming 2.0.6 stable release which will help
with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303 You
can click on the version like to see other important fixes.

Can you do a few migrates tonight so we can see the stats/behavior in
MMS? Please let us know when you do this so we can take a look at the
data to see if this fix will help.

[Prasanna .S]

Hey Scott,

I'm going to do this now. Will start it and stop it quickly and give you the exact times.

Thanks,

-Prasanna

[Prasanna .S]

4:37 the balancer has started and disk shot up.

[Prasanna .S]

Balancer stopped at 4:45

[Prasanna .S]

Let me know if you need anything else..

[Scott Hernandez]

Okay, looking at the receiving shard mongo55, and donor shard mongo43,
it look like the fix in 2.0.6 (due out in the next two weeks of so)
will help you out a lot.

[Prasanna .S]

Scott,

I tried to understand what's going on there. But with a quick glance, I assume, a lock was being held longer than necessary and was fixed. It's unclear how this can help me. The only way to reliably make sure the site keeps running while a moveChunk is executing, is if, you yield every second within moveChunk and make sure all pending reads complete, before the next moveChunk period starts.

Are you guys doing it, or planning to do it? Is there any other high-level way to guarantee the reads can be answered within a couple of seconds?

Thanks

[Scott Hernandez]

The logic there is to do each document write, during the migrate, in a
separate lock so other operations can be interleaved. We have seen
this change dramatically reduce the constant lock percentage and allow
reads, and other writes through during a initial move.

There is already a delay between moves which doesn't seem to be your problem.

[Prasanna .S]

Scott,

The below are heuristics, and not guarantees. What you want from a DB stand-point is the latter.

The delay between moves, won't provide a guarantee, unless you make sure that 

(1) the next iteration of moveChunk, does not start unless all pending reads complete.

(2) Each iteration takes less than 1second.

Thoughts?

[Scott Hernandez]

There are lots of ways to look at this with some competing goals.
While it might be paramount for some situations/use-cases to complete
all reads before a move starts, it might not be for others (which I
have seen). This fix will allow reads (and other writes) to get
processes while chunks moves happen, period. The issue of locking, and
contentions on those locks, is very important and there are lots of
improvements being made which help across the boards.

Feel free to add comments or create new issues with your ideas of how
your system would like things to best behave:
http://jira.mongodb.org/browse/server

[Prasanna .S]

Scott,

> While it might be paramount for some situations/use-cases to complete all reads before a move starts, it might not be for others (which I

have seen).

Can you please explain the "might not be for others" part? Can you give me an example?

[Prasanna .S]

Let me re-phrase that: When would someone ever rather have the site to timeout to make progress on replication?

[Scott Hernandez]

A simple example is a system where they want the balancer to run as
quickly as possible at low traffic times to balancer data and don't
care about the impact of (user) reads. This can be a big deal when you
are already under-provisioned and need to add a new shard to handle
users more effectively -- it has come up a few times.

[Prasanna .S]

That use case can easily be supported by just shutting off the reads from the application. Basically they are anyways taking downtime -- that doesn't take special support, does it?

In the normal case, don't you think the reads are the highest priority? Come on, this is production.. And I'm telling you, your top clients have this problem. Whats worse for a DB than downtime?

[Scott Hernandez]

You seem to looking at this as a binary result. Slow reads do not mean
downtime. There are many use-cases where many times performance
degradation during maintenance windows is acceptable but downtime
(stopping reads) is not.

We are working with everyone to make sure they can get the best
performance and uptime for their systems. If you have more specific
areas where you are seeing issue, please create a new thread/post or
jira issue with the problem (or vote/watch for existing ones).

Nobody thinks the system is perfect as is (how could it be when there
are so many different use cases), and there are lots of improvements,
fixes and changes going in to address these areas for improvement.

[Shi Shei]

Put me in the list as well. We are suffering from exactly the same issues that Prasanna is describing.
Mongo pretends to be fast with commodity hardware when you use sharding but even with sharding and high performance hardware we don't see it.

Have a look at our MMS (group name "idealo", all hosts containing "offerstore"). Profiling is enabled for queries taking longer than 3 seconds - and there are a lot of them!
We followed and have mostly applied 10gens production rules but the results are still invisible.
Sorry to say but with such expensive hardware, mongo performance is not that good as expected.

If you have some pointers, we'll gladly follow your instructions. However, just waiting for the next and still next mongo release is quite demotivating.

>> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak@gmail.com>

>> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > --
>> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >>> >>>> >> > to
>> >> >> >> >>> >>>> >> > the
>> >> >> >> >>> >>>> >> > Google
>> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >>> >>>> >> > mongod...@googlegroups.com.
>> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >>
>> >> >> >> >>> >>>> >> --
>> >> >> >> >>> >>>> >> You received this message because you are subscribed
>> >> >> >> >>> >>>> >> to
>> >> >> >> >>> >>>> >> the
>> >> >> >> >>> >>>> >> Google
>> >> >> >> >>> >>>> >> Groups
>> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >>> >>>> >> mongod...@googlegroups.com.
>> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >>
>> >> >> >> >>> >>>> >
>> >> >> >> >>> >>>> > --
>> >> >> >> >>> >>>> > You received this message because you are subscribed
>> >> >> >> >>> >>>> > to
>> >> >> >> >>> >>>> > the
>> >> >> >> >>> >>>> > Google
>> >> >> >> >>> >>>> > Groups
>> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >>> >>>> > mongod...@googlegroups.com.
>> >> >> >> >>> >>>> > To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>> --
>> >> >> >> >>> >>>> You received this message because you are subscribed to
>> >> >> >> >>> >>>> the
>> >> >> >> >>> >>>> Google
>> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >>> >>>> mongod...@googlegroups.com.
>> >> >> >> >>> >>>> To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>
>> >> >> >> >>> >>> --
>> >> >> >> >>> >>> You received this message because you are subscribed to
>> >> >> >> >>> >>> the
>> >> >> >> >>> >>> Google
>> >> >> >> >>> >>> Groups
>> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >>> >>> mongod...@googlegroups.com.
>> >> >> >> >>> >>> To unsubscribe from this group, send email to

>> >> >> >> >>> >>> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >> --
>> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >>> >> the
>> >> >> >> >>> >> Google
>> >> >> >> >>> >> Groups
>> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >>> >> mongod...@googlegroups.com.
>> >> >> >> >>> >> To unsubscribe from this group, send email to

>> >> >> >> >>> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> > --
>> >> >> >> >>> > You received this message because you are subscribed to the
>> >> >> >> >>> > Google
>> >> >> >> >>> > Groups
>> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >>> > mongod...@googlegroups.com.
>> >> >> >> >>> > To unsubscribe from this group, send email to

>> >> >> >> >>> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>>
>> >> >> >> >>> --
>> >> >> >> >>> You received this message because you are subscribed to the
>> >> >> >> >>> Google
>> >> >> >> >>> Groups
>> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >>> To post to this group, send email to
>> >> >> >> >>> mongod...@googlegroups.com.
>> >> >> >> >>> To unsubscribe from this group, send email to

>> >> >> >> >>> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> For more options, visit this group at
>> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongod...@googlegroups.com.
>> >> >> >> > To unsubscribe from this group, send email to

>> >> >> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongod...@googlegroups.com.
>> >> >> >> To unsubscribe from this group, send email to

>> >> >> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongod...@googlegroups.com.
>> >> >> > To unsubscribe from this group, send email to

>> >> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> >> To unsubscribe from this group, send email to

>> >> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> To unsubscribe from this group, send email to

>> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to

>> > mongodb-user+unsubscribe@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongod...@googlegroups.com.
>> To unsubscribe from this group, send email to

>> mongodb-user+unsubscribe@googlegroups.com.

>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

> mongodb-user+unsubscribe@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

>> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak@gmail.com>

>> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > --
>> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >>> >>>> >> > to
>> >> >> >> >>> >>>> >> > the
>> >> >> >> >>> >>>> >> > Google
>> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >>> >>>> >> > mongod...@googlegroups.com.
>> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >>
>> >> >> >> >>> >>>> >> --
>> >> >> >> >>> >>>> >> You received this message because you are subscribed
>> >> >> >> >>> >>>> >> to
>> >> >> >> >>> >>>> >> the
>> >> >> >> >>> >>>> >> Google
>> >> >> >> >>> >>>> >> Groups
>> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >>> >>>> >> mongod...@googlegroups.com.
>> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >>
>> >> >> >> >>> >>>> >
>> >> >> >> >>> >>>> > --
>> >> >> >> >>> >>>> > You received this message because you are subscribed
>> >> >> >> >>> >>>> > to
>> >> >> >> >>> >>>> > the
>> >> >> >> >>> >>>> > Google
>> >> >> >> >>> >>>> > Groups
>> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >>> >>>> > mongod...@googlegroups.com.
>> >> >> >> >>> >>>> > To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>> --
>> >> >> >> >>> >>>> You received this message because you are subscribed to
>> >> >> >> >>> >>>> the
>> >> >> >> >>> >>>> Google
>> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >>> >>>> mongod...@googlegroups.com.
>> >> >> >> >>> >>>> To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>
>> >> >> >> >>> >>> --
>> >> >> >> >>> >>> You received this message because you are subscribed to
>> >> >> >> >>> >>> the
>> >> >> >> >>> >>> Google
>> >> >> >> >>> >>> Groups
>> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >>> >>> mongod...@googlegroups.com.
>> >> >> >> >>> >>> To unsubscribe from this group, send email to

>> >> >> >> >>> >>> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >> --
>> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >>> >> the
>> >> >> >> >>> >> Google
>> >> >> >> >>> >> Groups
>> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >>> >> mongod...@googlegroups.com.
>> >> >> >> >>> >> To unsubscribe from this group, send email to

>> >> >> >> >>> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> > --
>> >> >> >> >>> > You received this message because you are subscribed to the
>> >> >> >> >>> > Google
>> >> >> >> >>> > Groups
>> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >>> > mongod...@googlegroups.com.
>> >> >> >> >>> > To unsubscribe from this group, send email to

>> >> >> >> >>> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>>
>> >> >> >> >>> --
>> >> >> >> >>> You received this message because you are subscribed to the
>> >> >> >> >>> Google
>> >> >> >> >>> Groups
>> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >>> To post to this group, send email to
>> >> >> >> >>> mongod...@googlegroups.com.
>> >> >> >> >>> To unsubscribe from this group, send email to

>> >> >> >> >>> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> For more options, visit this group at
>> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongod...@googlegroups.com.
>> >> >> >> > To unsubscribe from this group, send email to

>> >> >> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongod...@googlegroups.com.
>> >> >> >> To unsubscribe from this group, send email to

>> >> >> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongod...@googlegroups.com.
>> >> >> > To unsubscribe from this group, send email to

>> >> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> >> To unsubscribe from this group, send email to

>> >> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> To unsubscribe from this group, send email to

>> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to

>> > mongodb-user+unsubscribe@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongod...@googlegroups.com.
>> To unsubscribe from this group, send email to

>> mongodb-user+unsubscribe@googlegroups.com.

>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

> mongodb-user+unsubscribe@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

>> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak@gmail.com>

>> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > --
>> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >>> >>>> >> > to
>> >> >> >> >>> >>>> >> > the
>> >> >> >> >>> >>>> >> > Google
>> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >>> >>>> >> > mongod...@googlegroups.com.
>> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >>
>> >> >> >> >>> >>>> >> --
>> >> >> >> >>> >>>> >> You received this message because you are subscribed
>> >> >> >> >>> >>>> >> to
>> >> >> >> >>> >>>> >> the
>> >> >> >> >>> >>>> >> Google
>> >> >> >> >>> >>>> >> Groups
>> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >>> >>>> >> mongod...@googlegroups.com.
>> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >>
>> >> >> >> >>> >>>> >
>> >> >> >> >>> >>>> > --
>> >> >> >> >>> >>>> > You received this message because you are subscribed
>> >> >> >> >>> >>>> > to
>> >> >> >> >>> >>>> > the
>> >> >> >> >>> >>>> > Google
>> >> >> >> >>> >>>> > Groups
>> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >>> >>>> > mongod...@googlegroups.com.
>> >> >> >> >>> >>>> > To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>> --
>> >> >> >> >>> >>>> You received this message because you are subscribed to
>> >> >> >> >>> >>>> the
>> >> >> >> >>> >>>> Google
>> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >>> >>>> mongod...@googlegroups.com.
>> >> >> >> >>> >>>> To unsubscribe from this group, send email to

>> >> >> >> >>> >>>> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>
>> >> >> >> >>> >>> --
>> >> >> >> >>> >>> You received this message because you are subscribed to
>> >> >> >> >>> >>> the
>> >> >> >> >>> >>> Google
>> >> >> >> >>> >>> Groups
>> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >>> >>> mongod...@googlegroups.com.
>> >> >> >> >>> >>> To unsubscribe from this group, send email to

>> >> >> >> >>> >>> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >> --
>> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >>> >> the
>> >> >> >> >>> >> Google
>> >> >> >> >>> >> Groups
>> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >>> >> mongod...@googlegroups.com.
>> >> >> >> >>> >> To unsubscribe from this group, send email to

>> >> >> >> >>> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> > --
>> >> >> >> >>> > You received this message because you are subscribed to the
>> >> >> >> >>> > Google
>> >> >> >> >>> > Groups
>> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >>> > mongod...@googlegroups.com.
>> >> >> >> >>> > To unsubscribe from this group, send email to

>> >> >> >> >>> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>>
>> >> >> >> >>> --
>> >> >> >> >>> You received this message because you are subscribed to the
>> >> >> >> >>> Google
>> >> >> >> >>> Groups
>> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >>> To post to this group, send email to
>> >> >> >> >>> mongod...@googlegroups.com.
>> >> >> >> >>> To unsubscribe from this group, send email to

>> >> >> >> >>> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> >>> For more options, visit this group at
>> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongod...@googlegroups.com.
>> >> >> >> > To unsubscribe from this group, send email to

>> >> >> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongod...@googlegroups.com.
>> >> >> >> To unsubscribe from this group, send email to

>> >> >> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongod...@googlegroups.com.
>> >> >> > To unsubscribe from this group, send email to

>> >> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> >> To unsubscribe from this group, send email to

>> >> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user+unsubscribe@googlegroups.com.

>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> To unsubscribe from this group, send email to

>> >> mongodb-user+unsubscribe@googlegroups.com.

>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to

>> > mongodb-user+unsubscribe@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongod...@googlegroups.com.
>> To unsubscribe from this group, send email to

>> mongodb-user+unsubscribe@googlegroups.com.

>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

> mongodb-user+unsubscribe@googlegroups.com.

[Scott Hernandez]

Please give 2.0.6 a try when it comes out in a week or two. If you
want to test now you can use the dev/unstable 2.1.1 release.

>> >> >> >> >> >>> >>>> >> >> Tilak<hrushike...@gmail.com>

>> >> >> >> >> >>> >>>> >> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > --
>> >> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > the
>> >> >> >> >> >>> >>>> >> > Google
>> >> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >> >>> >>>> >> > mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> >> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> --
>> >> >> >> >> >>> >>>> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> subscribed
>> >> >> >> >> >>> >>>> >> to
>> >> >> >> >> >>> >>>> >> the
>> >> >> >> >> >>> >>>> >> Google
>> >> >> >> >> >>> >>>> >> Groups
>> >> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > --
>> >> >> >> >> >>> >>>> > You received this message because you are
>> >> >> >> >> >>> >>>> > subscribed
>> >> >> >> >> >>> >>>> > to
>> >> >> >> >> >>> >>>> > the
>> >> >> >> >> >>> >>>> > Google
>> >> >> >> >> >>> >>>> > Groups
>> >> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >> >>> >>>> > mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> --
>> >> >> >> >> >>> >>>> You received this message because you are subscribed
>> >> >> >> >> >>> >>>> to
>> >> >> >> >> >>> >>>> the
>> >> >> >> >> >>> >>>> Google
>> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >> >>> >>>> mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> --
>> >> >> >> >> >>> >>> You received this message because you are subscribed
>> >> >> >> >> >>> >>> to
>> >> >> >> >> >>> >>> the
>> >> >> >> >> >>> >>> Google
>> >> >> >> >> >>> >>> Groups
>> >> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >> >>> >>> mongod...@googlegroups.com.
>> >> >> >> >> >>> >>> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> --
>> >> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >> >>> >> the
>> >> >> >> >> >>> >> Google
>> >> >> >> >> >>> >> Groups
>> >> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >> >>> >> mongod...@googlegroups.com.
>> >> >> >> >> >>> >> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > --
>> >> >> >> >> >>> > You received this message because you are subscribed to
>> >> >> >> >> >>> > the
>> >> >> >> >> >>> > Google
>> >> >> >> >> >>> > Groups
>> >> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >> >>> > mongod...@googlegroups.com.
>> >> >> >> >> >>> > To unsubscribe from this group, send email to

>> >> >> >> >> >>> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>> --
>> >> >> >> >> >>> You received this message because you are subscribed to
>> >> >> >> >> >>> the
>> >> >> >> >> >>> Google
>> >> >> >> >> >>> Groups
>> >> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >> >>> To post to this group, send email to
>> >> >> >> >> >>> mongod...@googlegroups.com.
>> >> >> >> >> >>> To unsubscribe from this group, send email to

>> >> >> >> >> >>> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> For more options, visit this group at
>> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> >> > Google
>> >> >> >> >> > Groups
>> >> >> >> >> > "mongodb-user" group.
>> >> >> >> >> > To post to this group, send email to
>> >> >> >> >> > mongod...@googlegroups.com.
>> >> >> >> >> > To unsubscribe from this group, send email to

>> >> >> >> >> > mongodb-user...@googlegroups.com.

>> >> >> >> >> > For more options, visit this group at
>> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> You received this message because you are subscribed to the
>> >> >> >> >> Google
>> >> >> >> >> Groups
>> >> >> >> >> "mongodb-user" group.
>> >> >> >> >> To post to this group, send email to
>> >> >> >> >> mongod...@googlegroups.com.
>> >> >> >> >> To unsubscribe from this group, send email to

>> >> >> >> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> For more options, visit this group at
>> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongod...@googlegroups.com.
>> >> >> >> > To unsubscribe from this group, send email to

>> >> >> >> > mongodb-user...@googlegroups.com.

>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongod...@googlegroups.com.
>> >> >> >> To unsubscribe from this group, send email to

>> >> >> >> mongodb-user...@googlegroups.com.

>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongod...@googlegroups.com.
>> >> >> > To unsubscribe from this group, send email to

>> >> >> > mongodb-user...@googlegroups.com.

>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> >> To unsubscribe from this group, send email to

>> >> >> mongodb-user...@googlegroups.com.

>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user...@googlegroups.com.

>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> To unsubscribe from this group, send email to

>> >> mongodb-user...@googlegroups.com.

>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to

>> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> >> Tilak<hrushike...@gmail.com>

>> >> >> >> >> >>> >>>> >> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > --
>> >> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > the
>> >> >> >> >> >>> >>>> >> > Google
>> >> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >> >>> >>>> >> > mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> >> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> --
>> >> >> >> >> >>> >>>> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> subscribed
>> >> >> >> >> >>> >>>> >> to
>> >> >> >> >> >>> >>>> >> the
>> >> >> >> >> >>> >>>> >> Google
>> >> >> >> >> >>> >>>> >> Groups
>> >> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > --
>> >> >> >> >> >>> >>>> > You received this message because you are
>> >> >> >> >> >>> >>>> > subscribed
>> >> >> >> >> >>> >>>> > to
>> >> >> >> >> >>> >>>> > the
>> >> >> >> >> >>> >>>> > Google
>> >> >> >> >> >>> >>>> > Groups
>> >> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >> >>> >>>> > mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> --
>> >> >> >> >> >>> >>>> You received this message because you are subscribed
>> >> >> >> >> >>> >>>> to
>> >> >> >> >> >>> >>>> the
>> >> >> >> >> >>> >>>> Google
>> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >> >>> >>>> mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> --
>> >> >> >> >> >>> >>> You received this message because you are subscribed
>> >> >> >> >> >>> >>> to
>> >> >> >> >> >>> >>> the
>> >> >> >> >> >>> >>> Google
>> >> >> >> >> >>> >>> Groups
>> >> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >> >>> >>> mongod...@googlegroups.com.
>> >> >> >> >> >>> >>> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> --
>> >> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >> >>> >> the
>> >> >> >> >> >>> >> Google
>> >> >> >> >> >>> >> Groups
>> >> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >> >>> >> mongod...@googlegroups.com.
>> >> >> >> >> >>> >> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > --
>> >> >> >> >> >>> > You received this message because you are subscribed to
>> >> >> >> >> >>> > the
>> >> >> >> >> >>> > Google
>> >> >> >> >> >>> > Groups
>> >> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >> >>> > mongod...@googlegroups.com.
>> >> >> >> >> >>> > To unsubscribe from this group, send email to

>> >> >> >> >> >>> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>> --
>> >> >> >> >> >>> You received this message because you are subscribed to
>> >> >> >> >> >>> the
>> >> >> >> >> >>> Google
>> >> >> >> >> >>> Groups
>> >> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >> >>> To post to this group, send email to
>> >> >> >> >> >>> mongod...@googlegroups.com.
>> >> >> >> >> >>> To unsubscribe from this group, send email to

>> >> >> >> >> >>> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> For more options, visit this group at
>> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> >> > Google
>> >> >> >> >> > Groups
>> >> >> >> >> > "mongodb-user" group.
>> >> >> >> >> > To post to this group, send email to
>> >> >> >> >> > mongod...@googlegroups.com.
>> >> >> >> >> > To unsubscribe from this group, send email to

>> >> >> >> >> > mongodb-user...@googlegroups.com.

>> >> >> >> >> > For more options, visit this group at
>> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> You received this message because you are subscribed to the
>> >> >> >> >> Google
>> >> >> >> >> Groups
>> >> >> >> >> "mongodb-user" group.
>> >> >> >> >> To post to this group, send email to
>> >> >> >> >> mongod...@googlegroups.com.
>> >> >> >> >> To unsubscribe from this group, send email to

>> >> >> >> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> For more options, visit this group at
>> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongod...@googlegroups.com.
>> >> >> >> > To unsubscribe from this group, send email to

>> >> >> >> > mongodb-user...@googlegroups.com.

>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongod...@googlegroups.com.
>> >> >> >> To unsubscribe from this group, send email to

>> >> >> >> mongodb-user...@googlegroups.com.

>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongod...@googlegroups.com.
>> >> >> > To unsubscribe from this group, send email to

>> >> >> > mongodb-user...@googlegroups.com.

>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> >> To unsubscribe from this group, send email to

>> >> >> mongodb-user...@googlegroups.com.

>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user...@googlegroups.com.

>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> To unsubscribe from this group, send email to

>> >> mongodb-user...@googlegroups.com.

>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to

>> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> >> Tilak<hrushike...@gmail.com>

>> >> >> >> >> >>> >>>> >> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > --
>> >> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > the
>> >> >> >> >> >>> >>>> >> > Google
>> >> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >> >>> >>>> >> > mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> >> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> --
>> >> >> >> >> >>> >>>> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> subscribed
>> >> >> >> >> >>> >>>> >> to
>> >> >> >> >> >>> >>>> >> the
>> >> >> >> >> >>> >>>> >> Google
>> >> >> >> >> >>> >>>> >> Groups
>> >> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > --
>> >> >> >> >> >>> >>>> > You received this message because you are
>> >> >> >> >> >>> >>>> > subscribed
>> >> >> >> >> >>> >>>> > to
>> >> >> >> >> >>> >>>> > the
>> >> >> >> >> >>> >>>> > Google
>> >> >> >> >> >>> >>>> > Groups
>> >> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >> >>> >>>> > mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> --
>> >> >> >> >> >>> >>>> You received this message because you are subscribed
>> >> >> >> >> >>> >>>> to
>> >> >> >> >> >>> >>>> the
>> >> >> >> >> >>> >>>> Google
>> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >> >>> >>>> mongod...@googlegroups.com.
>> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>>> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> --
>> >> >> >> >> >>> >>> You received this message because you are subscribed
>> >> >> >> >> >>> >>> to
>> >> >> >> >> >>> >>> the
>> >> >> >> >> >>> >>> Google
>> >> >> >> >> >>> >>> Groups
>> >> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >> >>> >>> mongod...@googlegroups.com.
>> >> >> >> >> >>> >>> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >>> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> --
>> >> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >> >>> >> the
>> >> >> >> >> >>> >> Google
>> >> >> >> >> >>> >> Groups
>> >> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >> >>> >> mongod...@googlegroups.com.
>> >> >> >> >> >>> >> To unsubscribe from this group, send email to

>> >> >> >> >> >>> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > --
>> >> >> >> >> >>> > You received this message because you are subscribed to
>> >> >> >> >> >>> > the
>> >> >> >> >> >>> > Google
>> >> >> >> >> >>> > Groups
>> >> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >> >>> > mongod...@googlegroups.com.
>> >> >> >> >> >>> > To unsubscribe from this group, send email to

>> >> >> >> >> >>> > mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>> --
>> >> >> >> >> >>> You received this message because you are subscribed to
>> >> >> >> >> >>> the
>> >> >> >> >> >>> Google
>> >> >> >> >> >>> Groups
>> >> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >> >>> To post to this group, send email to
>> >> >> >> >> >>> mongod...@googlegroups.com.
>> >> >> >> >> >>> To unsubscribe from this group, send email to

>> >> >> >> >> >>> mongodb-user...@googlegroups.com.

>> >> >> >> >> >>> For more options, visit this group at
>> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> >> > Google
>> >> >> >> >> > Groups
>> >> >> >> >> > "mongodb-user" group.
>> >> >> >> >> > To post to this group, send email to
>> >> >> >> >> > mongod...@googlegroups.com.
>> >> >> >> >> > To unsubscribe from this group, send email to

>> >> >> >> >> > mongodb-user...@googlegroups.com.

>> >> >> >> >> > For more options, visit this group at
>> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> You received this message because you are subscribed to the
>> >> >> >> >> Google
>> >> >> >> >> Groups
>> >> >> >> >> "mongodb-user" group.
>> >> >> >> >> To post to this group, send email to
>> >> >> >> >> mongod...@googlegroups.com.
>> >> >> >> >> To unsubscribe from this group, send email to

>> >> >> >> >> mongodb-user...@googlegroups.com.

>> >> >> >> >> For more options, visit this group at
>> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongod...@googlegroups.com.
>> >> >> >> > To unsubscribe from this group, send email to

>> >> >> >> > mongodb-user...@googlegroups.com.

>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongod...@googlegroups.com.
>> >> >> >> To unsubscribe from this group, send email to

>> >> >> >> mongodb-user...@googlegroups.com.

>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongod...@googlegroups.com.
>> >> >> > To unsubscribe from this group, send email to

>> >> >> > mongodb-user...@googlegroups.com.

>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> >> To unsubscribe from this group, send email to

>> >> >> mongodb-user...@googlegroups.com.

>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongod...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to

>> >> > mongodb-user...@googlegroups.com.

>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongod...@googlegroups.com.
>> >> To unsubscribe from this group, send email to

>> >> mongodb-user...@googlegroups.com.

>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > To unsubscribe from this group, send email to

>> > mongodb-user...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/mongodb-user/-/YRlv9SXrV6QJ.

> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

> mongodb-user...@googlegroups.com.

[Prasanna .S]

Scott,

> You seem to looking at this as a binary result. Slow reads do not mean downtime.

If this were true I'd be fine. Performance degradation is fine, downtime is not. Right now the result is binary.

So, how do we guarantee, performance degradation will not cause downtime? You make sure the read query queue, doesn't get infinitely long. Alternatively, you can ensure, because of moveChunk, reads wait at most 1 second longer.

My worry is, you are proposing a heuristic to reduce lock contention, but no guarantees. I worked on a database for Microsoft Bing which runs on 300K machines, and the diff between the two scare me. The diff is, on a long enough timeframe, heuristics absolutely break, and guarantees do not.

I definitely do want a convincing answer to this one. If you can somehow tell me, there won't be downtime in the future, because of this issue, that'd be great. You have to understand though, there is right now, and it's not pretty.

Thanks,

-Prasanna

[Prasanna .S]

> Nobody thinks the system is perfect as is (how could it be when there
> are so many different use cases), and there are lots of improvements,
> fixes and changes going in to address these areas for improvement.

Of-course, I agree with you on this. If my queries just run 10x slower during maintenance, I won't complain here.

Saying the same to down-time is not acceptable. At-least if downtimes should be like 1 in a million or billion chance of happening, I can bear with it. It happens *all* the time to *everyone* using sharding.

Let's figure out where we disagree on:

1) FourSquare and other of your top clients, run with moveChunk off during production, which otherwise causes downtime? Yes/No

2) This is the highest priority of MongoDB as a database, to solve right now. Yes/No

If both the answers are Yes, we both agree and I'd just make a Jira thread and monitor when it ships. You guys have been incredibly responsive and that's the main reason why I haven't moved away or blogged in TechCrunch about this otherwise fatal problem.

I'm open to hearing alternate explanations: Is there a way you can guarantee, moveChunk won't take more than 10% of disk-time and cpu time etc. I suspect any such approaches, will end up being way more complex, than executing things based on priorities.

Thanks,

-Prasanna

[Eliot Horowitz]

> 1) FourSquare and other of your top clients, run with moveChunk off during
> production, which otherwise causes downtime? Yes/No

No, most people run with balancing on 24/7.

> 2) This is the highest priority of MongoDB as a database, to solve right
> now. Yes/No

The general issue here is overall concurrency, which is the highest priority.

As Scott said, 2.0.6 will have some significant improvements for this
case as well.

[Prasanna .S]

Eliot, Foursquare does not. MongoHQ folks recommend people not to. Bump stopped using MongoDB. How many more examples do you want?

[Eliot Horowitz]

I can't discuss foursquare further.
Many people (most by my count) have balancing on 24x7

There are a lot of factors that change performance, notable doc size
and disk speed.

There are definitely scenarios where balancing can cause problems
because of concurrency, and those are what we are working on.
2.0.5 had one big one and 2.0.6 will have a very large one.
2.2 will add a lot more concurrency overall, so will help tremendously
with this issue as well.

[Prasanna .S]

> I can't discuss foursquare further.

You added them as your top customer.

> Many people (most by my count) have balancing on 24x7

Can you tell me what's different about them? The people I asked seem to be having problems running it 24x7. All of us certainly have data fitting in memory. Can you give me one contact?

> There are definitely scenarios where balancing can cause problems
> because of concurrency, and those are what we are working on.

On one hand I'm happy that you are working on it. On the other, I'm not happy about the guarantees being very weak. "Improve" is a bad word to use, when associated with DB downtime. 

[Eliot Horowitz]

>> I can't discuss foursquare further.
> You added them as your top customer.

And they are a client, so I cannot discuss it.

>> Many people (most by my count) have balancing on 24x7
>
> Can you tell me what's different about them? The people I asked seem to be
> having problems running it 24x7. All of us certainly have data fitting in
> memory. Can you give me one contact?

Most people running do not have all data fitting in memory.
If no one step on this thread if you contact me offline I can probably
set something up.

>> There are definitely scenarios where balancing can cause problems
>> because of concurrency, and those are what we are working on.
>
> On one hand I'm happy that you are working on it. On the other, I'm not
> happy about the guarantees being very weak. "Improve" is a bad word to use,
> when associated with DB downtime.

Guarantees are hard when we don't control hardware, etc...
There are a lot of factors, so a 100% guarantee would take a lot to get to.

[Eliot Horowitz]

On Fri, May 11, 2012 at 10:52 AM, Prasanna .S <mypra...@gmail.com> wrote:
> I think we'd agree on 3 high level ways to provide a guarantee in this
> specific instance.
> 1. Replicate from the slaves. Mysql does this. -- Probably too complex for
> you guys at this time.

Do you mean chained replication, or something else?
We already do chained replication in certain cases.

> 2. Throttle and replicate only as long as the disk util is < 50% or so AND
> cpu util is < 50%. -- Probably complex as well.

This is tricky.
For example, some people run with disk util > 50% all the time.

> 3. Have the moveChunk interruptible. If reads and writes come in-between,
> consider them as higher priority, so that a read/write is never waiting for

> more than a second. -- Should not be extremely hard to do.

This is the direction the changes in 2.0.5 and 2.0.6 have gone.

> Do you agree the above 3 ways provide a guarantee, regardless of hardware?

No - I do not think those three things provide a guarantee.
When you add load to a system, there are a lot of factors at play, so
I would definitely not say even with those 3 things, that it could not
cause an issue.
Especially if you're operating in a system where the hardware isn't
predictable itself, there are many possible issues.

What I can guarantee is that we will keep working on it till its not an issue.

> Sorry to be an assole here Eliot. You can google search for me, and I've
> never ever debated an open source software this passionately. I really got
> burnt this time in production. People who use mongo regularly said it's a
> very common problem. I think, if others knew what they were getting into,
> half the CTOs won't choose mongo.

I'm very sorry you got burnt, I don't know the specifics of your case,
or why it happened, but certainly happy to try and understand the
specifics.
If you drill down into specifics, the causes are often very different.

[Prasanna .S]

> No - I do not think those three things provide a guarantee.

I was saying any one of the 3 provide a guarantee. 

Let's take the last one alone - 

a) moveChunk executes it's iteration, iff all reads are completed. 

b) we guarantee that a single moveChunk iteration takes < 1s.

(b) Can be a bit tricky because you have to make sure, the iteration which includes the time the writes are waiting in disk buffers, finish within that 1s. But definitely not very hard.

I'm hypothesizing that this provides a complete guarantee, regardless of hardware. Can you explain a bit, why my reasoning is flawed? 

> If you drill down into specifics, the causes are often very different.

First of all, thank you so much for offering to help. I have never seen this level of commitment and professionalism elsewhere and I really appreciate it.

From my basic drill-down, it was the last part of moveChunk, which was deleting the date from the master-shard that was causing tonnes of disk writes. It makes me think, even if you make sure the moveChunk only marks a chunk as deleted in Primary, but doesn't physically touch the disk, just unloads the pages from RAM, that might work too.

Very happy to co-operate with you if you want to investigate further on the machines. Glad to provide shell level access.

[Eliot Horowitz]

> a) moveChunk executes it's iteration, iff all reads are completed.
> b) we guarantee that a single moveChunk iteration takes < 1s.

1 second is too long for some apps.
there are mongo users who have 20ms SLAs on db reads for example.

> (b) Can be a bit tricky because you have to make sure, the iteration which
> includes the time the writes are waiting in disk buffers, finish within that
> 1s. But definitely not very hard.
>
> I'm hypothesizing that this provides a complete guarantee, regardless of
> hardware. Can you explain a bit, why my reasoning is flawed?

For example, if you were on a single spinning disk, and you need to
load data that will cause a seek, and you do this a few times per
second, it could degrade read performance enough to have a significant
impact.
Now, if you're at that point you probably wait too long to shard, but
its a gray area, so wouldn't want to guarantee anything/

> From my basic drill-down, it was the last part of moveChunk, which was
> deleting the date from the master-shard that was causing tonnes of disk
> writes. It makes me think, even if you make sure the moveChunk only marks a
> chunk as deleted in Primary, but doesn't physically touch the disk, just
> unloads the pages from RAM, that might work too.

Interesting. We do have to actually delete the data so the space can
be re-used, but might be able to do it more efficiently.
The code should be yielding before actually doing anything, so kind of
surprising it caused too much load on the primary.
How active is your disk at steady state?

There is one quick change that would be interesting to try.
When we first added sharding a couple years ago, we put it a pedantic
safety measure of writing out all the data we delete after a chunk is
moved just to be sure.
I wonder how much load that is adding.
Would be trivial to remove.

Do you have a lab where you can reproduce this easily?

[Prasanna .S]

On Fri, May 11, 2012 at 7:36 AM, Eliot Horowitz <el...@10gen.com> wrote:

>> I can't discuss foursquare further.

> You added them as your top customer.

And they are a client, so I cannot discuss it.

If you want to take this offline, I can connect with my friends at foursquare. As far as I know, the lead there just told me they don't run it 24x7.

 

>> Many people (most by my count) have balancing on 24x7
>
> Can you tell me what's different about them? The people I asked seem to be
> having problems running it 24x7. All of us certainly have data fitting in
> memory. Can you give me one contact?

Most people running do not have all data fitting in memory.
If no one step on this thread if you contact me offline I can probably
set something up.

Thanks - I just want to understand what's different. We both just disagree on the impact of this problem.

 

>> There are definitely scenarios where balancing can cause problems
>> because of concurrency, and those are what we are working on.
>
> On one hand I'm happy that you are working on it. On the other, I'm not
> happy about the guarantees being very weak. "Improve" is a bad word to use,
> when associated with DB downtime.

Guarantees are hard when we don't control hardware, etc...
There are a lot of factors, so a 100% guarantee would take a lot to get to.

I think we'd agree on 3 high level ways to provide a guarantee in this specific instance.

1. Replicate from the slaves. Mysql does this. -- Probably too complex for you guys at this time.

2. Throttle and replicate only as long as the disk util is < 50% or so AND cpu util is < 50%. -- Probably complex as well.

3. Have the moveChunk interruptible. If reads and writes come in-between, consider them as higher priority, so that a read/write is never waiting for more than a second. -- Should not be extremely hard to do.

Do you agree the above 3 ways provide a guarantee, regardless of hardware?

Sorry to be an assole here Eliot. You can google search for me, and I've never ever debated an open source software this passionately. I really got burnt this time in production. People who use mongo regularly said it's a very common problem. I think, if others knew what they were getting into, half the CTOs won't choose mongo.

 

>

[Prasanna .S]

inline...

On Fri, May 11, 2012 at 8:24 AM, Eliot Horowitz <el...@10gen.com> wrote:

> a) moveChunk executes it's iteration, iff all reads are completed.
> b) we guarantee that a single moveChunk iteration takes < 1s.

1 second is too long for some apps.
there are mongo users who have 20ms SLAs on db reads for example.

Sure, 1s is an example time unit. If you go lower, I can't be less happier ;)

 

> (b) Can be a bit tricky because you have to make sure, the iteration which
> includes the time the writes are waiting in disk buffers, finish within that
> 1s. But definitely not very hard.
>
> I'm hypothesizing that this provides a complete guarantee, regardless of
> hardware. Can you explain a bit, why my reasoning is flawed?

For example, if you were on a single spinning disk, and you need to
load data that will cause a seek, and you do this a few times per
second, it could degrade read performance enough to have a significant
impact. 

Now, if you're at that point you probably wait too long to shard, but
its a gray area, so wouldn't want to guarantee anything/

Time_1: Time taken for read X to complete without replication.

Time_2: The same with replication.

Guarantee is this: 

Time_2 <= Time_1 + 1 second;

i.e., The disk read queue filling up problem has nothing to do with moveChunk. moveChunk only added one second to the otherwise downtime causing environment. 

Makes sense right?

 

> From my basic drill-down, it was the last part of moveChunk, which was
> deleting the date from the master-shard that was causing tonnes of disk
> writes. It makes me think, even if you make sure the moveChunk only marks a
> chunk as deleted in Primary, but doesn't physically touch the disk, just
> unloads the pages from RAM, that might work too.

Interesting.  We do have to actually delete the data so the space can
be re-used, but might be able to do it more efficiently.
The code should be yielding before actually doing anything, so kind of
surprising it caused too much load on the primary.

Disk going to 100% was the issue.

 

How active is your disk at steady state?

< 5%; Usually 1% or less.

 

There is one quick change that would be interesting to try.
When we first added sharding a couple years ago, we put it a pedantic
safety measure of writing out all the data we delete after a chunk is
moved just to be sure.
I wonder how much load that is adding.
Would be trivial to remove.

Awesome. Nice to hear that.

 

Do you have a lab where you can reproduce this easily?

I don't really have a lab. Only have the production environment as of now. Each moveChunk gives about 4-12 minutes of downtime. If you want we can try it sometime later in the night.

I suspect a vanilla setup, should be able to reproduce this though, since it's pretty commonly happening across companies.

[Jeff Lee]

Hey Eliot,

We've been having a problem with this for a while.  I can force load a chunk into memory so that the actual move and metadata change takes less than 20s but the removal can take 15 to 20 min during which the lock pct on the primary goes to > 90%.

The lock yields for other operations but our average application latency doubles during this period.

Could this be caused by use of a shard key other than _id?

The change you mentioned sounds promising...can you point me to the code?

Thanks.

[Eliot Horowitz]

How many indexes do you both have?

Jeff, what version?

For the change, there is actually a command line arg: --noMoveParanoia

Again, sample docs or a reproducible case would be great, as there are
a lot factors.

[Jeff Lee]

Awesome...I'll run some tests with that and gather some info for a ticket if that doesn't doesn't do the trick.

Thanks!

[Kenny Gorman]

Just speaking out loud here, but what about a 'deferred delete' option?

aka: not to delete the old data post move, but rather just leave it in place and somehow mark it as 'old'. Then properly timed and throttled jobs could clean up later? I think this came up before in some discussions.

You are relying on the config image to ensure you don't get logical corruption however with this approach.

I have used that approach for large manually sharded PostgreSQL clusters with success.

-kg

[Adrien Mogenet]

On Friday, May 11, 2012 7:30:20 PM UTC+2, k...@kennygorman.com wrote:

Just speaking out loud here, but what about a 'deferred delete' option?

This is the choice of HBase, and probably many other products. I have to admit that I think it's a valuable trade-off : we did not meet a lot of situations where getting free space immediately does matter. Btw, I have to run repairDatabase() as crontab to get free space back, so I think that "deferred delete" can be helpful, used with a great defragmentation policy, but I'm not sure this is the topic here... 

 

>> mongodb-user+unsubscribe@googlegroups.com.

>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.

> To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.

[Glenn Maynard]

On Fri, May 11, 2012 at 10:11 AM, Prasanna .S <mypra...@gmail.com> wrote:

Let's take the last one alone - 

a) moveChunk executes it's iteration, iff all reads are completed. 

I think what you mean is for moveChunk to periodically yield long enough for everything previously in the read queue to finish.

That would probably just result in your four minute chunk migrations taking hours or days, though, which isn't really a working solution.  Giving priority to reads doesn't help if the reads end up starving out chunk migrations.

On Fri, May 11, 2012 at 12:30 PM, Kenny Gorman <k...@kennygorman.com> wrote:

Just speaking out loud here, but what about a 'deferred delete' option?

This won't help much if you have a constant load.  You have to spend the I/O time sooner or later.

People keep talking about workarounds--yields and delays just shift the time around.  Even if Mongo did any of those, you'd still have a problem; it'd just have taken a bit longer to surface.  The focus should probably be on figuring out why the deletes are taking so much I/O to begin with.

--
Glenn Maynard

[Prasanna .S]

Hey Glenn,

inline..

On Fri, May 11, 2012 at 3:50 PM, Glenn Maynard <gl...@zewt.org> wrote:

On Fri, May 11, 2012 at 10:11 AM, Prasanna .S <mypra...@gmail.com> wrote:

Let's take the last one alone - 

a) moveChunk executes it's iteration, iff all reads are completed. 

I think what you mean is for moveChunk to periodically yield long enough for everything previously in the read queue to finish.

That would probably just result in your four minute chunk migrations taking hours or days, though, which isn't really a working solution.  Giving priority to reads doesn't help if the reads end up starving out chunk migrations.

At 100% capacity, you need to choose. Yes, it'll starve out chunk migrations. The other option is to starve out reads. When one is about keeping the site up, and the other is about a background operation, I think the priorities are pretty clear.

Note that there can be alternate ways of providing a guarantee:

[1] You can say reads will be at 90% capacity, and 10% capacity (CPU AND Disk) goes to chunk movements.

[2] You can say reads will always execute before chunk migrations complete. Yes this will starve chunkMigrations, which is probably fine. Something has to take a hit right? Note that, the other option is, read queue keeps getting incrementally larger, which on saturation - you'll hit timeouts - regardless of weather your timeout numbers are 100 ms or 10s.

 

On Fri, May 11, 2012 at 12:30 PM, Kenny Gorman <k...@kennygorman.com> wrote:

Just speaking out loud here, but what about a 'deferred delete' option?

This won't help much if you have a constant load.  You have to spend the I/O time sooner or later.

Yes, this is true. One alternate solution is to not do the IO at all from the primary. Just mark it as ignored and not run the compaction. It'll take extra disk space, but all databases have de-fragmentation problems. Esp, compare this with the magnitude of the other problem - downtime.

 

People keep talking about workarounds--yields and delays just shift the time around.  Even if Mongo did any of those, you'd still have a problem; it'd just have taken a bit longer to surface.  The focus should probably be on figuring out why the deletes are taking so much I/O to begin with.

Unfortunately, that'll make things better but not solve the problem. Let's say you make the deletes 10x faster, you are talking about 24s of downtime instead of 4 minutes. Throttling the background process reliably is the only way out.

 

--
Glenn Maynard

[Glenn Maynard]

On Fri, May 11, 2012 at 9:18 PM, Prasanna .S <mypra...@gmail.com> wrote:

inline..

(FYI, you don't need to tell people this on technical lists.)

 

Note that there can be alternate ways of providing a guarantee:

[1] You can say reads will be at 90% capacity, and 10% capacity (CPU AND Disk) goes to chunk movements.

[2] You can say reads will always execute before chunk migrations complete. Yes this will starve chunkMigrations, which is probably fine. Something has to take a hit right?

Starving migrations isn't fine.  If you're at capacity, it's just as fatal: it means you'll stay at capacity forever, because you don't have the spare capacity to bring up another shard to handle the load and stop being overloaded.

Note that, the other option is, read queue keeps getting incrementally larger, which on saturation - you'll hit timeouts - regardless of weather your timeout numbers are 100 ms or 10s.

Depending on your workload, that's often not the case.  If you're a website, it means browser requests take longer, which is self-balancing: browsers will only allow so many requests to be in the air at a time.  It'll slow down rather than sending more and more requests, so your work queues won't grow without bound.  (They can still grow enough to cause timeouts, but you can go much further over capacity before that happens.)

If your use case isn't like that, then you can't let yourself end up in this situation to begin with--since *neither* option is acceptable.

Are your records very small (on the order of kilobytes) and fragmented, which might cause the deletes to die from the seeking?  (I recall seeing a ticket to improve document locality.  If that isn't being done at all right now, that could make a massive difference.)

People keep talking about workarounds--yields and delays just shift the time around.  Even if Mongo did any of those, you'd still have a problem; it'd just have taken a bit longer to surface.  The focus should probably be on figuring out why the deletes are taking so much I/O to begin with.

Unfortunately, that'll make things better but not solve the problem. Let's say you make the deletes 10x faster, you are talking about 24s of downtime instead of 4 minutes. Throttling the background process reliably is the only way out.

On a quick look at the code, yields already delete; the code path here appears to be:

MoveChunkCommand::run

OldDataCleanup::doRemove

Helpers::removeRange

cc->yieldSometimes

Have you traced this area and see where the time is being spent?  (I havn't looked at this part of the code before, so I could be looking at the wrong thing entirely.)

(I do wonder what happens if MoveChunkCommand::run follows the "doing delete inline" path and doRemove calls sleepsecs(1) in the current thread, while it's waiting for the delete to complete.  I don't know the thread model well enough to know what happens, or why the non-threaded path exists to begin with.)

--
Glenn Maynard

[Glenn Maynard]

On Fri, May 11, 2012 at 12:30 PM, Kenny Gorman <k...@kennygorman.com> wrote:

Just speaking out loud here, but what about a 'deferred delete' option?

aka: not to delete the old data post move, but rather just leave it in place and somehow mark it as 'old'. Then properly timed and throttled jobs could clean up later?  I think this came up before in some discussions.

You are relying on the config image to ensure you don't get logical corruption however with this approach.

I think this can already happen, actually.  Data is deleted at the end of MoveChunkCommand::run, after marking the chunk migration complete.  If the shard crashes at that point, the data is left on the shard.  This will waste some disk space, but otherwise not break anything: nobody will ask the shard for the data, and the stray data will be cleaned up if the chunk range is sent back to that shard later on (by MigrateStatus::_go #2).

It looks like putting off the deletes would be relatively straightforward: the "6." part of MoveChunkCommand could add to a queue of deletes, which could be done later.  If deletes are currently the major bottleneck of migrating, this could be useful: if your shard is extremely overloaded, it would let you migrate a lot of chunks off the shard at once, and wait to delete the data until after the balancing has given the shard back enough I/O capacity to actually do it.  The harder part would be the heuristic to tell when load is low enough to perform the delete.

(I don't think this would help the OP, though.  He seems to be having deletes not yield properly--this would just put off the problem.)

--
Glenn Maynard

[josh]

Having priority for production reads/writes vs priority for the
balancer, both cases have some good use cases to support them infact
they both sound appealing. Am I the only one that thinks this should
be a config parameter? I could make use of both ie, production during
normal running and if we get a node running hot, we could switch to
giving the balancer priority...

[Glenn Maynard]

On Wed, May 16, 2012 at 5:16 AM, josh <jo...@sitemaker.cc> wrote:

Having priority for production reads/writes vs priority for the
balancer, both cases have some good use cases to support them infact
they both sound appealing. Am I the only one that thinks this should
be a config parameter? I could make use of both ie, production during
normal running and if we get a node running hot, we could switch to
giving the balancer priority...

Fiddly tuning parameters are a last-ditch cop-out; they should only be considered if all else fails.  It should just work.

Prasanna: Have you looked at this any further?  I'm interested in knowing what's making deletes take so long, and why (it seems) they're not yielding properly.  (FYI, my interest is as another developer using Mongo, not as a developer of Mongo.  I don't want to get bit later, either!)

--
Glenn Maynard

[Prasanna .S]

On Mon, May 14, 2012 at 11:17 AM, Glenn Maynard <gl...@zewt.org> wrote:

On Fri, May 11, 2012 at 9:18 PM, Prasanna .S <mypra...@gmail.com> wrote:

inline..

(FYI, you don't need to tell people this on technical lists.)

 

Note that there can be alternate ways of providing a guarantee:

[1] You can say reads will be at 90% capacity, and 10% capacity (CPU AND Disk) goes to chunk movements.

[2] You can say reads will always execute before chunk migrations complete. Yes this will starve chunkMigrations, which is probably fine. Something has to take a hit right?

Starving migrations isn't fine.  If you're at capacity, it's just as fatal: it means you'll stay at capacity forever, because you don't have the spare capacity to bring up another shard to handle the load and stop being overloaded.

At that point downtime is un-avoidable. Let's talk about the default case: System is way below capacity. A "background" process called shard balancing starts. Site goes down. This absolutely can't happen. We are not talking about corner cases here. You have to throttle migrations.

 

Note that, the other option is, read queue keeps getting incrementally larger, which on saturation - you'll hit timeouts - regardless of weather your timeout numbers are 100 ms or 10s.

Depending on your workload, that's often not the case.  If you're a website, it means browser requests take longer, which is self-balancing: browsers will only allow so many requests to be in the air at a time.  It'll slow down rather than sending more and more requests, so your work queues won't grow without bound.  (They can still grow enough to cause timeouts, but you can go much further over capacity before that happens.)

There are enough browsers. It's usually not a question of multiple requests from a single browser. Practically speaking, self-balancing isn't really there.

 

If your use case isn't like that, then you can't let yourself end up in this situation to begin with--since *neither* option is acceptable.

Read above. It's not a special situation. I din't let myself get into this. System has ample spare capacity.

while (true) writeToDisk() will kill any capacity.

 

Are your records very small (on the order of kilobytes) and fragmented, which might cause the deletes to die from the seeking?  (I recall seeing a ticket to improve document locality.  If that isn't being done at all right now, that could make a massive difference.)

Yes my records are very small. The advise I get is to break up the chunks with a reduced chunkSize. I'm giving this work-around a try right now.

 

People keep talking about workarounds--yields and delays just shift the time around.  Even if Mongo did any of those, you'd still have a problem; it'd just have taken a bit longer to surface.  The focus should probably be on figuring out why the deletes are taking so much I/O to begin with.

Unfortunately, that'll make things better but not solve the problem. Let's say you make the deletes 10x faster, you are talking about 24s of downtime instead of 4 minutes. Throttling the background process reliably is the only way out.

On a quick look at the code, yields already delete; the code path here appears to be:

MoveChunkCommand::run

OldDataCleanup::doRemove

Helpers::removeRange

cc->yieldSometimes

Have you traced this area and see where the time is being spent?  (I havn't looked at this part of the code before, so I could be looking at the wrong thing entirely.)

(I do wonder what happens if MoveChunkCommand::run follows the "doing delete inline" path and doRemove calls sleepsecs(1) in the current thread, while it's waiting for the delete to complete.  I don't know the thread model well enough to know what happens, or why the non-threaded path exists to begin with.)

Unclear. Haven't traced my production database this much.

 

--
Glenn Maynard

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com

To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com

See also the IRC channel -- freenode.net#mongodb

[Prasanna .S]

No, I have not. Eliot got in touch with me and offerred to look into it, which is really great! Investigations are yet to happen. So will update here if we find something.

 

--
Glenn Maynard

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com

To unsubscribe from this group, send email to

[Glenn Maynard]

On Wed, May 16, 2012 at 10:14 AM, Prasanna .S <mypra...@gmail.com> wrote:

At that point downtime is un-avoidable. Let's talk about the default case: System is way below capacity. A "background" process called shard balancing starts. Site goes down. This absolutely can't happen. We are not talking about corner cases here. You have to throttle migrations.

Migrations already yield.  The solution isn't to add more throttling, when we don't yet know why the yields already in there aren't doing the job.  The problem needs to be identified before coming up with solutions.

 There are enough browsers. It's usually not a question of multiple requests from a single browser. Practically speaking, self-balancing isn't really there.

I guess that depends on your page's usage patterns.

 

If your use case isn't like that, then you can't let yourself end up in this situation to begin with--since *neither* option is acceptable.

Read above. It's not a special situation. I din't let myself get into this. System has ample spare capacity.

while (true) writeToDisk() will kill any capacity.

Mongo doesn't do that.  It does while(true) { writeData(); yieldToOtherRequests(); }.  (I don't know the details of yielding, eg. how it handles fairness.)

Yes my records are very small. The advise I get is to break up the chunks with a reduced chunkSize. I'm giving this work-around a try right now.

https://jira.mongodb.org/browse/SERVER-3294 would probably help that--actually, it seems like a serious requirement for having lots of very small records.  I don't know if there's any locality strategy in Mongo right now at all.

--
Glenn Maynard

[Prasanna .S]

Update on this thread. We have solved the issue following Eliot's advice. We had to limit chunks to contain at-most 100K documents, and things worked fine. 

Thanks to Eliot and MongoHQ guys for guiding us solve this in production!

--
Glenn Maynard

[news...@syslog.com]

On May 18, 5:57 am, "Prasanna .S" <myprasa...@gmail.com> wrote:
> We had to limit chunks to contain at-most 100K documents

How did you set that?

[Prasanna .S]

change the chunkSize. My guess is sharding seems to be breaking it down automatically.

[Glenn Maynard]

On Fri, May 18, 2012 at 4:57 AM, Prasanna .S <mypra...@gmail.com> wrote:

Update on this thread. We have solved the issue following Eliot's advice. We had to limit chunks to contain at-most 100K documents, and things worked fine. 

This should really be handled transparently, eg. by applying both a chunk size limit and a document count limit.  A reliable database backend shouldn't require manual fiddling with knobs to behave reliably.

--
Glenn Maynard

[Eliot Horowitz]

Glenn - agreed.
We're working on a bunch of different things for 2.2, including making
the deletes faster/less impactful and probably lowering the maximum
number of documents in a chunk as well.