Well, you can get where the chunks are and the ranges for the keys from the config servers directly, however I guess you've seen this already. I would think you can do this with mongosniff for each of the shards, however it's a little hacky.
However, if your queries modify data, if would suggest reading it back directly from each replica set looking for a match - this would be really easy. Once you know which shard it hit you can work out the exact chunk from the config server data.
> I am currently working on a problem in MongoDB for which I need to identify, for a set of automatically generated queries, which shard, chunk is getting hit (or the mongos is fetching the data from which of them). I have deployed MongoDB with 3 replica sets. I checked the log files but could not find any relevant information. It is possible I have not looked at all the files though. Is there some strategy that anybody can help me with which does not require a change in source code? In case there is none, then can you suggest me some code changes that may help.
> Thanks and Regards,
> Mainak.
> -- > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
Thanks Russ. So the problem is I am generating the query and insert (no update) operation from a script. Now as the script runs MongoDB will do its balancing stuff. During this scripts' execution I want to capture the shard hit information for every read query executed. Now the script is a blackbox to me and I cannot change it. Do you think I can use mongosniff somehow in that case as well? I had changed mongodb's logging to be verbose but it did not help. I am not completely sure Mongo logs this information at all. For mongosniff I have to use the diagnostic logging functionality I believe. How can I do that without taking the mongo deployment down?
On Friday, October 5, 2012 1:27:55 AM UTC-5, Russell Smith wrote:
> Well, you can get where the chunks are and the ranges for the keys from > the config servers directly, however I guess you've seen this already. I > would think you can do this with mongosniff for each of the shards, however > it's a little hacky.
> However, if your queries modify data, if would suggest reading it back > directly from each replica set looking for a match - this would be really > easy. Once you know which shard it hit you can work out the exact chunk > from the config server data.
> On Oct 4, 2012, at 11:04 PM, Mainak Ghosh <famous...@gmail.com<javascript:>> > wrote:
> > Hi,
> > I am currently working on a problem in MongoDB for which I need to > identify, for a set of automatically generated queries, which shard, chunk > is getting hit (or the mongos is fetching the data from which of them). I > have deployed MongoDB with 3 replica sets. I checked the log files but > could not find any relevant information. It is possible I have not looked > at all the files though. Is there some strategy that anybody can help me > with which does not require a change in source code? In case there is none, > then can you suggest me some code changes that may help.
> > Thanks and Regards, > > Mainak.
> > -- > > You received this message because you are subscribed to the Google > > Groups "mongodb-user" group. > > To post to this group, send email to mongod...@googlegroups.com<javascript:> > > To unsubscribe from this group, send email to > > mongodb-user...@googlegroups.com <javascript:> > > See also the IRC channel -- freenode.net#mongodb
You could by connecting a new mongos and making the script go through that. You should then be able to see where it's connecting to. If there are lots of queries you'll probably want to modify mongosniff to log to disk in a nicer format though.
May I ask what you're trying to achieve with this? Working out which chunks are hottest?
> Thanks Russ. So the problem is I am generating the query and insert (no update) operation from a script. Now as the script runs MongoDB will do its balancing stuff. During this scripts' execution I want to capture the shard hit information for every read query executed. Now the script is a blackbox to me and I cannot change it. Do you think I can use mongosniff somehow in that case as well? I had changed mongodb's logging to be verbose but it did not help. I am not completely sure Mongo logs this information at all. For mongosniff I have to use the diagnostic logging functionality I believe. How can I do that without taking the mongo deployment down?
> Thanks and Regards,
> Mainak.
> On Friday, October 5, 2012 1:27:55 AM UTC-5, Russell Smith wrote:
> Well, you can get where the chunks are and the ranges for the keys from the config servers directly, however I guess you've seen this already. I would think you can do this with mongosniff for each of the shards, however it's a little hacky.
> However, if your queries modify data, if would suggest reading it back directly from each replica set looking for a match - this would be really easy. Once you know which shard it hit you can work out the exact chunk from the config server data.
> On Oct 4, 2012, at 11:04 PM, Mainak Ghosh <famous...@gmail.com> wrote:
> > Hi,
> > I am currently working on a problem in MongoDB for which I need to identify, for a set of automatically generated queries, which shard, chunk is getting hit (or the mongos is fetching the data from which of them). I have deployed MongoDB with 3 replica sets. I checked the log files but could not find any relevant information. It is possible I have not looked at all the files though. Is there some strategy that anybody can help me with which does not require a change in source code? In case there is none, then can you suggest me some code changes that may help.
> > Thanks and Regards, > > Mainak.
> > -- > > You received this message because you are subscribed to the Google > > Groups "mongodb-user" group. > > To post to this group, send email to mongod...@googlegroups.com > > To unsubscribe from this group, send email to > > mongodb-user...@googlegroups.com > > See also the IRC channel -- freenode.net#mongodb
> -- > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
You could by connecting a new mongos and making the script go through that. You should then be able to see where it's connecting to. If there are lots of queries you'll probably want to modify mongosniff to log to disk in a nicer format though.
May I ask what you're trying to achieve with this? Working out which chunks are hottest?
> Thanks Russ. So the problem is I am generating the query and insert (no update) operation from a script. Now as the script runs MongoDB will do its balancing stuff. During this scripts' execution I want to capture the shard hit information for every read query executed. Now the script is a blackbox to me and I cannot change it. Do you think I can use mongosniff somehow in that case as well? I had changed mongodb's logging to be verbose but it did not help. I am not completely sure Mongo logs this information at all. For mongosniff I have to use the diagnostic logging functionality I believe. How can I do that without taking the mongo deployment down?
> Thanks and Regards,
> Mainak.
> On Friday, October 5, 2012 1:27:55 AM UTC-5, Russell Smith wrote:
> Well, you can get where the chunks are and the ranges for the keys from the config servers directly, however I guess you've seen this already. I would think you can do this with mongosniff for each of the shards, however it's a little hacky.
> However, if your queries modify data, if would suggest reading it back directly from each replica set looking for a match - this would be really easy. Once you know which shard it hit you can work out the exact chunk from the config server data.
> On Oct 4, 2012, at 11:04 PM, Mainak Ghosh <famous...@gmail.com> wrote:
>> Hi,
>> I am currently working on a problem in MongoDB for which I need to identify, for a set of automatically generated queries, which shard, chunk is getting hit (or the mongos is fetching the data from which of them). I have deployed MongoDB with 3 replica sets. I checked the log files but could not find any relevant information. It is possible I have not looked at all the files though. Is there some strategy that anybody can help me with which does not require a change in source code? In case there is none, then can you suggest me some code changes that may help.
>> Thanks and Regards, >> Mainak.
>> -- >> You received this message because you are subscribed to the Google >> Groups "mongodb-user" group. >> To post to this group, send email to mongod...@googlegroups.com >> To unsubscribe from this group, send email to >> mongodb-user...@googlegroups.com >> See also the IRC channel -- freenode.net#mongodb
> -- > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
I am planning to use MongoDB for heavy data and before starting out I want
to tests its performance. This is a part of that.
I am new to MongoDB so first thanks for your help. It is giving me a
direction to go to. Though I completely do not understand this mongosniff
technique. Can you please break it down a bit further?
On Friday, October 5, 2012, Russell Smith wrote:
> Hi Mainak,
> You could by connecting a new mongos and making the script go through
> that. You should then be able to see where it's connecting to. If there are
> lots of queries you'll probably want to modify mongosniff to log to disk in
> a nicer format though.
> May I ask what you're trying to achieve with this? Working out which
> chunks are hottest?
> On Oct 5, 2012, at 8:33 AM, Mainak Ghosh <famousmai...@gmail.com<javascript:;>>
> wrote:
> > Hi,
> > Thanks Russ. So the problem is I am generating the query and insert (no
> update) operation from a script. Now as the script runs MongoDB will do its
> balancing stuff. During this scripts' execution I want to capture the shard
> hit information for every read query executed. Now the script is a blackbox
> to me and I cannot change it. Do you think I can use mongosniff somehow in
> that case as well? I had changed mongodb's logging to be verbose but it did
> not help. I am not completely sure Mongo logs this information at all. For
> mongosniff I have to use the diagnostic logging functionality I believe.
> How can I do that without taking the mongo deployment down?
> > Thanks and Regards,
> > Mainak.
> > On Friday, October 5, 2012 1:27:55 AM UTC-5, Russell Smith wrote:
> > Well, you can get where the chunks are and the ranges for the keys from
> the config servers directly, however I guess you've seen this already. I
> would think you can do this with mongosniff for each of the shards, however
> it's a little hacky.
> > However, if your queries modify data, if would suggest reading it back
> directly from each replica set looking for a match - this would be really
> easy. Once you know which shard it hit you can work out the exact chunk
> from the config server data.
> > On Oct 4, 2012, at 11:04 PM, Mainak Ghosh <famous...@gmail.com<javascript:;>>
> wrote:
> > > Hi,
> > > I am currently working on a problem in MongoDB for which I need to
> identify, for a set of automatically generated queries, which shard, chunk
> is getting hit (or the mongos is fetching the data from which of them). I
> have deployed MongoDB with 3 replica sets. I checked the log files but
> could not find any relevant information. It is possible I have not looked
> at all the files though. Is there some strategy that anybody can help me
> with which does not require a change in source code? In case there is none,
> then can you suggest me some code changes that may help.
> > > Thanks and Regards,
> > > Mainak.
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "mongodb-user" group.
> > > To post to this group, send email to mongod...@googlegroups.com<javascript:;>
> > > To unsubscribe from this group, send email to
> > > mongodb-user...@googlegroups.com <javascript:;>
> > > See also the IRC channel -- freenode.net#mongodb
> > --
> > You received this message because you are subscribed to the Google
> > Groups "mongodb-user" group.
> > To post to this group, send email to mongodb-user@googlegroups.com<javascript:;>
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe@googlegroups.com <javascript:;>
> > See also the IRC channel -- freenode.net#mongodb
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com<javascript:;>
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com <javascript:;>
> See also the IRC channel -- freenode.net#mongodb
It's like tcp dump but specifically for the mongo protocol. You'd need to set it up to forward mongo connections and then watch / process the output of where things were going. There may be a way of getting this from mongos directly (as it's doing the routing) but if there is I can't find it / don't know of it.
> I am planning to use MongoDB for heavy data and before starting out I want to tests its performance. This is a part of that.
> I am new to MongoDB so first thanks for your help. It is giving me a direction to go to. Though I completely do not understand this mongosniff technique. Can you please break it down a bit further?
> Thanks and Regards,
> Mainak.
> On Friday, October 5, 2012, Russell Smith wrote:
> Hi Mainak,
> You could by connecting a new mongos and making the script go through that. You should then be able to see where it's connecting to. If there are lots of queries you'll probably want to modify mongosniff to log to disk in a nicer format though.
> May I ask what you're trying to achieve with this? Working out which chunks are hottest?
> On Oct 5, 2012, at 8:33 AM, Mainak Ghosh <famousmai...@gmail.com> wrote:
> > Hi,
> > Thanks Russ. So the problem is I am generating the query and insert (no update) operation from a script. Now as the script runs MongoDB will do its balancing stuff. During this scripts' execution I want to capture the shard hit information for every read query executed. Now the script is a blackbox to me and I cannot change it. Do you think I can use mongosniff somehow in that case as well? I had changed mongodb's logging to be verbose but it did not help. I am not completely sure Mongo logs this information at all. For mongosniff I have to use the diagnostic logging functionality I believe. How can I do that without taking the mongo deployment down?
> > Thanks and Regards,
> > Mainak.
> > On Friday, October 5, 2012 1:27:55 AM UTC-5, Russell Smith wrote:
> > Well, you can get where the chunks are and the ranges for the keys from the config servers directly, however I guess you've seen this already. I would think you can do this with mongosniff for each of the shards, however it's a little hacky.
> > However, if your queries modify data, if would suggest reading it back directly from each replica set looking for a match - this would be really easy. Once you know which shard it hit you can work out the exact chunk from the config server data.
> > On Oct 4, 2012, at 11:04 PM, Mainak Ghosh <famous...@gmail.com> wrote:
> > > Hi,
> > > I am currently working on a problem in MongoDB for which I need to identify, for a set of automatically generated queries, which shard, chunk is getting hit (or the mongos is fetching the data from which of them). I have deployed MongoDB with 3 replica sets. I checked the log files but could not find any relevant information. It is possible I have not looked at all the files though. Is there some strategy that anybody can help me with which does not require a change in source code? In case there is none, then can you suggest me some code changes that may help.
> > > Thanks and Regards,
> > > Mainak.
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "mongodb-user" group.
> > > To post to this group, send email to mongod...@googlegroups.com
> > > To unsubscribe from this group, send email to
> > > mongodb-user...@googlegroups.com
> > > See also the IRC channel -- freenode.net#mongodb
> > --
> > You received this message because you are subscribed to the Google
> > Groups "mongodb-user" group.
> > To post to this group, send email to mongodb-user@googlegroups.com
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe@googlegroups.com
> > See also the IRC channel -- freenode.net#mongodb
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
> -- > Mainak Ghosh
> -- > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
