Config DB out of sync with actual data on shards

[zatter99]

Hi all,
we have experienced a problem where we cannot query specific data via
mongos. It turns out that mongos canot find the data and is looking at
the wrong shard. The data of that chunk actually resides on the other
shard which, using mongo client on that master, can be successfully
queried. We do use 1.8.3 with 2 replica sets of 3 instances each,
making up 2 shards.
Any hint on how we could figure out what went wrong?

Thanks.

[Scott Hernandez]

Are all your mongos instances using the same config servers? If you
setup some pointing to different config servers you could get this
behavior.

The config servers don't do replication in the way that replica sets
do so you need to make sure you setup things correctly.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[zatter99]

Let me double check, but I believe all mongos instances are using the
same 3 config servers.

[Scott Hernandez]

You can also check to make sure that all three config servers have the
same chunks; it is a pretty thing to get away with a config like that
for long. The system check for consistent reads from the config
servers.

It is also possible the data you found was old (left over from a
migration which failed); in scenarios like that you can duplicate or
data on the old shard. If you have deleted the data since then it will
look like the data is in the wrong place, but isn't really.

Is it possible that you wrote directly to the shard, not going through
mongos? This can also cause data to be on the incorrect shard, for
example.

From my experiences the most likely cause of problems like this is the
last option, writing directly to the shard.

[Christian Schellschmidt]

Hi

We have enables sharding last week and we have the same behaviour. Some data are on the wrong shard. 

I've checked the mongos, there are going all to the same 3 config server. I have checked all config server and the sharding status. All three point to node01 for chunk between  "public.id" : "29a7926c-b932-469a-8c25-b507021651a8" } -->> { "public.id" : "37b13393-5ff3-4b8e-86f7-838bdfb0aeea" . But the document with this public.id from this range are on node02. 

new document written correct to node01 but we don't find the old one in our queries. I found the documents only, when the queries runs direct on the second shard.

[Scott Hernandez]

Please start a new thread/topic with your info. Please provide all the
sharding commands you have fun and the current
db.printShardingStatus() output (posting to gist/pastebin/pastie/etc
is best).

Can you also post the code you are using to write to those documents
(inserts/update/remove/findAndModify)? Just an example of your common
operations would help.

Please also run the query with explain() at the end and include it.

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/3y4DlWvoo5oJ.

[Asya Kamsky]

You can connect to config server directly via mongo shell and run
> use config
> db.chunks.find()

http://docs.mongodb.org/manual/reference/config-database/
This page has explanations of all collections in the config db. All three config servers must have the same contents.

Asya
P.s. it's best not to add to a very old thread - start a new thread with your question.