Groups keyboard shortcuts have been updated
Dismiss
See shortcuts
More sharding irregularities
32 views
Skip to first unread message
Philip Southam
Nov 17, 2010, 8:51:14 PM11/17/10
to mongod...@googlegroups.com
So after yesterday's adventure (http://groups.google.com/group/mongodb-user/browse_thread/thread/74a074813d007bfd) I thought I was out of the woods. Unfortunately I'm seeing more data irregularities. This time on queries of indexed fields (not _id).
This is happening across multiple objects, but here's an example of what's going on. I've got an object that contains the following fields; _id, name, slug, obj_type, db_id, etc...
Now if I query by the slug "2-fast-2-furious", which just so happens to be unique in this collection I get one result. (The following commands are run from a mongo -> mongos connection unless otherwise noted)
> var m = db.base.find({slug:'2-fast-2-furious'});
> m.length()
1
But if I do a count db.base.find({slug:'2-fast-2-furious'}).count(), I am told that there is two results.
> db.base.find({slug:'2-fast-2-furious'}).count();
2
I do an explain on the find:
> db.base.find({slug:'2-fast-2-furious'}).explain()
{
"clusteredType" : "SerialServer",
"shards" : {
"set1/mongo0-a:10000,mongo1-a:10000,mongo2-a:10000" : [
{
"cursor" : "BtreeCursor slug_1",
"nscanned" : 1,
"nscannedObjects" : 1,
"n" : 1,
"millis" : 0,
"indexBounds" : {
"slug" : [
[
"2-fast-2-furious",
"2-fast-2-furious"
]
]
}
}
],
"set2/mongo0-a:20000,mongo1-a:20000,mongo2-a:20000" : [
{
"cursor" : "BtreeCursor slug_1",
"nscanned" : 0,
"nscannedObjects" : 0,
"n" : 0,
"millis" : 0,
"indexBounds" : {
"slug" : [
[
"2-fast-2-furious",
"2-fast-2-furious"
]
]
}
}
],
"set3/mongo0-a:30000,mongo1-a:30000,mongo2-a:30000" : [
{
"cursor" : "BtreeCursor slug_1",
"nscanned" : 1,
"nscannedObjects" : 1,
"n" : 0,
"millis" : 0,
"indexBounds" : {
"slug" : [
[
"2-fast-2-furious",
"2-fast-2-furious"
]
]
}
}
]
},
"nscanned" : 2,
"nscannedObjects" : 2,
"n" : 1,
"millisTotal" : 0,
"millisAvg" : 0,
"numQueries" : 3,
"numShards" : 3
}
It shows a result was found on both shard 1 and 3. So I connected directly to the masters of shards 1 and 3 and indeed the same record exists on both of them. According to the printShardingStatus output, it should only exist on shard 1, The id for both the records found on the masters is "LK83" (which is correct)
--- Sharding Status ---
sharding version: { "_id" : 1, "version" : 3 }
shards:
{
"_id" : "shard1",
"host" : "set1/mongo0-a:10000,mongo1-a:10000,mongo2-a:10000"
}
{
"_id" : "shard2",
"host" : "set2/mongo0-a:20000,mongo1-a:20000,mongo2-a:20000"
}
{
"_id" : "shard3",
"host" : "set3/mongo0-a:30000,mongo1-a:30000,mongo2-a:30000"
}
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "movieclips", "partitioned" : true, "primary" : "shard1" }
movieclips.base chunks:
{ "_id" : { $minKey : 1 } } -->> { "_id" : "234H" } on : shard3 { "t" : 4000, "i" : 0 }
{ "_id" : "234H" } -->> { "_id" : "CK2Fd" } on : shard3 { "t" : 5000, "i" : 0 }
{ "_id" : "CK2Fd" } -->> { "_id" : "HT2w" } on : shard3 { "t" : 7000, "i" : 0 }
{ "_id" : "HT2w" } -->> { "_id" : "Nb2v" } on : shard1 { "t" : 7000, "i" : 1 }
{ "_id" : "Nb2v" } -->> { "_id" : "Yz7gx" } on : shard2 { "t" : 3000, "i" : 2 }
{ "_id" : "Yz7gx" } -->> { "_id" : "e2sb4" } on : shard2 { "t" : 6000, "i" : 2 }
{ "_id" : "e2sb4" } -->> { "_id" : "jCY6e" } on : shard2 { "t" : 6000, "i" : 3 }
{ "_id" : "jCY6e" } -->> { "_id" : "s5MW" } on : shard1 { "t" : 3000, "i" : 6 }
{ "_id" : "s5MW" } -->> { "_id" : "zyxK" } on : shard1 { "t" : 3000, "i" : 7 }
{ "_id" : "zyxK" } -->> { "_id" : { $maxKey : 1 } } on : shard2 { "t" : 3000, "i" : 0 }
Now if I drop the index and do another explain:
> db.base.find({slug:'2-fast-2-furious'}).explain()
{
"clusteredType" : "SerialServer",
"shards" : {
"set1/mongo0-a:10000,mongo1-a:10000,mongo2-a:10000" : [
{
"cursor" : "BasicCursor",
"nscanned" : 41696,
"nscannedObjects" : 41696,
"n" : 1,
"millis" : 46,
"indexBounds" : {
}
}
],
"set2/mongo0-a:20000,mongo1-a:20000,mongo2-a:20000" : [
{
"cursor" : "BasicCursor",
"nscanned" : 41287,
"nscannedObjects" : 41287,
"n" : 0,
"millis" : 50,
"indexBounds" : {
}
}
],
"set3/mongo0-a:30000,mongo1-a:30000,mongo2-a:30000" : [
{
"cursor" : "BasicCursor",
"nscanned" : 41320,
"nscannedObjects" : 41320,
"n" : 0,
"millis" : 121,
"indexBounds" : {
}
}
]
},
"nscanned" : 124303,
"nscannedObjects" : 124303,
"n" : 1,
"millisTotal" : 217,
"millisAvg" : 72,
"numQueries" : 3,
"numShards" : 3
}
It looks to have found only one. I've tried db.base.reIndex(), dropping all the indexes and re-creating them (via dropIndex andensureIndex), running a db.repairDatabase() multiple times in many different orders and still cannot get the correct behavior.
On a more broad scale, which is what got my attention in the first place.
> db.base.find({obj_type:207}).count();
1586
> var c = db.runCommand({distinct:'base', key:'_id', query:{obj_type:207}});
> c.values.length
1444
There's a disparity of 124 movies (obj_type 207) in our counts. The same scenario is true across the collection, no matter what the obj_type. Need some suggestions as to how I can rectify this.
Eliot Horowitz
Nov 17, 2010, 9:57:16 PM11/17/10
to mongod...@googlegroups.com
That's expected given what happened.
There is data that its the wrong place.
There is data that its the wrong place.
This script should fix it:
https://github.com/mongodb/mongo-snippets/blob/master/sharding/chunk-checker.js
Though I would take a backup first.
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
Philip Southam
Nov 17, 2010, 10:25:10 PM11/17/10
to mongod...@googlegroups.com
So just to confirm, I run this directly against one of my config instances (I've got three) and not the mongos instance/client?
# mongo chunk-checker.js mongo-config0-a:40000
Thanks,
Eliot Horowitz
Nov 17, 2010, 10:53:11 PM11/17/10
to mongod...@googlegroups.com
Correct.
If you run as is first it will just what has to be done without doing anything.
If you run as is first it will just what has to be done without doing anything.
Philip Southam
Nov 17, 2010, 11:08:00 PM11/17/10
to mongod...@googlegroups.com
I guess I expected a little more fanfare upon running, but it was quite quick and painless...
running:
# mongo chunk-checker.js --host mongo-config0-a:40000
--then--
# mongo chunk-checker.js --host mongo-config0-a:40000 --shell
> cleanUpChunks();
was almost a little to easy. Thanks again for the fix.
Dimetrio I
Aug 14, 2013, 7:17:27 AM8/14/13
to mongod...@googlegroups.com, phi...@movieclips.com
If authorization is enabled
четверг, 18 ноября 2010 г., 8:08:00 UTC+4 пользователь Philip Southam написал:
insert
shards[shard].auth("admin","USER","PASS")
before
var count = c.count(query);
in chunk-checker.js
четверг, 18 ноября 2010 г., 8:08:00 UTC+4 пользователь Philip Southam написал:
Reply all
Reply to author
Forward
0 new messages