Temporary errors on mongo findOne (null response for existing keys)

[Dario Guzik]

Hi!

Some context first: We are working on a project using mongoDB as our
main data storage. There are three shards which are recplica sets of 3
instances. Each one of them with enough RAM an CPU (quad cores and
16GB RAM)
Most collections are sharded but some processing queues that are not
sharded.
We are experiencing some problems when using this queues. When We
start the process that reads these queues and does a lot of writting
sometimes the mongos asking for a key that is allocated in the default
shard (the same that holds the queues) gets a null response, but then
you can manually check that key and it is there in all three instances
of the rs. I assume it has something to do with the heavy writting
load that the rs is coping with.
I need help to find out which path should We take to solve this
errors. Expanding the rs to 5 members is a good idea? moving some
queues elsewhere?

I have db.stats() from the shard and some mongostats log if it helps.

[Dario Alejandro Guzik]

Here you can find stats:

PRIMARY> db.stats()

{

        "db" : "social",

        "collections" : 23,

        "objects" : 56243345,

        "avgObjSize" : 614.3731572864309,

        "dataSize" : 34554401444,

        "storageSize" : 37738672016,

        "numExtents" : 313,

        "indexes" : 34,

        "indexSize" : 4673139968,

        "fileSize" : 47155511296,

        "nsSizeMB" : 16,

        "ok" : 1

}

2012/4/23 Dario Guzik <da...@guzik.com.ar>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Dario Alejandro Guzik]

What bothers me most is that many read operations that should be super fast are taking 100+ ms and still no clue of what might be happening

2012/4/23 Dario Alejandro Guzik <da...@guzik.com.ar>

[Randolph Tan]

Hi,

I have a couple questions.

1. Can you explain what you meant by "key that is allocated in the default shard (the same that holds the queues) ". Specifically, what is a key? Are you referring to the actual document?

2. Did you drop the collection and created a new one with the same name?

3. Can you attach the sample docs of these queries?

Increasing the replica set members can make write performance worse if you use safe writes. However, adding a shard can help distribute the write load.

}

2012/4/23 Dario Guzik <da...@guzik.com.ar>

To unsubscribe from this group, send email to mongodb-user+unsubscribe@googlegroups.com.

[Dario Alejandro Guzik]

2012/4/23 Randolph Tan <rand...@10gen.com>

Hi,

I have a couple questions.

1. Can you explain what you meant by "key that is allocated in the default shard (the same that holds the queues) ". Specifically, what is a key? Are you referring to the actual document?

Sorry, a little confusing. The key I'm referring to is actually a document on an unsharded collection. I am doing findOne(field:something) using the Java API 2.7.2 and the result is null. I know the object is there and no exception is thrown. If I retry the findOne 100ms later I get the document.

 

2. Did you drop the collection and created a new one with the same name?

No, actually It;s a kind of difficult task since the DB is already in production stage and has 500K documents and it's being used

 

3. Can you attach the sample docs of these queries?

yep, rather simple documents:

{ "_id" : ObjectId("4f7330d6487c95ae7196eab5"), "value" : "AAACfZA9IiSpcBABYpw3pXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", "client_id" : "175268255910551", "expires" : ISODate("2012-05-27T15:40:04.978Z"), "scope" : [       "scope1",      "scope2",   "scope3" ], "status" : "valid", "next_update" : ISODate("2012-04-27T15:40:53.676Z"), "inner_id" : "13XXXXX773" }

 the query is over inner_id (not the real name because of privacy issues)

indexes are

[

        {

                "v" : 1,

                "key" : {

                        "_id" : 1

                },

                "ns" : "mybase.data_collection",

                "name" : "_id_"

        },

        {

                "v" : 1,

                "key" : {

                        "inner_id" : 1

                },

                "unique" : true,

                "ns" : "mybase.data_collection",

                "name" : "inner_id_1"

        },

        {

                "v" : 1,

                "key" : {

                        "next_update" : 1

                },

                "ns" : "mybase.data_collection",

                "name" : "next_update_1"

        }

To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/d5nC5I89DSAJ.

To post to this group, send email to mongod...@googlegroups.com.

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[Randolph Tan]

Are you using slaveOk? It could just be that the document was not fully replicated to the secondaries yet.

[Dario Alejandro Guzik]

I am checking right now if the misses corresponds to new documents to check if it is a replication timing issue.

Just to check: findAndModify is slower than update? (leaving aside other aspects like concurrency handling, etc)

2012/4/23 Randolph Tan <rand...@10gen.com>

To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/asi0puyd_3UJ.

To post to this group, send email to mongod...@googlegroups.com.

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[Randolph Tan]

Yes, findAndModify is slightly slower than the equivalent update given that it has to send the original document back.