[mongodb-user] slow findandmodify performance

[Dustin Norlander]

Have a capped collection working as a queue. capped at ~10 gigs. The
database is on an amazon large slice (8gigs ram), and the database is
only used to insert task, and use findandmodify to get the task back
out.

each of my findandmodify is taking 8 to 9 millis which is way too
slow. released_on date field is indexed (Ascending).

Any help?



profiling -- querying on the id and insert performance are fine, the :

> db.system.profile.find()
{ "ts" : "Thu May 13 2010 16:53:04 GMT+0000 (UTC)", "info" : "insert
twitter.tweet_cache", "millis" : 0 }
{ "ts" : "Thu May 13 2010 16:53:04 GMT+0000 (UTC)", "info" : "query
twitter.tweet_cache ntoreturn:1 reslen:3007 nscanned:1
query: { _queue.released_on: { $lt: new Date(1273769584364) } }
nreturned:1 bytes:2991", "millis" : 8 }
{ "ts" : "Thu May 13 2010 16:53:04 GMT+0000 (UTC)", "info" : "update
twitter.tweet_cache query: { _id: 13901668438 } nscanned:1
1 key updates ", "millis" : 0 }
{ "ts" : "Thu May 13 2010 16:53:04 GMT+0000 (UTC)", "info" : "query
twitter.tweet_cache ntoreturn:1 idhack reslen:3007 bytes:2991",
"millis" : 0 }
{ "ts" : "Thu May 13 2010 16:53:04 GMT+0000 (UTC)", "info" : "query
twitter.$cmd ntoreturn:1 command reslen:3031 bytes:3015", "millis" :
9 }
{ "ts" : "Thu May 13 2010 16:53:04 GMT+0000 (UTC)", "info" : "insert
twitter.tweet_cache", "millis" : 0 }



here is the queue java code I am using to do the findandmodify:

BasicDBObject query = new BasicDBObject();
query.put("_queue.released_on", new BasicDBObject("$lt",new Date()));


BasicDBObject set = new BasicDBObject();
set.put("_queue.released_on", new Date(new Date().getTime() +
this.releaseLockAfter))
BasicDBObject inc = new BasicDBObject();
inc.put("_queue.retries", 1);

BasicDBObject update = new BasicDBObject();
update.put("$set", set);
update.put("$inc", inc);

BasicDBObject command = new BasicDBObject();
command.put("findandmodify", coll.getName());
command.put("query", query);
command.put("update", update);
command.put("new", true);

DBObject result = coll.getDB().command(command);





Mongo stat output:
insert/s query/s update/s delete/s getmore/s command/s mapped vsize
res % locked % idx miss conn time
247 146 173 0 0 74 94242 94452
606 0.776 0 11 16:46:59
250 150 76 0 0 76 94242 94452
607 0.971 0 11 16:47:00
111 135 167 0 0 68 94242 94452
607 0.885 0 11 16:47:01
139 149 174 0 0 76 94242 94452
608 0.933 0 11 16:47:02
196 143 165 0 0 72 94242 94452
608 0.967 0 11 16:47:03
83 133 73 0 0 68 94242 94452
608 0.863 0 11 16:47:04
206 129 166 0 0 65 94242 94452
609 0.858 0 11 16:47:05
115 141 166 0 0 72 94242 94452
609 0.889 0 11 16:47:06
179 150 79 0 0 76 94242 94452
609 0.782 0 11 16:47:07

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Eliot Horowitz]

How large are your objects?
My guess is that its because findAndModify isn't implemented in a
terribly fast manor (pretty easy to fix)

[Dustin Norlander]

Thanks for the help. The objects themselves aren't terribly large
(its a cache of twitter data) but the collection has about 10 million
of them.

Is there a ticket I should vote for for fixing the findandmodify speed?

[Eliot Horowitz]

http://jira.mongodb.org/browse/SERVER-1117

[Mathias Stearn]

Could you make sure you have an index on _id? findAndModify assumes
one, but they aren't created automatically for capped collections.

On Thu, May 13, 2010 at 11:44 PM, Eliot Horowitz