Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
mongodb-user
Home
+ new post
About this group
Join this group
map reduce taking forever to reduce 26 records!
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   4 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
landon.silla  
View profile  
 More options Oct 26 2012, 5:04 pm
From: "landon.silla" <landon.si...@brandads.com>
Date: Fri, 26 Oct 2012 14:04:06 -0700 (PDT)
Local: Fri, Oct 26 2012 5:04 pm
Subject: map reduce taking forever to reduce 26 records!
Print | Individual message | Show original | Report this message | Find messages by this author

Here's what I'm doing:

mongos> db.campaign_raw_data_459_imp.count()
21800002
mongos> db.campaign_raw_data_459_imp.find({ts:1350585328}).count()
26
mongos>  map = function () { emit(this.cookie, 1);}
function () {
    emit(this.cookie, 1);

}

mongos>  reduce = function (key, values) {return 1;}
function (key, values) {
    return 1;
}

mongos>
mongos> db.campaign_raw_data_459_imp.mapReduce(map, reduce, {out: { replace
: "garbage"}}, query={ts:1350585328})

 //This is hanging and taking forever, over 10 minutes now

There are 21m documents in this collection.  And, for a given
ts=1350585328, there are 26 records.  The goal here is to count how many
unique cookies are in the matched records.  So it should find all the match
documents, 26 of them.  And then drop them in buckets based on cookie, and
then count the buckets.  I would presume that it would do the find, based
on the query, FIRST, and then do a map/reduce with those returned values.
 If that's the case, the size of the collection shouldn't matter at all.
 It did the count() in the second line in a blink of the eye.

I have this indexed on ts, and I have a three shard setup with three
replica sets.

Why is this taking so long?  For prod, I'm going to open up ts to be a
range, so it will have many many more matched documents than just 26.  


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
landon.silla  
View profile  
 More options Oct 26 2012, 5:22 pm
From: "landon.silla" <landon.si...@brandads.com>
Date: Fri, 26 Oct 2012 14:22:45 -0700 (PDT)
Local: Fri, Oct 26 2012 5:22 pm
Subject: Re: map reduce taking forever to reduce 26 records!
Print | Individual message | Show original | Report this message | Find messages by this author

Update, I'm still waiting for the mapReduce to finish ....


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Jenna deBoisblanc  
View profile  
 More options Nov 7 2012, 10:37 am
From: Jenna deBoisblanc <jenna.deboisbl...@10gen.com>
Date: Wed, 7 Nov 2012 07:37:42 -0800 (PST)
Local: Wed, Nov 7 2012 10:37 am
Subject: Re: map reduce taking forever to reduce 26 records!
Print | Individual message | Show original | Report this message | Find messages by this author

Did the command ever finish? Could you do post the output of,

db.campaign_raw_data_459_imp.find({ts: 1350585328}).explain() ?

Could you also post the output of, db.currentOp() while the command is
running?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Jenna deBoisblanc  
View profile  
 More options Nov 7 2012, 10:45 am
From: Jenna deBoisblanc <jenna.deboisbl...@10gen.com>
Date: Wed, 7 Nov 2012 07:45:43 -0800 (PST)
Local: Wed, Nov 7 2012 10:45 am
Subject: Re: map reduce taking forever to reduce 26 records!
Print | Individual message | Show original | Report this message | Find messages by this author

Ok, I believe the issue is the syntax of your query-

> db.campaign_raw_data_459_imp.mapReduce(map, reduce, {out: { replace :

"garbage"}}, query={ts:1350585328})

should be,

> db.campaign_raw_data_459_imp.mapReduce(map, reduce, {out: { replace :

"garbage"}, query: {ts:1350585328}})

to be more clear, the query should be in the same object as "out", i.e.-
{out: { replace : "garbage"}, query: {ts:1350585328}}

Your current MR command probably doesn't use the query and must scan all of
the documents in the collection.  The documentation admittedly doesn't do a
great job illustrating the correct syntax, and I will see if we can make
the docs more clear.

Please let me know if this resolves the problem.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2013 Google