Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
mongodb-user
Home
+ new post
About this group
Join this group
How to running Mapreduce on Secondary database to output other db.
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   8 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Tom Vo  
View profile  
 More options May 13 2012, 10:59 pm
From: Tom Vo <tom...@nexcel.vn>
Date: Sun, 13 May 2012 19:59:45 -0700 (PDT)
Local: Sun, May 13 2012 10:59 pm
Subject: How to running Mapreduce on Secondary database to output other db.
Print | Individual message | Show original | Report this message | Find messages by this author
Dear all
We setuped Replication Set for our system. And we faced out the
problem about performance when used map reduce to out put data. Could
we run map/reduce on Secondary database to output into other data?
Because warning message "too much data for in memory map/reduce"
appeared when we use option "inline:1".

Thanks and Best Regards.
======================================================
Diagram Mongodb
Version 2.0.4 - 64 bit
Memory (RAM) : 32.0GB
OS: Windows Server 2008 R2 Standard


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Scott Hernandez  
View profile  
 More options May 13 2012, 11:08 pm
From: Scott Hernandez <scotthernan...@gmail.com>
Date: Sun, 13 May 2012 23:08:34 -0400
Local: Sun, May 13 2012 11:08 pm
Subject: Re: [mongodb-user] How to running Mapreduce on Secondary database to output other db.
Print | Individual message | Show original | Report this message | Find messages by this author
No, secondaries cannot save data -- that would let them have data the
primary doesn't, and would be lost in terms of replica set
consistency; that is one reason that you can do an inline mapReduce
(where no perm. data is saved) on secondaries.

You should run your map/reduce on the primary.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tom Vo  
View profile  
 More options May 14 2012, 12:52 am
From: Tom Vo <tom...@nexcel.vn>
Date: Sun, 13 May 2012 21:52:32 -0700 (PDT)
Local: Mon, May 14 2012 12:52 am
Subject: Re: How to running Mapreduce on Secondary database to output other db.
Print | Individual message | Show original | Report this message | Find messages by this author
Thanks for your reply
Please help me fix error message "too much data for in memory map/
reduce" when we ran map reduce.
Could you tuning the script below help us.
db.runCommand({
        mapreduce: test,
        map: function Map() {
                emit(this._id.Name,{"Number": this.value.Number});
        },
        reduce: function Reduce(key, values) {
                var Number = 0
                values.forEach(function (val) {
                        Number += val.Number;
                })
                return {
                        "Number": Number
                }
        },
        query: {
                "_id.Times": {
                        $lte: d_i
                }
        },
        out: {
                inline: 1
        }
        ,verbose: true
        ,jsMode: false

});

Best Regards

On May 14, 10:08 am, Scott Hernandez <scotthernan...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Scott Hernandez  
View profile  
 More options May 14 2012, 1:25 am
From: Scott Hernandez <scotthernan...@gmail.com>
Date: Mon, 14 May 2012 01:25:32 -0400
Local: Mon, May 14 2012 1:25 am
Subject: Re: [mongodb-user] Re: How to running Mapreduce on Secondary database to output other db.
Print | Individual message | Show original | Report this message | Find messages by this author
It means you are *returning* too much data. Can you break up the
map/reduce into multiple batches based on a query to only select part
of your collection by _id.Name?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tom Vo  
View profile   Translate to Translated (View Original)
 More options May 14 2012, 1:50 am
From: Tom Vo <tom...@nexcel.vn>
Date: Mon, 14 May 2012 12:50:55 +0700
Local: Mon, May 14 2012 1:50 am
Subject: Re: [mongodb-user] Re: How to running Mapreduce on Secondary database to output other db.
Print | Individual message | Show original | Report this message | Find messages by this author

Dear sir
How to break up the multiple batches. It's the same query:

set d_i = 10
Select  _id.Name , count( value.Number ) as Number
From  test
Group by  _id.Name
Having  _id.Times <= d_i

Best regards.

On Mon, May 14, 2012 at 12:25 PM, Scott Hernandez
<scotthernan...@gmail.com>wrote:

--

Thanks and Best Regards
--------------------------------------------------------------------------- ----------------------------------
*Vo Tan Hau* (*TOM*)
*Senior Database Administrator*
NEXCEL SOLUTIONS LTD
*SMS Tower,Lot 40, Quang Trung Software City, District 12, HCMC, Vietnam.*
*Tel*:+84-8-37154278 *-* *Fax*:+84-8-37154279 www.nexcel.vn*
Messenger *(*Skype+Yahoo+Live*): Vohau2002


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Scott Hernandez  
View profile  
 More options May 14 2012, 8:40 am
From: Scott Hernandez <scotthernan...@gmail.com>
Date: Mon, 14 May 2012 08:40:09 -0400
Local: Mon, May 14 2012 8:40 am
Subject: Re: [mongodb-user] Re: How to running Mapreduce on Secondary database to output other db.
Print | Individual message | Show original | Report this message | Find messages by this author
Using the primary is the easiest approach, and best supported.

To break up the query for map/reduce into many batches you would have
to know the range of values for _id.Name so you could filter on them,
one range at a time, to produce results small enough for a series of
inline map/reduce commands.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tom Vo  
View profile  
 More options May 14 2012, 11:15 am
From: Tom Vo <tom...@nexcel.vn>
Date: Mon, 14 May 2012 22:15:03 +0700
Local: Mon, May 14 2012 11:15 am
Subject: Re: [mongodb-user] Re: How to running Mapreduce on Secondary database to output other db.
Print | Individual message | Show original | Report this message | Find messages by this author

Dear all
This trouble have been fixed.
Thanks for your support.

On Mon, May 14, 2012 at 7:40 PM, Scott Hernandez
<scotthernan...@gmail.com>wrote:

--

Thanks and Best Regards
--------------------------------------------------------------------------- ----------------------------------
*Vo Tan Hau* (*TOM*)
*Senior Database Administrator*
NEXCEL SOLUTIONS LTD
*SMS Tower,Lot 40, Quang Trung Software City, District 12, HCMC, Vietnam.*
*Tel*:+84-8-37154278 *-* *Fax*:+84-8-37154279 www.nexcel.vn*
Messenger *(*Skype+Yahoo+Live*): Vohau2002


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Hansen  
View profile  
 More options Sep 29 2012, 8:06 pm
From: Mark Hansen <m...@digitalbrandmine.com>
Date: Sat, 29 Sep 2012 17:06:09 -0700 (PDT)
Local: Sat, Sep 29 2012 8:06 pm
Subject: Re: [mongodb-user] Re: How to running Mapreduce on Secondary database to output other db.
Print | Individual message | Show original | Report this message | Find messages by this author

I have a similar issue
(see https://groups.google.com/forum/?fromgroups=#!topic/mongodb-user/29Ee...).
 However, using the primary is not an option fo us.  The primary is
dedicated to handling large data loading tasks.  We cannot do the
map-reduce inline because the results sets are large and the distribution
of data values is unknown, so we cannot break up the queries very easily.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2013 Google