Request Advice / Workaround for Running Map/Reduce on Slave

[VJ]

Hello. I've looked around and haven't found a workaround, but wanted
to check anyway. Our data analysts need near real-time access to our
Production data. Since I want to keep these queries off the master,
I've setup a traditional slave. They need to be able to run complex
map reduce queries, and the inline option is not scalable. I
understand that this is how the slave and the non-primary replica set
members were designed. Does anyone have any workaround ideas.
Thanks.

[Barrie]

Hey VJ,

What kind of aggregations are you trying to do, specifically? When
are you going into production?
You can take a look at the new aggregation framework here, which will
be available in the next stable release (it's already available in the
unstable release of 2.1.0): http://www.mongodb.org/display/DOCS/Aggregation+Framework

Barrie

[VJ]

Hi Barrie. We're already in production, running 2.0.2. I have looked
at the upcoming aggregation framework, and have been anxiously
awaiting 2.2. However, I didn't think this new framework would allow
any type of writing (even to another DB) on a slave?
Thanks for your prompt response.

[Eliot Horowitz]

I take it inline map/reduce doesn't work for you?

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>

[Chris Westin]

Unfortunately, the initial production release of the aggregation
framework won't have output capabilities. That will come in a future
release; you can upvote and/or watch this ticket for that:
https://jira.mongodb.org/browse/SERVER-3253 .

Can you be more specific about what you mean by the inline option for
map-reduce output not being scalable?

Depending on your needs, you might want to upvote and/or watch this
ticket: https://jira.mongodb.org/browse/SERVER-4264 .

Chris

[VJ]

Hello. Apologies in advance if you get this post twice. I posted
this earlier, but do not see it in the thread.

Eliot, Chris, thanks for your replies. M/R inline is OK when the
result set is not large. Many of our M/R queries return 1000s and
10000s records. Our data analyst needs to be able to modify and run
these M/R queries, and then manipulate the results outside of Mongo
(i.e. Excel, etc). I would like them to be able to do this on a Mongo
slave server.

Also, inline seems to be more strict when handling nulls. For
example, some M/R queries work fine when the output is a collection,
but generate errors when inline is specified. I know that this is a
user (my) error, but still an annoyance.

I've reviewed the tickets, and voted appropriately. Thanks again.

[Chris Westin]

The inline document size limit is 16MB -- are you sure your results
won't fit in that?

Could you give us an example of the differences in null handling?
They should be the same; the only difference should be the output
coming to the command vs being stored in a collection.

Chris

[VJ]

An example of the null handling issue is below. If I test (and fix)
for null in the map, this m/r query runs fine.

Thanks.

rtb@mon-raw> map = function () {
... emit(this.geo.country,{count:1});
... }
function () {
emit(this.geo.country, {count:1});
}
rtb@mon-raw> reduce = function (key, values) {
... var result = {count:0};
... values.forEach(function(doc) {
... result.count += doc.count;
... });
... return result;
... }
function (key, values) {
var result = {count:0};
values.forEach(function (doc) {result.count += doc.count;});
return result;
}
rtb@mon-raw> db.request.mapReduce(map,
... reduce,
... {query:{"created.simple":
{$gte:"20120207192900",$lt:"20120207193000"}},
... out:"test_coll"})
{
"result" : "test_coll",
"timeMillis" : 12494,
"counts" : {
"input" : 4229,
"emit" : 4229,
"reduce" : 133,
"output" : 62
},
"ok" : 1,
}
rtb@mon-raw> db.request.mapReduce(map,
... reduce,
... {query:{"created.simple":
{$gte:"20120207192900",$lt:"20120207193000"}},
... out:{inline:1}})
Thu Feb 16 16:43:46 uncaught exception: map reduce failed:{
"assertion" : "assertion db/commands/mr.cpp:400",
"errmsg" : "db assertion failure",
"ok" : 0
}
rtb@mon-raw>

[Chris Westin]

For now, it sounds like you can workaround the null in the map by
modifying the code. In the meantime, I submitted a server ticket for
this: https://jira.mongodb.org/browse/SERVER-5001 . You might want
to upvote and/or watch that.

Chris

[VJ]

Great. Thanks. I voted on this ticket.

On Feb 17, 1:57 pm, Chris Westin <cwes...@yahoo.com> wrote:
> For now, it sounds like you can workaround the null in the map by
> modifying the code.  In the meantime, I submitted a server ticket for

> this:  https://jira.mongodb.org/browse/SERVER-5001.  You might want

[jonathan]

I am having this issue too.   I didn't quite follow what the work around was, can you please explain how you worked around the issue?

Thanks!

Jonathan

[Chris Westin]

Jonathan: which issue are you having? The inability to run map/
reduce with $out on a secondary (see https://jira.mongodb.org/browse/SERVER-4264),
or the assertions when running map/reduce that produces nulls and uses
$inline (see https://jira.mongodb.org/browse/SERVER-5001)?

There wasn't a workaround given above. Please follow one or both of
the tickets above to keep updated on the status of these issues.

Chris